Applying Peer Reviews in Software Engineering Education: An Experiment and Lessons Learned Vahid Garousi, Member, IEEE

Transcription

1 182 IEEE TRANSACTIONS ON EDUCATION, VOL. 53, NO. 2, MAY 2010 Applying Peer Reviews in Software Engineering Education: An Experiment and Lessons Learned Vahid Garousi, Member, IEEE Abstract Based on the demonstrated value of peer reviews in the engineering industry, numerous industry experts have listed it at the top of the list of desirable development practices. Experience has shown that problems (defects) are eliminated earlier if a development process incorporates peer reviews and that these reviews are as effective as or even more effective than testing. It is therefore important for engineering students to peer review each other s work during design projects. However, surprisingly, few engineering courses in universities and colleges include peer-review activities in their design projects. The author thus decided to incorporate peer reviews in the design project of a senior software engineering course in two offerings of the course. The purpose of this article is to present the experimental findings, lessons learned, possible challenges, and recommendations that may be used to promote learning and also the use of peer-review activities in teaching other software, electrical, and computer engineering courses. The results of the experiment show promising signs of using peer review in the design project of the course. Index Terms Design projects, experimental study, lessons learned, peer review, quantitative and qualitative analysis, software engineering. I. INTRODUCTION P EER review is an important step in the design and implementation of engineering systems and projects. This activity provides an evaluation of design concepts, documents, and artifacts to meet quality objectives. Peer reviews are usually performed by independent teams or individuals not associated with the original design team [1]. Particulary in software engineering, peer review refers to a type of software review in which a work product (such as requirements specification, design document, or source code) is examined by its designer (author) and one or more colleagues in order to evaluate its technical content and quality [2]. The purpose of peer review is to provide a disciplined engineering practice for detecting and correcting defects in design artifacts to prevent their persistence into operational use of the artifact or the product [3]. Peer review has also been found to be one of the most effective ways to promote quality and productivity of design processes not only in software engineering but also in other engi- Manuscript received August 27, 2008; revised November 03, First published June 30, 2009; current version published May 05, This work was supported in part by the Alberta Ingenuity New Faculty Award No and the Natural Sciences and Engineering Research Council of Canada (NSERC) under Discovery Grant No The author is with the Software Quality Engineering Research Group (Soft- Qual), Department of Electrical and Computer Engineering, Schulich School of Engineering, University of Calgary, Calgary, AB T2N 1N4, Canada ( vgarousi@ucalgary.ca). Digital Object Identifier /TE neering disciplines including electrical [4], civil [5], [6], mechanical [7], and fire protection engineering [8]. Data collected during the peer-review process is used not only to correct defects but also to evaluate and improve the development process itself. Reports from the software industry, for example, indicate that inspections (one type of peer review) have gained wide acceptance as a development tactic and can take up to 15% of the time allotted to a software project [9]. Based on the demonstrated value of peer reviews in software engineering, numerous industry experts have listed it at the top of the list of desirable software development practices [10], [11]. Furthermore, peer reviewing of project artifacts is an especially important tactic to use during the analysis and design phases since the correction of a defect found early in development can be 10 to 100 times less expensive to fix than rework performed at the system testing stage. For example, a study at NASA s Jet Propulsion Laboratory (JPL), Pasadena, CA, found the ratio of the cost of fixing software defects during inspections to fixing them during formal testing ranged from 1:10 to 1:34 [12]; at the IBM Santa Teresa Lab, San Jose, CA, the ratio was 1:20 [13]; and at the IBM Rochester Lab, Rochester, MN, it was 1:13 [14]. However, surprisingly, not many design-oriented engineering courses in universities and colleges currently include peer-review activities in their courses ([15] [17] are among the notable exceptions). This observation is drawn from the author s experience in having spent more than eight years of his education and career in three Canadian universities. On the other hand, the engineering industry heavily relies on different types of peer-review activities such as team review, inspection, and walkthrough. Engineering educators cannot afford to graduate students who are unable to perform effective peer reviews when they need to critically examine the quality of products, artifacts, and documents generated by their peers. In addition to helping students learn more about the peerreview process, several published articles have also indicated that the peer-review process may offer educational benefits for the participants, such as improved analytical abilities and defect detection abilities (e.g., [18], [19]). There have been other studies in the literature (e.g., [20] [24]) that have empirically studied the impact of different types of peerreview activities in design-oriented courses (which will be discussed in detail in Section II). Interestingly, all of the empirical studies found related to this topic have dealt with software engineering courses. This article attempts to add to the existing body of knowledge by providing more insights and more evidence on using peer reviews in engineering education. The experiment partly replicates ideas from several previous works to investigate and assess the repeatability of the lessons /$ IEEE

2 GAROUSI: APPLYING PEER REVIEWS IN SOFTWARE ENGINEERING EDUCATION 183 TABLE I CATEGORIZATION OF THE RELATED WORKS learned in those works (e.g., [20] [24]). Also, a few other aspects of the peer-review experiment are novel, which resulted in interesting findings, for example, the quantitative and qualitative analysis of defects found during the process and the analysis of accuracy and quality of peer reviews. In this experiment, the author applied peer reviews during the design project in two offerings of a fourth-year technical elective course in the software engineering program of a Canadian university. The outcomes from the instructor and students experience with peer reviews are reported in this paper. The author believes that the recommendations derived from the data analysis are of general interest, and it is hoped the findings will encourage more use of peer reviews in other similar courses since, after all, engineering is by nature a peer-review-based discipline. The rest of this article is structured as follows. A survey of the related work is presented in Section II. The experiment setup details (including the hypothesis, course project description, people involved, and the peer-review process) are explained in Section III. The outcomes of the experiment are reported and analyzed in Section IV. Conclusions and future work are discussed in Section V. II. RELATED WORK A snapshot of the use of peer reviews in engineering education and industry is shown in Table I. The table shows there are not many courses (in the education category) that involve students in peer-review processes. Interestingly, the software engineering category of courses [17], [20] [24] seems to be the dominant field of engineering using peer reviews. Other than software engineering, the author was not able to find any empirical analysis, covering such issues as lessons learned, of peer reviewing in other engineering disciplines. Due to space constraints, only a few of the main related works categorized under engineering education in Table I are discussed here. For a more comprehensive discussion, the reader is referred to [25]. The outline for the senior (fourth-year) design course (ECE 445) [16] at the University of Illinois at Urbana-Champaign includes a team review phase. The followings are excerpts from the course outline: Each team must peer review another; Sponsors are invited to participate in reviews, as they wish, and to make input during those reviews. The course outline of Construction Contracts (CON) 492 [15] at Bradley University s Department of Construction Engineering, Peoria, IL, includes a topic on the role of inspection in construction operations. However, with the information available online, it is not clear how detailed the inspection process is taught in that course. The work in [20] is a comparative study of academic and industrial software projects with the goal of improving academic software engineering projects. As one aspect of the experiment, controlled peer reviews were incorporated in two academic projects (at two Canadian universities: University of British Columbia, Vancouver, BC, and École Polytechnique de Montréal, Montreal, QC) and in an industrial project to compare their effectiveness. Several interesting and promising practical recommendations from the use of peer reviews are reported. For example, the project moderators showed students a video of a typical peer-review meeting from the industrial projects. The authors believe that experienced software engineers should act as moderators, at least for one of the first review meetings. An experiment report with the use of reciprocal peer reviews in a computer science course (i.e., human-computer interaction laboratory) at Indiana University, Bloomington, is reported in [22]. Sullivan, the author and also the instructor, reported that intensive peer review empowered students to master the course concepts, hone their teamwork and communication skills, master the peer-review process, and learn to learn from each other. As an interesting and useful experience, Sullivan reported that for students who are not accustomed to doing peer reviews, inspections (as a more formal type of review) can be too big of a step. For students, the prospect of maintaining inspection statistics can be so intimidating as to prevent mastery of the peer-review basics. He thus suggested that walkthroughs (less formal compared to inspections) can be used as a stepping stone toward building the skills needed for inspections. In Tyran s quantitative experimental study in [23], one of the closest studies to the work presented here, the author studied the impact of a software inspection exercise (a type of peer review) in a software systems analysis and design course in the Western Washington University, Bellingham, in Overall, Tyran found the software inspection exercise to be a worthwhile component of the course. Given the importance of inspections to the practice of systems analysis and design, it seemed very appropriate to allocate some class time to this important topic. The exercise provided an experiential approach to help students gain a better feel for the software inspection process. By working through the exercise, students gained enough confidence to attempt the approach on their own (using the steps discussed in that exercise). The author also observed that once students have completed the exercise, they will often apply their new inspection skills to other courses that have team-based design projects.

3 184 IEEE TRANSACTIONS ON EDUCATION, VOL. 53, NO. 2, MAY 2010 The work in [26] describes techniques that were used in different competency-based, introductory engineering design courses at Washington State University, Pullman, as part of the Transferable Integrated Design in Engineering Education (TIDEE) coalition. The courses emphasized process improvement rather than the evaluation of the product and results. Activities included peer evaluation of team deliverables and led to developing ways to measure performance, reflecting on team processes and setting realistic goals that can be achieved. Feedbacks for the team came from applause for their efforts, a written peer evaluation, and a self-evaluation. The overall outcome and student perceptions of the use of peer review were reported to be outstanding. The Association for Computing Machinery (ACM) curriculum guideline for undergraduate degree programs in software engineering (also known as Software Engineering 2004) [30] is one of the popular guidelines in the area. Although this guideline does not explicitly suggest using peer reviews in software engineering education, it does, however, encourage discussions with peers in different courses. For example, Group Dynamics and Communication is one of the proposed course topics in which the guideline suggests teaching students how to negotiate basic agreements with peers. In summary, compared to similar works, the current article advances the state of the art concerning the use of peer reviews in education in five main aspects: An in-depth survey of the use of peer reviews in engineering education and industry in the above brief discussion in this section, with more comprehensive details being given in [25]; Systematic evaluations of peer-review usage, such as the questions instructors should consider to evaluate a particular peer-review setting (Section III-A); New metrics to assess the effectiveness of peer reviews in education, including the number of reported defects during the process, and the number of false-positive and falsenegative errors in reports (Sections IV-A and IV-B); A realistic analysis of costs and potential challenges/drawbacks that need to be taken into account by instructors considering the use of peer reviews (Section IV-F); An experiment that shows promising evidence of the benefits of using peer review in the design project of a senior-level course and can serve as another evidence report for the usefulness of peer reviews. III. EXPERIMENTAL SETUP A. Hypothesis To formulate the hypothesis of this study, the goal-questionmetric (GQM) framework [31], a popular approach for experimentation in software engineering, was used. The goal of this study is to characterize and analyze the effects and impacts of peer review and the level of formality of this review on both the quality of software artifacts designed and built by senior software engineering students and on the students learning, from the point of view of both educators and students in the context of a senior engineering course. With the above goal in mind, using the GQM approach, the following research questions (RQs) were posed. Note that these questions are meant to compare the use of peer reviewing to its absence. RQ 1. How does peer reviewing impact the number of defects found during the analysis, design, and development phases of the software development process? RQ 2. How accurate are students in peer reviewing and identifying defects (issues)? For this, the numbers of falsepositive and false-negative errors in each peer review report were measured by the instructor. RQ 3. Is the success of incorporating peer review in a course predictable? For this question, the results of the two course offerings in two different terms are compared. RQ 4. How does students confidence in performing effective peer reviews change before and after the exercise? RQ 5. What are students perceptions of their knowledge gained through peer reviewing? RQ 6. How satisfied are students with the peer-review process? To address the above questions using the GQM approach, appropriate measurements were gathered during the experiment and are presented and analyzed in Section IV. B. Project Description The author incorporated peer reviews during the course project of a fourth-year technical elective course, Component-Based Software Engineering, SENG , in the software engineering program of the University of Calgary, Calgary, AB, Canada, during the Winter terms of 2007 and The course outline is available on the Web [17]. The course project was to design, implement, and test a Webbased flight ticket reservation system using the concepts learned in the course. The final system, which included only the flight ticket reservation feature, was supposed to be similar to popular online travel reservation systems such as or The objectives of the course project were the following: to understand the theoretical and practical concepts in component-based software engineering; to obtain hands-on experience during the course project with the Enterprise Java Beans (EJB) and Web component technologies in the Java 2 platform Enterprise Edition (J2EE). The course project consisted of three milestones. Milestone 1 Analysis models of the system specified using the Unified Modeling Language (UML) (due at the end of the fourth week of the course). Milestone 2 System prototype, detailed UML design models, and the component specification definitions (due at the end of the eighth week). Milestone 3 Final working system and final report (the report and the system due at the end of the 12th week). C. People Involved There were 23 fourth-year students involved in this study during two course offerings: 14 students in Winter 2007 and 9 in Winter Students worked in pairs. Where the class had an odd number of students, one student was allowed to work alone.

4 GAROUSI: APPLYING PEER REVIEWS IN SOFTWARE ENGINEERING EDUCATION 185 Fig. 1. The adapted design process with built-in peer review. TABLE II PEER REVIEWS ASSIGNMENT TABLE Thus, there were seven and five teams in each of the terms, respectively. There was one instructor for the course (the author of this article). The course also had a teaching assistant (TA) in Winter 2008 who was a graduate student of software engineering. D. Peer-Review Process To start the process in each of the two terms, a peer-review assignment table was devised by the instructor and students. For brevity, the settings of both terms are shown together in Table II, where groups 1 5 denote the Winter 2008 teams and groups 6 12 are the Winter 2007 teams. To balance the workload evenly, each team reviewed the project artifacts of two other teams, and each team was reviewed by two other teams. For example, the artifacts of team 1 were reviewed by teams 2 and 3. The students were asked to follow the iterative waterfall software development process model with a built-in peer review after each phase (as depicted in Fig. 1). The system requirements specification was provided as a detailed document by the instructor. High-level software analysis documents were first generated by student teams. According to the body of knowledge in model-driven software engineering, these documents were of the type Platform Independent Models (PIMs), a type of Unified Modeling Language (UML) model as defined by the Model-Driven Architecture (MDA) standard [32]. Peer reviews were done on analysis artifacts. After peer-review results were in, teams were asked to consider all the reviews they received and decide whether they wanted to change the artifact reviewed, based on the provided comments. They also reported all the necessary details and changes (if any) in their reports. A similar scenario was applied for the design stage. These design documents were of the type Platform-Specific Models (PSMs), as defined by the MDA standard [32]. The last phase of the design process with peer review was implementation of the system in which source (program) code was the artifact under peer review. To let students practice some autonomy in managing their projects, the detailed timing for submitting projects for peer reviews and the deadlines for getting these back from other teams were not dictated, and it was up to teams to come up with a reasonable timing plan; in other words, the peer-review process was not micromanaged. However, the instructor was open to hear and help out with any difficulties that arose with respect to timing in peer reviews, miscommunication, personal issues, and so on. As for the level of formality and type of peer review (team review, inspection, walkthrough, peer desk-check, and ad hoc review), Sullivan s advice [22] was followed: For students who are not accustomed to doing peer reviews, inspections (as a more formal type of review) can be too big a step. For students, the prospect of maintaining inspection statistics can be so intimidating as to prevent them mastering the basics of peer review. Thus, Sullivan [22] recommended that walkthroughs can be used as a stepping stone toward building the skills needed for inspections. For this reason, the walkthrough paradigm was followed in the course as well. A walkthrough is defined as an informal review in which the author (developer) of a work product (artifact) sends the product to a group of peers and solicits comments [33]. The instructor explicitly discussed with the students what had to be classified as a defect. At each of the three peer-review stages analysis, design and implementation the classification and meaning of defects in that stage was discussed by the instructor in the class. For example, according to the MDA standard [32], the analysis software models are composed of

5 186 IEEE TRANSACTIONS ON EDUCATION, VOL. 53, NO. 2, MAY 2010 Fig. 2. Defects found during the peer-review process by each team. use-case diagrams 1, use-case descriptions, and class diagrams. An example of a defect in a use-case diagram would be if the modeler has neglected to specify a system use-case, such as Search Flights (omission errors). Also, a modeler may have made an error in modeling the relationship between use-cases, for example, specifying that the use-case Search Flight should include the use-case Login, which is not necessitated by the system requirements. In terms of the granularity of defects, students were asked to specify each defect as one defect item, as in the examples described above, and not to group, or generalize, several defects into one. This practice is usually followed in the software industry to list all the defects explicitly. The expectations for the peer-review reports, such as the required structure of the report, granularity, and detail of the reported defects, were communicated with the students before the process started. For instance, it was suggested that a hierarchical structure would be the best for the peer-review reports, thus first listing all of the defective artifacts in each of the analyses, designs, or code phases, and then itemizing the defects for each of these artifacts as found in the peer-review process. The instructor did not micromanage the timing details and thoroughness of the peer reviews. The students were given the freedom to choose their own defect checklist and degree of thoroughness. As discussed in Section IV-E, student comments indicated that they experienced some difficulties, and some of them 1 In software engineering, a use-case diagram is a type of behavioral diagram defined by the UML and created from a use-case analysis. Its purpose is to present a graphical overview of the functionality provided by a system in terms of actors, their goals (represented as use-cases), and any dependencies between those use-cases. [34]. wished they had been given more details about peer-review expectations as well as more formal training on how to carry out peer review. Given the time constraints of a 13-week course, and also the intensive nature of the component-based software engineering subject matter, the instructor set the above guidelines, and the outcome proved to be satisfactory. Certainly, new ideas and findings have emerged from this experiment for example, that more details about peer-review expectations should be provided to students in the future. IV. EXPERIMENTAL RESULTS A. Defects Found During the Process To fulfill the RQ 1 (Section III-A), Fig. 2 shows the distribution and average values of the number of defects found and reported during the peer-review process for the deliverables of each team, as reviewed by another team. Note that these metrics are the actual number of defect items listed in the review reports. As it will be discussed in Section IV-B, there were a few false-positive and false-negative defects, which impacted the accuracy and quality of review reports. Teams and in Fig. 2 denote the artifacts of the first and the second teams peer reviewed by each reviewer team (as per the peer-review assignment in Table II). For example, team #3 reviewed the artifacts of teams 1 and 2. Three main observations from the metrics in Fig. 2 are discussed next. Since the curves are not homogenous, it can be observed that a reviewing team found different numbers of defects for different teams at each milestone. For example, team 3

6 GAROUSI: APPLYING PEER REVIEWS IN SOFTWARE ENGINEERING EDUCATION 187 Fig. 3. The number of FP and FN errors versus all defects found during the peer review process. reported 15 and 11 defects for teams 1 and 2, respectively, at milestone 1. This result is to be expected since different teams artifacts can have different qualities. Also, a reviewing team found different numbers of defects for a fixed team at different milestones. For instance, team 3 reported 15 and 2 defects for team 1 at milestones 1 and 2, respectively. This result is both interesting and positive as it potentially shows the usefulness of peer reviewing. By receiving constructive feedback in the form of the list of defects at milestone 1, team 1 members seem to have worked hard to increase the quality of their deliverables at milestone 2. As the average values show, the numbers of defects generally have a decreasing trend when moving from milestone 1 to 2 to 3. Some of the milestone 3 documents were even judged to be perfect (0 defects) by some reviewing teams. This trend strongly suggests that peer reviewing made an improvement to the quality of projects. This trend also acts as evidence that once again confirms the fundamental ripple effect of defects propagation in engineering design, which has been described, for example, in software engineering [35]. If defects are detected and fixed in the early design life cycle, such as in analysis, fewer defects will be propagated to the later stages. Removing one single defect in a high-level analysis software model could perhaps be said to have saved many hours of a student s time in this course by obviating the need to detect and remove many more propagated defects associated with that original defect in the source code, in later stages of the design phase, and in the testing phase. B. Accuracy and Quality of Peer Reviews If instructors choose to incorporate peer review in engineering courses, they should ensure students perform accurate and high-quality peer reviews. To control and measure the above aspects (RQ 2), all peer-review reports were reviewed by the instructor and analyzed for two types of errors: False-Positive (FP): Defects (issues) reported by the reviewer teams, which were identified as nondefects by the instructor. A reported defect might be that the use-case Search Flight should include the Login use-case, which means that the users who want to search for flights should login first; this condition is not a correct assumption as per the system requirements. False-Negative (FN): Issues missed (not reported) by the reviewer teams, such as the use-case Search Flight being missing from the analysis model of the system. The instructor provided guidelines to the students indicating that they should report the issues accurately and reminded the students that a good peer review should have as few FP or FN errors as possible, preferably none at all. After each of the three peer review stages (analysis, design, and implementation), the instructor discussed in class the types of common FP or FN defects observed in the review reports, and students were asked to pay special attention to such issues in the next milestones. Fig. 3 shows the total number of FP and FN errors versus all defects reported during the peer-review process. For brevity, all the metrics from both review reports submitted by each team are added to a single number. So team 1, who reviewed team 4 s and 5 s projects, reported 15 defects in milestone 1. There were no FP errors in their reports, while there were three FN errors; team 1 had missed reporting three defects upon examination by the instructor. This particular team reported only two defects in the last milestone, one of which was an FP error that is to say, not actually a defect in the code developed by team 4, but a misunderstanding by team 1. On the other hand, team 1 was not able to catch (and report) an actual defect in the code of team 4 in milestone 3, thus making an FN error. The number of real defects each team was supposed to detect at the end of each milestone, had they done the most careful review, is equal to number of reported defects number of FP errors number of FN errors (reminder: FP and FN errors were judged by the instructor). For example, for team 1 in milestone 1, this target was defects. Again, there is a positive sign in that the numbers of FP and FN errors have a slightly decreasing trend. This result can be explained as the instructor discussed in class the types of common FP or FN defects observed in each round of the review process. It should be noted that since the FP and FN errors were judged based upon the instructor s knowledge in the subject matter, their numbers can be subjective. However, the instructor is considered an expert educator and researcher in the area. At the end, there were 311 reported defects in peer-review reports. Eighty-seven of these were FP errors (a ratio of 27.97%), and student teams missed a total of 95 (FN errors) of all the defects present in all project artifacts. However, since there were

7 188 IEEE TRANSACTIONS ON EDUCATION, VOL. 53, NO. 2, MAY 2010 Fig. 4. Comparing the number of defects reported, FP and FN defects in the two course offerings. two teams reviewing a given artifact document, by considering the number of common unreported (missed) defects, the number of unreported defects for all projects dropped to 47, 14.73% of all 319 actual defects. In other words, students were able to catch about 85.27% of all defects present in all project artifacts as detected by the instructor. There may be many reasons for defects going unreported (FN errors), among them perhaps that the subject matter of the course was challenging (even deemed by some colleagues of the instructor to be more suitable for graduate students rather than for undergraduates) and that students either did not have much background in or skills for peer reviewing before taking this course. C. Comparing the Results of the Two Course Offerings RQ 3 asks whether the success of peer-review application is predictable. To analyze this, the same peer-review setting was incorporated in two offerings of the same course (Winter 2007 and 2008). The success of the two course offerings can be compared by a closer look at groups 1 5 (the Winter 2008 offering) and groups 6 12 (the Winter 2007 offering) in Figs. 2 and 3. According to Fig. 2, similar trends were observed for both offerings in terms of number of defects reported by peer reviews in each milestone. The trend was also decreasing from milestone 1 to 2 and then 3 in both offerings. According to Fig. 3, the number of FP and FN errors in milestones 1 and 3 in the Winter 2008 course offering was slightly less than those in the Winter 2007 class. This trend was also observed in milestone 2. Quantitatively speaking, there were 19 FP and 31 FN errors altogether in the three milestones in the Winter 2008 peer-review reports, while those numbers were 68 and 64 for Winter 2007, respectively. Assuming that the student population was at the same level in terms of their average background of the subject matter, the instructor attributes this decrease to the fact that the instructor placed more emphasis on familiarizing students with common types of FP and FN errors in the class before each peer-review round. The instructor would recommend other instructors hold similar class discussions about common mistakes in reporting defects. For a more detailed analysis, the average number of defects reported and FP and FN errors in the two course offerings per project milestone are compared in Fig. 4. Encouragingly, the trends are decreasing. Another positive sign is that the metrics for the Winter 2008 class are all lower than those for Winter 2007, except for the number of reported defects in the review reports in milestones 1 and 3. The root-cause analysis of this observation for milestone 1 (project analysis phase) led to the finding that the requirements specification of the project was intentionally made slightly broader in the Winter 2008 class, which thus perhaps led to more defects being reported in that analysis phase. One possible root-cause of the above observation for milestone 3 (coding phase) was that the Web and application server on which the students were working in the milestone to develop and deploy their systems suffered from an outage and poor performance in the last few days before the deadline. Students were thus not able to develop their codes to the high quality they had intended. This technical problem led to their having slightly more defects in their code, as identified by their peers, compared to the 2007 class. D. Questionnaire Results To answer RQs 4 6, similar to those found in a few other empirical studies on peer reviews (e.g., [23]), the author prepared a questionnaire to quantitatively measure students success, perception, and satisfaction with the peer-review process. The questionnaire questions and its statistical results are shown in Table III. The questionnaire format was adapted from two questionnaires by Tyran [23] and Wiegers [36]. Questions 1 and 2 for the questionnaire (Table III) evaluated students confidence in conducting a successful peer review before and after the exercise. Questions 3 and 4 evaluated the perception of knowledge gains. Questions 5 8 evaluated students satisfaction with the process. The 7-point Likert scale shown in Table IV was used for all eight questions. The questionnaire was distributed to the students in the class after the project had finished, with all three milestones being completed. Students participation was voluntary, anonymous, and confidential. Twenty-one (out of 23) students filled out and returned the questionnaire. Most students also provided written comments on the questionnaire forms, which will be discussed in Section IV-E. The number of respondents (students) of this questionnaire was low (21), and it is thus hard to draw highly confident statistical conclusions. However, the results analysis of this questionnaire can provide insights and how-to s for larger future experiments. To complement the following statistical analysis, a qualitative interpretive analysis is also presented in Section IV-E to answer RQs 4 6 based on students comments. The average and standard deviations (STD) of the questionnaire results are reported in Table III. The histogram of questionnaire results is also visualized as a stacked-column chart in Fig. 5. The main interesting observations based on the histogram values follow. Out of 168 question answers (21 students * 8 questions), there was no strongly disagree answer for any question. A primary reason for doing the walkthrough exercise was to help students learn more about the peer-review process. Ideally, as an outcome of doing the exercise, the author wanted the students to build their confidence regarding the peer-review process. It appears that the exercise was effective in this regard. More precisely, 14 of 21 (66.7%) and

8 GAROUSI: APPLYING PEER REVIEWS IN SOFTWARE ENGINEERING EDUCATION 189 Fig. 5. Histogram of questionnaire results. TABLE III PERCEPTION AND SATISFACTION QUESTIONNAIRE AND ITS STATISTICAL DATA TABLE IV THE 7-POINT LIKERT SCALE USED IN THE QUESTIONNAIRE 20 of 21 (95.2%) students agreed (at least somewhat) that they had enough confidence in conducting successful peer reviews before and after the exercise, respectively. Hopefully, this increased level of confidence will enable students to conduct effective software peer review in their future careers, as it is the norm in the industry. In addition to helping students learn more about the peerreview process, the author hoped that students would gain other educational benefits. Several published articles have indicated that the peer-review process may offer educational benefits for the participants, such as imparting improved analytical abilities and defect detection abilities [18], [19]. The answers to questions 3 and 4 indicate a noticeable knowledge gain in preparing high-quality design documents and being more effective in identifying defects, as about 95.2% and 90.5% of students answered at least somewhat agree to questions 3 and 4, respectively. The answers to question 5 indicate that only 2 out of 21 students (9.5%) were not satisfied with the process (the number of somewhat disagree answers). According to the answers to question 6, 4 out of 21 students (23.8%) thought performing the exercise was not a very pleasant experience. This issue will be addressed in the discussions on lessons learned and recommendations (Section IV-G). One and two students answered neither for questions 7 and 8, respectively, while other students have positive answers on those questions. The questionnaire results are compared next with another empirical study by Tyran [23] on the use of peer review in classroom settings. Note that only questions 1 6, which were adapted from Tyran s work [23], are compared. The comparison of average values and the STDs of questions 1 6 in Tyran s study [23] versus this work are shown in Fig. 6. The main interesting observations based on this comparison are as follows.

9 190 IEEE TRANSACTIONS ON EDUCATION, VOL. 53, NO. 2, MAY 2010 Fig. 6. Comparison of the questionnaire results with that of Tyran s [23]. Both average and STD values have similar trends in both studies. By comparing answers to questions 1 and 2, it can be seen that students confidence in conducting a successful peer review has increased in both studies, as would be expected. The average pre-exercise confidence in the study (question 1) was slightly more than in Tyran s case [23]. This higher level was perhaps due to the fact that the students in this course had prior knowledge of software peer reviews, possibly learned in their industrial internships. Both studies show similar trends in students perceptions of knowledge gains (questions 3 and 4). However, in terms of students satisfaction with the exercise (questions 5 and 6), Tyran s setting [23] offered better perceptions. This result can be explained by Tyran having defined a semiformal peer-review process and a detailed potential defect checklist for his students, while in the current study, the students had the freedom to choose their own defect checklist and degree of formality in peer reviews. This issue will be revisited in the lessons learned and recommendations (Section IV-G). E. Comments by Students and Qualitative Interpretive Analysis Most students provided written comments on the peer-review process in their milestone reports and also in the questionnaire forms. A selected list of those comments follows, with similar comments being merged, and analysis being provided as necessary. In general, we found the reviews very beneficial and useful. It turns out that this [providing peer reviews] is actually a hard document for me to write, I am not very experience[d in] giving critiques. However, since I am forced to I will try to write it as I would like to receive it. Analysis: The student finds it hard to write reviews, and there is some discomfort ( I am forced ) in providing constructive feedback on other people s work. Perhaps undergraduate students have not learned during their education to criticize the product (artifact) and not the people. The student mentions, I would like to receive it [peer review of my work], so (s)he tries to provide a review in turn. This comment is a positive sign as one can see that students like to receive peer review and for this will make the effort to perform peer review. However, during the training process, educators should make the effort to teach students not to be afraid of reviewing other people s work products. These changes were considered but not made due to time constraints. Analysis: This comment shows that this team considered the reviews but did not have time to implement the changes. Of course, although it is positive to see that students were open to review comments, they did not practice the skill of making changes based on reviews. The instructor had frequently reminded the teams to allocate enough time to different phases of the review process, but still some teams did not manage to perform all scheduled tasks. In the assessment of projects, a small portion of marks were deducted for each such shortcoming in the peer-review process. In general, we found the reviews fewer in number compared to the milestone 1 peer reviews however we still found them to be very beneficial and useful. Analysis: It is interesting to see that this team had the impression (in just milestone 2) that the number of reported defects has decreased compared to milestone 1. This decrease corresponds with the finding at the end of the experiment, as discussed in Section IV-A. Peer reviews are generally extremely useful. I fully support the idea [peer reviewing] and feel that it is a valuable exercise in a classroom setting. The peer reviews made it easy to correct my mistakes because I could evaluate what solution was best and change my solution so, in a sense, peer reviews made it easy to copy. Analysis: The student first expresses that peer reviews are useful since (s)he can learn from other teams work. This is also in accordance with the recommendation suggested in [21] that students should review the work of teams from prior semesters before embarking on their own projects. In doing so, they can learn from the mistakes of others and practice continuous improvement. However, then the student raises the concern that such a learning experience can lead to copying (plagiarism) as there is a fine line between learning from others and plagiarism. This comment and the possibility of plagiarism in the experiment are discussed in detail in Section IV-F. A qualitative interpretive analysis can be performed based on the above student comments to qualitatively address RQs 4 6. RQ 4: Students did not provide explicit written comments in the questionnaire about whether their confidence in performing effective peer reviews increased after the exercise. However, their positive comments imply that they have gained new expertise, and their confidence should have thus increased. To gather their explicit opinions about this question, the instructor invited the students by to send their anonymous written feedbacks on this aspect a few months after the course had finished. Two students, who were both working in the software industry at the time, responded to the invitation; they both responded that

10 GAROUSI: APPLYING PEER REVIEWS IN SOFTWARE ENGINEERING EDUCATION 191 their confidence in performing effective peer reviews significantly increased after the exercise. One of them mentioned that (s)he is actually involved in peer-review activities at the time of this writing. RQ 5: The comments that the students made in favor of peer reviews denote that they have learned new skills from peer reviewing. RQ 6: Again, the positive comments imply that students were satisfied with the peer review process. F. Costs, Challenges, and Drawbacks So far, only the advantages of peer reviewing have been discussed. The associated costs and potential challenges or drawbacks are now discussed, all of which need to be taken into account if an instructor considers using peer review in her/his course. Cost and Overhead Factor: One immediate concern might be the overhead time needed to train students for peer review. While this is a valid point, after graduation, students will be entering a work environment that extensively follows peer-review practices. Thus, it makes perfect sense to invest the overhead cost to train them and ask them to practice peer reviewing. The feedback the author has received so far from a few of the students in this experiment who are now working in the software industry acknowledges that they are now very happy to have practiced peer reviewing during the course and they are now better at it. Interpersonal Challenges: Recall one of the student comments from Section IV-E: It turns out that this [providing peer reviews] is actually a hard document for me to write,. Thus, it seems that this student feels it is hard to provide a feedback for his/her peers as perhaps (s)he is thinking this can lead to interpersonal and interfriendship challenges. However, again in the context of students getting ready for the work place, it is necessary to control such uncomfortable feelings and to train students so they are willing to provide constructive feedback, even to their peers. Possibility of Plagiarism: Recall one of the student comments from Section IV-E: The peer reviews made it easy to correct my mistakes because I could evaluate what solution was best and change my solution so, in a sense, peer reviews made it easy to copy. Thus, the possibility of plagiarism might be perceived to be a concern. Due to the nature of this Component-Based Software Engineering project, however, the different elements of the analysis and the design documents were so interconnected that copying only one part of the diagram could not help a team take advantage of another team s work. Also, as senior fourth-year students, some level of trust had been built between the instructor and students. Careful inspections by the instructor did not reveal any major plagiarism issue in the team projects. Having faced a similar challenge when performing a controlled experiment [24] using more than 100 undergraduate students in the University of Bari, Bari, Italy, Bianchi et al. implemented the following approach: Subjects always worked in two big rooms with enough space to avoid plagiarism and confusion. We were always present to answer questions and preventing unwanted communication [24]. The same approach could not be applied in the current study since the course project was not confined to the weekly lab hours. G. Lessons Learned and Recommendations As discussed in Section III-D, a less formal peer-review process was intentionally applied in this project. As Sullivan discussed in [22], for students, the prospect of maintaining inspection statistics can be so intimidating as to prevent their mastering of the peer-review basics. Thus, Sullivan [22] believes that walkthroughs can be a less complex activity. For this reason, the walkthrough paradigm was followed in the course. Also, a suitable tradeoff analysis should be used by each instructor to determine the choice of formality level for the peerreview process. If there is enough time in the beginning of a course to briefly and formally train students on peer review, it will certainly lead to better outcomes and satisfaction in the end. However, as there is always more material than a course has time to cover, it remains a challenge to find the best balance between formal peer-review training and having students learn through the hands-on experience of peer review in the project. It will be a great benefit to education in general if the researchers in the education community can provide more substantiated evidence that student peer reviews can be an adequate substitute for instructor reviewing. As discussed next, the current work is a partial initial evidence in that direction. Recall from Section IV-B that the students in this experiment were able to catch about 85.27% of all defects present in all project artifacts. Although the above aggregate accuracy measure is by itself quite promising, with more experience in orchestrating peer-review processes, by allocating more than two teams to peer review another team s artifact, and with increased use of peer-review techniques in other, previous courses, educators can hope to get better, more accurate peer-review results by students. Applying peer reviews greatly enhances the experience of students, leading to a more complete learning experience. To show that this was a homogenous task in this course, the instructor reviews were conducted in parallel with student peer reviews, and the results were compared. It was observed that the number of defects identified by the instructor that were not reported in the peer reviews was minimal. So as not to overly increase the students workloads, only two student peer reviews per artifact were conducted. The instructor reviews were conducted in all cases to confirm the measurements and also when there was a discrepancy between the different student peer reviews. The implementation of such a strategy actually increased the instructor s workload, as the subsidiary goal was to measure the effectiveness of the entire process. However, in future settings, this might alter the work of the instructor (probably with a small increase in total workload). The instructor will still need to administer the peer reviews and control the quality of the reviews rather than conducting reviews him/herself and assessing the student artifacts. On the other hand, students will be required to work much harder and will gain the great benefit of learning the competencies and skills of peer reviewing as well as having the opportunity to learn from each other.