College Students Attitudes Toward Methods of Collecting Teaching Evaluations: In-Class Versus On-Line

Transcription

1 College Students Attitudes Toward Methods of Collecting Teaching Evaluations: In-Class Versus On-Line CURT J. DOMMEYER PAUL BAUM ROBERT W. HANNA California State University, Northridge Northridge, California T he evaluation of faculty instruction is an ongoing and ubiquitous rite on college and university campuses throughout the United States and around the world. The vast majority of U.S. institutions require it in some form (Wachtel, 1994), and the amount of research on this activity is enormous (Wachtel, 1998), involving thousands of research papers (Marsh & Dunkin, 1992). According to Cashin (1988), there are probably more studies of student evaluations of faculty than the combination of all other means of evaluating faculty teaching. Although this research dates back to the late 1920s with the early work of Brandenburg and Remmers (1927), until the last couple of decades it has been largely concerned with validity and reliability issues. In a comprehensive review of research concerning student evaluation of college and university instructors, Wachtel (1998) noted that after nearly seven decades of research on the use of student evaluations of teaching effectiveness, it can safely be stated that the majority of researchers believe that student ratings are a valid, reliable, and worthwhile means of evaluating teaching (p. 2). Yet there are many others who would argue this point, as more recent research has indicated that methodological factors and situational characteristics can affect the validity of ABSTRACT. In this study, the authors investigated college students attitudes toward in-class and on-line methods of gathering teaching evaluations of faculty. Sixteen professors who taught two sections of the same class were randomly assigned to have one of their sections evaluated by the in-class method and the other by the on-line method. Students also completed a post-teaching-evaluation survey measuring their attitudes toward the method of evaluation used. The on-line method had a lower response rate than did the in-class method. Online respondents complained that the evaluation process may not be anonymous and that it was time consuming and involved complicated log-on procedures. Suggestions are offered for improving the on-line evaluation process. student evaluations. It has been shown, for example, that student evaluations may be affected by student perceptions of grade leniency (Nimmer & Stone, 1991), by instructor enthusiasm (Williams & Ceci, 1997), by student perceptions of anonymity (Blunt, 1991; Feldman, 1979), and by the early formation of student opinions about a course or instructor well before a course is underway (Hewett, Chastain, & Thurber, 1988). Students attitudes toward the traditional methodology of student evaluations have received relatively little research attention. According to Marlin (1987), student satisfaction as a factor in student ratings has been largely ignored. He noted that some students complain that faculty and administrators pay scant attention to student opinions and that teachers do not modify their behavior based on student comments on rating forms. Abbott, Wulff, Nyquist, Ropp, and Hess (1990) stressed that students willingness to participate in the evaluation method is directly related to their satisfaction with the evaluation process. To increase student participation in the evaluations of faculty members, university administrators must learn students attitudes toward current and proposed methods for gathering teaching evaluations. The wide availability of computers and the Internet, both on and off campus, has afforded students numerous learning opportunities, such as quick access to lecture material, literature databases, professors, and classmates. The wide Internet access also provides a new mode by which students may evaluate their professors namely, on-line evaluations. The traditional 1-day inclass evaluation of instructors via optically scanned forms may eventually give way to evaluation via the Internet, allowing for extended periods of student evaluation time. The on-line method promises more convenience to students September/October

2 and ease of use for administrators (Layne, DeCristoforo, & McGinty, 1999). Although electronic surveys are a fairly recent technique, they have been compared with traditional survey methods and been proven effective across a variety of organizational applications such as market research, personnel evaluations, and psychological counseling (Sproull & Kiesler, 1990; Rosenfeld, Booth-Kewley, & Edwards, 1993). Until the last few years, there has been an absence of published research comparing on-line student evaluations of faculty with traditional in-class paper-andpencil methods. A recent study comparing these two methods was conducted by Layne et al. (1999). The researchers studied both graduate and undergraduate students, the majority of whom were in engineering-related disciplines and were computer literate. They concluded that students had higher levels of satisfaction with the electronic (on-line) method of faculty evaluation over the traditional (in-class) method. Voluntary comments suggested that although the on-line method was perceived by students as having less anonymity, they saw it as less wasteful of resources (paper) and class time. An unexpected finding was that students who completed the survey on-line were more likely to provide voluntary comments than students who completed it in class. Overall, it appeared that the method of evaluation did not affect the responses given by the students. Nonetheless, the survey results revealed a lower response rate for students using the on-line method (47.8%) compared with students using the inclass method (60.6%). The student attitudes that might account for these response rate differences were not directly determined. The authors suggested that effective use of the on-line method requires greater institutional endorsements and guarantees of confidentiality. They also suggested that incentives may be necessary to achieve higher response rates. In another comparable study, Baum, Chapman, Dommeyer, and Hanna (2001) compared the in-class and online evaluation methods among business undergraduate students. They found an even larger difference in response rates 12 Journal of Education for Business between the on-line method (32.8%) and the in-class method (76.8%) and some tendency toward higher student ratings of instructors with the on-line method. In this article, we extend that study by comparing student attitudes toward the in-class and on-line evaluation methods. Objectives Our primary purpose in this study was to compare the attitudes of college students who were asked to give their teaching evaluations of their professors in class with students who were asked to submit their evaluations on-line. The two groups were compared on their impressions of the evaluation method that they were asked to use. The following questions were addressed: 1. Which method of teaching evaluation (in-class vs. on-line) achieves the higher response rate to the teaching evaluations? 2. Does the method of teaching evaluation affect the reasons that students give for evaluating or not evaluating their professor? 3. Does the method of teaching evaluation affect students overall impressions of the teaching evaluation process? 4. Does the method of teaching evaluation affect students ratings of specific dimensions of the teaching evaluation process (e.g., feelings of anonymity, ability to express true feelings, simplicity, and convenience ). A secondary purpose of this research was to determine whether either method of teaching evaluation produces a response bias that is, a situation in which the respondents and nonrespondents to a survey differ on key characteristics (Malhotra, 1999). We addressed this issue by determining whether students who responded to a specific teaching evaluation process differed significantly from the nonrespondents on the following factors: gender, expected grade in the class, and rating of the professor s teaching. Method During the spring semester of 2000 at California State University, Northridge, 16 business professors were recruited to participate in an experiment on on-line teaching evaluations. All of the professors selected for the experiment were currently teaching at least two sections of the same class during the semester. The majority were teaching two identical sections of the same class in adjacent time periods. Each of the professors was asked to alter the method of teaching evaluations in the paired sections on a random basis: One section was scheduled for in-class evaluations, whereby the students would rate the professor on forms distributed in the classroom, and the other section was asked to submit its evaluations of the professor via the Internet. Both survey methods used the same evaluation form. Students selected for the on-line evaluations were informed that they could go to a Website any time during a 2-week period to submit their evaluations. The experimental procedures were designed to minimize biases that might be attributed to the professor, the class, or the time of day. Prior to the last week of the semester, the teaching evaluations from all students, both in-class and on-line, were completed. During the last week of the semester, each of the 16 professors in the experiment was asked to distribute a post-teaching-evaluation survey to his or her in-class and on-line students to gather their impressions of the evaluation method they were asked to use. The questionnaires were distributed during class time at a meeting that most students were expected to attend. Students were told that their responses to the survey would be anonymous, that their instructor would not examine their answers, and that the completed surveys would be taken to the dean s office for analysis. The post-teaching-evaluation survey was two pages long and contained 16 questions. It began by asking whether the student had completed the earlier teaching evaluations and then asked why the student did (or did not) complete the evaluation of his or her professor. Those students who had completed the teaching evaluation were asked to indicate the method of evaluation that they used (either in-class or on-line), to explain their likes and dislikes of the method, and to rate the method they

3 used by agreeing or disagreeing with eight belief statements. All of the students, whether they completed the earlier teaching evaluations or not, were asked to indicate their gender, the grade they expected in the class as of the time of the survey, and their rating of their professor s teaching ability on a 10- point scale. Results Altogether, 1,549 students were enrolled in the classes given the postteaching-evaluation surveys. Of the group, 987 students completed the survey. However, 26 students either stated that they did their evaluations on-line when they were asked to do them in class or submitted them in class when they were asked to do them on-line. These surveys were eliminated from the analysis. Thus, usable responses were received from 961 students, resulting in a response rate of 62%. Randomization Check At the end of the post-teaching-evaluation survey, all respondents were asked to give their gender, their grade in the class at the time of the survey, and their rating of the instructor on a 10- point scale. We compared the in-class and on-line respondents to the postevaluation-survey on these variables. No significant differences were found between the two treatment groups, suggesting that the random assignment of the treatments to the 32 classes did not cause any selection bias in terms of gender, grade, or instructor evaluation. Comparing the Treatment Groups During the post-evaluation-survey, students were asked whether they had evaluated their professor s teaching performance during the previous weeks. Those who were asked to compete the teaching evaluations in class were significantly more likely to indicate that they had evaluated their professor than those who were asked to complete the evaluations on-line (92% vs. 60%, χ 2 (1) = 128, p <.001). These results are consistent with previous studies that have compared response rates of in-class and on-line teaching evaluations (Baum et al., 2001; Layne et al., 1999). When students who had provided teaching evaluations were asked why they gave them, the main reason given by both the in-class and on-line students was that they were asked or required to give the evaluations. However, students who did not submit teaching evaluations gave reasons for not providing the evaluations that were specific to the evaluation method. That is, nonrespondents to the in-class evaluations stated that their having missed class on the day of the evaluations was the main cause of their nonresponse. On-line nonrespondents, in contrast, were more likely to state that they forgot or missed the deadline or had computer problems (see Table 1). Students who completed the teaching evaluations were asked what they liked about the evaluation process; their answers are also displayed in Table 1. On-line respondents spoke mainly of the convenience, speed, or ease of the method, whereas in-class respondents listed those features as well as TABLE 1. Reasons Given for Not Evaluating the Professor, for Having Liked the Evaluation Method, and for Not Having Liked It, in Percentages of Respondents Respondent category Student response In-class On-line Reasons for not evaluating professor Forgot or missed deadline 67 Absent from class 64 6 Computer problems 13 Inconvenient 7 Do not care about evaluations 4 2 Miscellaneous 32 4 Total Reasons students liked evaluation method Convenient 7 39 Fast Easy Anonymous response 17 3 Had space for free response comments 14 2 Well designed questions 15 2 Professor left the room 13 Forces students to complete the evaluations 3 Uses up class time 3 Did not waste class time 5 Had plenty of time for the evaluation 1 Total 101 a 101 a Reasons students disliked evaluation method Answers may not be anonymous 9 24 Inconvenient 6 21 Questions were too broad 42 6 Too many questions 4 1 Other students can see your answers 2 1 Wasted valuable class time 27 Not enough time to complete survey 11 Took too much time 28 Complicated log-on process 12 Computer problems 8 Total 101 a 101 a Note. For reasons for not evaluating professor, in-class n = 25 and on-line n = 184. For reasons students liked evaluation method, in-class n = 101 and on-line n = 152. For reasons students disliked evaluation method, in-class n = 55 and on-line n = 91. a Percentages do not sum to 100 because of rounding errors. September/October

4 several others, namely anonymous response, had space for free response comments, well designed questions, and professor left the room. When students were asked to list what they disliked about their method of teaching evaluation, in-class respondents mainly mentioned that the questions were too broad, that valuable class time was wasted, and that they had insufficient time to complete the survey. On-line respondents complained primarily about the process taking too much time, that response to the survey may not be anonymous, that the evaluation process was inconvenient, and that the log-on process was complicated (see Table 1). Students who evaluated their professor were asked to use a 5-point Likert scale to rate the evaluation process on several specific dimensions (see Table TABLE 2. Respondent Attitudes Toward Method of Teaching Evaluation Average scores Statement a In-class On-line Dif. b 1. I felt relaxed when completing the evaluation of my professor When giving my evaluation, I felt free to express my true feelings I felt there was no way my professor could identify my evaluation with me * 4. I felt there was no way my instructor could determine if I had done or not done the evaluation * 5. It was very simple for me to provide my evaluation of my professor The process used to collect my evaluation was complicated ** 7. It was convenient for me to give my evaluation of my professor Giving my evaluation wasted good class time * Note. Scores are based on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Average scores are reported. a The statements above have been placed in logical groupings for presentation purposes. They were presented in a different order on the survey instrument. b We used a t test to compare the means. Difference scores are not shown unless they are statistically significant at or below the.10 alpha level. *p <.01.**p <.05. TABLE 3. Examining Nonresponse Bias for Each Method Provided Provided in-class on-line evaluations Test evaluations Factor Yes No statistic a Yes No Test statistic a Female 94% 6% 57% 43% Male 91% 9% χ 2 (1) = % 36% χ 2 (1) = 2.2 Expected grade b t(407) = t(496) =.09 Professor s teaching performance c t(410) = t(504) = 1.2 a None of the test statistics was significant at or below the.10 alpha level. b Averages based on a 4- point grade scale. c Averages based on a 10-point scale. 2). The results indicate that both treatment groups felt relaxed and free to express their true feelings when giving the evaluations (items 1 and 2). However, the in-class respondents were more likely to believe that their response to the survey was anonymous. Though both groups indicated that the teaching evaluation process was simple and uncomplicated, the in-class respondents were slightly more likely than the online ones to disagree that the process was complicated. Both groups agreed that the evaluation process that they used was convenient. Finally, the inclass respondents were more likely to believe that the teaching evaluations wasted good class time. Examining Each Evaluation Method s Nonresponse Bias We used a 10% significance level when examining nonresponse bias. For each method of teaching evaluation, no statistically significant differences were found between the respondents and nonrespondents on gender, expected grade in the class, or the rating of the professor s teaching performance (see Table 3). Thus, there is no evidence that either of the teaching evaluation methods incurred any significant nonresponse bias. Discussion and Conclusions Collecting teaching evaluations via the Internet clearly has advantages over the traditional, in-class method of data collection: On-line data collection eliminates paper costs; requires less class time; permits efficient processing of the data; is less vulnerable to influence of the faculty, and can be a fast, easy, and convenient method for students to submit their evaluations. Despite these advantages, the on-line method will not be widely adopted by universities if it cannot overcome some serious hurdles. It was shown in this survey and others that the on-line method produces a lower response rate to the teaching evaluations than the traditional paper-andpencil method. The lower response rate is most likely related to factors such as fear that responses to the on-line survey may not be anonymous and that the on- 14 Journal of Education for Business

5 line method can be inconvenient, timeconsuming, and prone to technical problems. Moreover, as shown in Table 1, the on-line survey may be the victim of student apathy, as the majority of the nonrespondents to the on-line survey forgot or missed the deadline date. Many of the impediments to on-line response can be effectively dealt with over time. As students become more computer literate and have greater access to computers, it will be less likely that inconvenience and technical problems will inhibit survey response. There are a number of ways that professors may be able to enhance survey response. First, they can provide a live demonstration of how to submit an on-line response, to reduce any computer-related fears and reinforce the importance of responding. Second, if a professor has the available class time and access to a bank of personal computers, he or she could use class time to direct students to a computer laboratory where they could submit their evaluations. However, this remedy does defeat some of the advantages of the on-line method. Third, to ensure that students do not forget or miss an on-line survey deadline date, at the end of each class professors could continually remind students of the deadline date and of the importance of the evaluations. Finally, professors might be able to use either positive or negative incentives to increase on-line response rates. For example, students could be told that if they bring the professor a validation slip indicating that they have submitted an evaluation on-line, they will be given a reward such as extra credit or an early notification of their course grade. Perhaps the greatest obstacle for the on-line method is how to deal with the students fear that their response to the on-line survey may not be anonymous. Much of this fear no doubt stems from the fact that students must submit their student identification number when accessing the on-line survey. Although students are told that their identification number will never be associated with their response, they may not believe it. It may take some time before all students trust the integrity of the on-line survey procedures. It may, in the future, be possible for students to submit their on-line evaluation without having to submit any identifying information. This could prove to be a tricky procedure, however, because there must be a way to keep track of each student s submission if students are to prove to their professor that they provided a response. Moreover, the data collection system must have a way to block on-line evaluations from people who are not in the professor s class as well as to prevent multiple evaluations from the same student. Though there was no evidence of nonresponse bias in either the in-class or online teaching evaluations, one should note that we assessed nonresponse bias by examining only three variables in the post-evaluation-survey. Had more variables been involved in this analysis, it is possible that other conclusions could have been drawn. It is also important to note that the post-evaluation-survey may have suffered from its own form of nonresponse bias, because valid surveys were received from only 62% of the students enrolled in the classes. Had more students responded to the post-evaluation-survey, the results may have been slightly different. REFERENCES Abbott, R. D., Wulff, D. H., Nyquist, J. D., Ropp, V. A., & Hess, C. W. (1990). Satisfaction with processes of collecting student opinions about instruction: The student perspective. Journal of Educational Psychology, 82, Baum, P., Chapman, K., Dommeyer, C., & Hanna, R. (2001). On-line versus in-class student evaluations of faculty. Paper presented at the Hawaii Conference on Business, Honolulu, June, Blunt, A. (1991). The effects of anonymity and manipulated grades on student ratings of instructors. Community College Review, 18, Brandenburg, G. C., & Remmers, H. H. (1927). A rating scale for instructors. Educational Administration and Supervision, 13, Cashin, W. E. (1988). Student rating of teaching: A summary of the research. Manhattan, KS: Center for Faculty Evaluation and Development, Kansas State University. Feldman, K. A. (1979). The significance of circumstances for college students ratings of their teachers and courses. Research in Higher Education, 10, Hewett, L., Chastain, G., & Thurber, S. (1988). Course evaluations: Are student ratings dictated by first impressions? Paper presented at the Annual Meeting of the Rocky Mountain Psychological Association, Snowbird, Utah. Layne, B. H., DeCristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40(2), Malhotra, N. K. (1999). Marketing research: An applied orientation (3rd ed.). Upper Saddle River, NJ: Prentice Hall. Marlin, J. W. (1987). Student perception of endof-course evaluations. Journal of Higher Education, 58, Marsh, H. W., & Dunkin, M. J. (1992). Students evaluations of university teaching: A multidimensional perspective. In J. C. Smart (ed.), Higher education: Handbook of theory and research, 8, pp New York: Agathon Press. Nimmer, J. G., & Stone, E. F. (1991). Effects of grading practices and time of rating on student ratings of faculty performance and student learning. Research in Higher Education, 32, Rosenfeld, P., Booth-Kewley, S., & Edwards, J. E. (1993). Computer-administered surveys in organizational settings: Alternatives, advantages, and applications. American Behavioral Scientist, 36(4), Sproull, L., & Kiesler, S. (1991). Connections: New ways of working in the networked organization. Cambridge, MA: MIT Press. Wachtel, H. K. (1994). A critique of existing practices for evaluating mathematics instruction (Doctoral dissertation, University of Illinois at Chicago, 1994). Dissertation Abstracts International, 56, no. 01A, p Wachtel, H. K. (1998). Student evaluation of college teaching effectiveness: A brief review. Assessment and Evaluation in Higher Education, 23(2), Williams, W. M., & Ceci, S. J. (1997). How m I doing? Problems with student ratings of instructors and courses. Change, 29(5), September/October

6