Are they the same? Comparing the instructional quality of online and faceto-face graduate education courses

Transcription

1 Assessment & Evaluation in Higher Education Vol. 32, No. 6, December 2007, pp Are they the same? Comparing the instructional quality of online and faceto-face graduate education courses Andrew Topper* GVSU College of Education, USA CAEH_A_ sgm / Assessment Original 2007 Taylor December AndrewTopper and & Article Francis (print)/ x & Evaluation 2007 Ltd Higher (online) Education As instructors adopt web-based learning environments, they must consider how students evaluations of learning reflect the overall quality of instruction. Traditional course evaluations are used for faculty retention, tenure and promotion decisions, but also provide instructors with valuable information on the quality of their instruction. This study looks at response rates and compares instructional quality, using student course evaluations along with additional data from online and face-to-face graduate education courses, to evaluate the effectiveness of instruction. A statistical analysis of students course evaluations showed no significant difference in instructional quality based on the format used. Together with comparisons of student work, these results provide additional evidence in support of the finding of no significant difference between formats in the area of instructional quality. Introduction The federal government is calling for more use of e-learning in the preparation and ongoing professional development of classroom teachers (Kleiman, 2005). As more opportunities for teacher learning are provided in web-based and online environments, it is important to ensure that their instructional quality is comparable with traditional, face-to-face (F2F) offerings. Student evaluations of college courses are often used in combination with other scholarly products to determine faculty promotion and tenure. But course evaluations have another important purpose: to measure the effectiveness or quality of instruction from the students perspective. For the purposes of this study, instructional quality is defined as a set of factors which, when taken together, combine to represent the overall quality of teaching *GVSU College of Education, 368C DeVos Hall, 401 W. Fulton, Grand Rapids, MI 49540, USA. toppera@gvsu.edu (website: ISSN (print)/issn X (online)/07/ Taylor & Francis DOI: /

2 682 A. Topper effectiveness as reported by the students in a college class (Jackson et al., 1999). This concept measures students overall satisfaction with the course and instructor as well as their perceptions of what was learned, and is influenced by the class size, instructor responsiveness, quality of the lectures and other learning activities used. While the use of course evaluations has been shown to be effective, the movement to online and web-based learning environments presents new challenges and opportunities, and raises several important questions: (1) How do web-based course evaluations compare with paper-based versions? (2) How do course evaluations from online or web-based courses compare with traditional, face-to-face courses? (3) What other products can be used to evaluate instructional quality? The first part of this study explores how comparable student course evaluation response rates are when submitted online vs. via paper, while the second part explores how instructional quality, as measured by course evaluations, compares for online and face-to-face courses. Background Students evaluations of undergraduate and graduate courses often directly influence tenure and promotion decisions in colleges and universities. The movement to online and web-based learning environments represents a change in the way these course evaluations are gathered, but also an opportunity to analyze instructional quality. One issue that has been identified in the literature with regard to the growing use of electronic course evaluations is low response rates. Moss and Hendry (2002) reported an average response rate of 64% for students in a medical program and recommend use of an reminder to encourage participation. Dommeyer et al. (2004) suggest that a mild grade incentive can be used to achieve response rates that are similar to paper-based course evaluations. Watt et al. (2002) suggest that electronic course evaluations, when used with reminders, result in similar response rates to paper evaluations. In another study done by Layne et al. (1999), evidence was reported that student course ratings were not influenced by mode and that response distributions did not vary according to the mode used, electronic or paper-based. Moving beyond the issue of course evaluation response rates, a variety of published studies have found that there is no significant difference between F2F and online courses, in terms of grades, students perceptions of learning, instructional quality or student satisfaction (Merisotis & Phipps, 1999; Phipps & Merisotis, 1999; Tucker, 2000). For example, Skylar et al. (2005) reported that there were no significant differences between students achievement, their satisfaction and course evaluation results for special education pre-service teachers. Purcel and Stertz (2005) found no statistically significant difference in student satisfaction or performance for in-service teachers. However, there is also counter-evidence in the literature that should not be ignored. For example, Buckley (2003) reported that students in a web-based undergraduate

3 Instructional quality of online and face-to-face graduate courses 683 nursing course gave it a significantly lower score than a web-enhanced course. The author did report that qualitative comments from students on the web-based course were mostly positive but this does raise questions regarding the differences students report between in-class (F2F) and online or web-based courses. It should be recognized, though, that these studies use different characteristics when comparing media types and, in some cases, different methods of evaluating instructional quality and student satisfaction, which may not always be valid measures of overall learning. What is lacking from these studies is a clear sense of how different F2F courses are from online courses, as measured by student course evaluations, for a specific audience and with a specific instructional goal in mind. Research design This study used quantitative data, from a variety of sources, to measure the differences between traditional, face-to-face (web-enhanced or blended) and fully online graduate education courses. The data consisted of student course evaluations, student work products and survey responses from selected online courses. The current study looked at over two years worth of data from traditional and online graduate courses using print and web-based student evaluations to explore the question of how these might be used to compare instructional effectiveness. The courses are part of a M.Ed. degree program at a US regional institution with a focus on integrating educational technology into K 12 teaching. Students chose the course section and format (online vs. face-to-face) when they enrolled and were not required to take any courses online. Graduate course numbers and titles used are as follows: EDG 619 Classroom Uses for the Microcomputer; EDG 620 Courseware Development; EDG 621 Topics in Educational Technology. Table 1 represents sample sizes and response rates for the course evaluation data used in this study, organized by course and term. As the table reflects, course evaluation data collected over the two-year period do not include all course sections for each term. For some terms, there were no course evaluations available and all three courses were not offered every term. Response rates Response rates have been identified in the literature as a possible concern when looking at web-based or electronic evaluations. In this study, response rates for web-based course evaluations were reasonably high (overall) and compared favorably with paper-based evaluations from other sections (see Table 1). While we should ultimately strive to reach a 100% response rate, we need to recognize that this may be impossible, and we may have to use incentives and reminders to improve the overall response rate for online or web-based course evaluations.

4 684 A. Topper Table 1. Sample sizes, response rates for course evaluation data Course Term Medium used Enrollment Course evaluations Response rate (%) EDG619 Fall 2002 Online Spring/Summer 2003 Face-to-Face Ditto Face-to-Face Fall 2004 Online Spring/Summer 2005 Face-to-Face EDG620 Fall 2002 Face-to-Face Winter 2003 Online Winter 2004 Online Fall 2004 Face-to-Face Winter 2005 Online EDG621 Winter 2003 Face-to-Face Spring/Summer 2003 Online Ditto Online Winter 2004 Face-to-Face Spring/Summer 2004 Online Winter 2005 Face-to-Face Spring/Summer 2005 Online Course evaluations The literature suggests that there may be no difference between F2F and online courses, in terms of instructional quality and student satisfaction as measured by student course evaluations (Spooner et al., 1999; Topper, 2005). In order to compare course evaluation data between F2F and online graduate education courses, the items listed in Table 2 were selected from those used in a university course evaluation (see Appendix A). These items reflect a set of conceptual factors that relate to the instructor and his/her methods of instruction, which are a subset of the factors in the larger university course evaluation. Students rate these items for each course using a fivepoint Likert scale, where 5 represents Excellent, 4 Good, 3 Average, 2 Below Average, and 1 Poor. Table 2. Course evaluation items (CEI) related to instructional quality 1. Course objectives were clearly presented 2. Instructor set realistically high standards for students 3. Instructor communicated effectively with students 4. Instructor objectives were reflected in his teaching 5. Instructor was sensitive to students needs 6. Instructor was committed to developing academic excellence in students

5 Instructional quality of online and face-to-face graduate courses 685 A Pearson chi-squared test was initially run on CEI data from multiple sections of each of the three graduate courses, but the expected count on all three was less than required, which violates one of the assumptions when using this statistical approach. As a result, the data were collapsed to create a two-by-two table with a comparison done between the format (F2F vs. Online) and those items rated by students as either Excellent or Good (4 or 5) as compared with Average, Below Average or Poor (1, 2 or 3). Grouping the data in this way allowed for use of a Fisher s exact test to compare the formats and made sense, given that the students ratings in the Average, Below Average or Poor scales were relatively small in number. This was viewed as a conservative way to combine the course evaluations, where the groups were divided between Good or Excellent and Average or lower, without jeopardizing the validity of the data or the intent to explore statistically significant differences across course formats. Table 3 summarizes the numerical data for all three courses using the six instructor-specific course evaluation items from Table 2. Sample sizes for Table 2 vary from Table 1 because some evaluation data were missing (e.g. n for EDG 619 = 74 vs. 75). There is no way to know why students may have responded the way they did on the course evaluations, and this is simply a limitation of this and similar studies. Table 3. Fisher s exact test scores and sample sizes for CEI data Course evaluation item Course n Fisher s exact test (2-sided) 1. Clear objectives EDG EDG EDG Realistically high standards EDG EDG EDG Communicated with students EDG EDG EDG Objectives reflected in teaching EDG EDG EDG Sensitive to student needs EDG EDG EDG Committed to academic excellence EDG EDG *EDG Notes: 1 For items 4 and 6 in EDG 621, all of the ratings were in the 4 or 5 range, with none in 1 3 for either of these items. For both of the formats, online and F2F, the resulting data showed equal distribution and a 1:1 correspondence across formats. P-values reported in Table 3 reflect how likely it is that one would see the observed outcome given that the null hypothesis is true or purely by chance alone. An observed difference of 0 reflects a very unlikely result due to chance, whereas a larger p-value, e.g , represents that it is likely there was no observed difference in how the groups responded to that item.

6 686 A. Topper The single course evaluation item that did fall below the 0.05 significance level was No. 6 for EDG 620, which was As is clear in Table 3, all of the other course evaluation items for all three of the graduate classes did not differ statistically significantly across the two formats: F2F and online. This suggests that, in terms of quantitative data, no statistical difference was found based on media used for instruction F2F vs. online for these courses. These results are consistent with other accounts published in the literature, adding to the growing evidence that courses are similar in quality across formats. Since this study was not set up as an experiment, and the data are based on students who self-selected into a particular class or format, there could be an existing difference that may have biased the results. Based on the design of the study, we are not attempting to provide evidence of cause effect relationships, but rather examining what was observed in the data gathered. There could certainly be alternative explanations for the observations beyond group membership. Given the relatively small sample sizes, and the fact that there could in fact be cohort differences across sections, it is important to recognize these limitations in the results. Given that students selected their format of choice, this may raise validity questions regarding this study and possibly have skewed the results observed. But are these true measures of instructional quality when assessed by student evaluations? If not, what other indicators might paint a more complete portrait of overall instructional quality? One problem with course evaluations is that they tend to be numeric, general and instructor-specific i.e. instructor set realistically high standards for students, instructor served as a role model for learning, etc. How helpful can these items be when looking at comparing instruction? Given the implicit weight the university places on student course evaluations, in decisions concerning promotion and tenure, it seems logical that they would have similar weight when comparing overall course quality. However, it is helpful to look at the work products created by students in these courses to see how they compare across formats. Doing so provides a more complete picture of students performance in each course. Common assessments of student work products Student work products, which might include common NCATE assessments such as journal entries, software or website evaluations, plans for integrating technology into the curriculum, and other technology-focused pedagogical projects can also be compared for quality across course formats. Possible quality factors might include completeness, evidence of professional growth, analytical thinking, and consideration of the inherent complexity associated with technology use in classroom settings. In this case, the instructors institution was reviewed for accreditation early in 2006, so they had recently instituted a series of common assessments and rubrics for students work products as part of an overall performance evaluation program. NCATE and other accrediting institutions provide an incentive for gathering, tracking and analyzing student performance data and also a focus on improving instruction

7 Instructional quality of online and face-to-face graduate courses 687 in both online and traditional class environments. Table 4 represents a comparison between two F2F and two online sections of a course (EDG 619) for student products, in this case curricular integration projects (CIP). Table 5 represents similar data from a common assignment in EDG 621 an analysis of a school district technology adoption plan. What is interesting about these data is how instructors assessments of student work products compare across the different formats. In Table 4, it seems that students in the F2F course generated better quality products (87% vs. 75%) but in Table 5, the opposite occurred (79% vs. 94%). In order to make sense of these data, it helps to understand that when the common assessments and corresponding rubrics were introduced in Fall 2004, the quality of student work during that first term was low for all courses, including online and F2F. Data from subsequent terms show a more consistent assessment of student work see Table 6 where the first term s data are discarded which probably reflects a growing sense of expectations for assignment quality amongst faculty and students over time. Dropping the first term s assessment results for both online and F2F formats should therefore not adversely impact this comparison. Table 6 provides evidence that the work products created in the courses examined, regardless of the format used, are comparable and further supports the argument that these courses are similar in terms of their overall instructional quality. In order to look more closely at how students who took the online courses rated them, additional survey data were gathered, but only from students in selected online courses. Course survey data Additional anecdotal data were gathered from selected online courses that support the claims made above concerning the overall quality of instruction. Table 7 shows Table 4. Instructor assessments of common assignments Course and assignment Format Distinguished Proficient Progressing EDG 619 CIP Online 33 75% 10 23% 1 2% F2F 14 87% 2 13% 0 0% Note: CIP = curricular integration project. Table 5. Instructor assessments of common assignments Course and assignment Format Distinguished Proficient Progressing EDG 621 Analysis Tech Adoption Plan Online 15 94% 1 6% 0 0% F2F 26 79% 6 18% 1 3%

8 688 A. Topper Table 6. Comparing levels of quality in student work Course and assignment Format Distinguished Proficient Progressing EDG 619 CIP Online 23 85% 4 15% 0 0% F2F 14 87% 2 13% 0 0% EDG 620 EWE Online 12 92% 1 8% 0 0% F2F 29 93% 2 7% 0 0% EDG 621 Poster Online 15 94% 1 6% 0 0% F2F 32 97% 1 3% 0 0% Note: CIP = curricular integration project; EWE = educational website evaluation; Poster = topical poster session. how students in four online course sections responded to the question: Overall, how would you rate the quality of this course? While these data are not conclusive on their own, they further support the finding that the online courses are perceived by students as comparable with traditional, F2F courses. These data provide more evidence, in the form of student surveys, that the overall quality was high or at least average i.e. comparable with other courses they have taken at this institution. Such anecdotal data help to confirm what the results of the statistical analysis and the comparison of student work products suggests: that the instructional quality of the graduate educational courses is the same regardless of the format used. Conclusions This work began with three questions, which will each be addressed in this section. The first question: How do web-based course evaluations compare with paper-based versions? As the evidence showed, course evaluations, in terms of response rates, compared favorably across format: F2F and online both had similar levels of response by students. While the goal may be for 100% responses on all course evaluations, this is ultimately difficult (and may be impossible) to attain, for either format. Perhaps offering students time to complete these course evaluations, whether online or in a classroom setting, and promoting their purpose and value will be sufficient, so long Table 7. Overall quality survey item results for online courses Term and year Course High Average Low Fall 2002 EDG Fall 2003 EDG Winter 2004 EDG620A Winter 2004 EDG620B Totals

9 Instructional quality of online and face-to-face graduate courses 689 as response rates for both formats do not drop below a respectable level. A worthwhile place to start is a discussion with both faculty and administration about what is a respectable level of student responses for course evaluations. The second question How do course evaluations from online or web-based courses compare with traditional, face-to-face (f2f) courses? also seemed to be answered, both in this study and also in the weight of other, similar studies. The data analyzed in this study, taken from over two years of online and F2F graduate education courses, seem to suggest strongly that there are no statistically significant differences in students evaluations of courses offered in either format. It is hoped that this will allow us to move beyond this question and on to what we believe is the more meaningful one: How can we ensure that the instructional quality or effectiveness of graduate education courses, regardless of the format in which they are offered, is sufficiently high? This will, it is hoped, lead to another discussion about what constitutes sufficiently high instructional quality, based on course evaluations, as part of a larger set of discussions on overall program quality. And finally, what other products can be used to evaluate instructional quality? Again, this study used NCATE common assessment data to demonstrate that the products created in graduate education courses are similar in quality, regardless of the format used: F2F or online. The results from this work may provide some ideas concerning the value of examining student work, in a variety of forms, to help evaluate the instructional quality or effectiveness of the course, regardless of the format. One benefit of this work is that it promotes discussion on these issues and will, it is hoped, lead to more consistent and effective instruction in whichever format is used. As has been pointed out above, the small sample sizes, possible cohort differences and students self-selecting course sections and medium used may all have impacted on the observed results. But, when taken together, the course evaluation and student work data, in combination with data reported in prior studies, suggest that from a student perspective, online and face-to-face (or blended) courses are the same. Future work While the initial results from this research are promising, and consistent with other results reported in the literature, there is still much work to be done. The data reported here support the argument that the instructional quality of F2F and online graduate education courses are comparable, with evidence drawn from course evaluations as well as the products created that demonstrate competency. It is hoped that these data will help to counter the claim often made that online courses are not of comparable quality to in-class or face-to-face offerings. Each term, we continue to gather more data from students in the form of course evaluations that can be used for future comparisons. Given the relatively small sample sizes in this study, as more data are gathered, we will continue to analyze them looking for evidence to support or refute the claims made here. We have also started conversations regarding evidence of quality instruction and ways to measure it, building on the items identified in this article. Any course,

10 690 A. Topper whether F2F or online, is probably taught better each time it is offered since the instructor learns more about the material and how to teach it. The goal is, or should be, to continue to improve the quality of instruction in whatever format is used. Notes on contributor Andrew Topper, Ph.D., is an associate professor who teaches graduate educational technology courses at Grand Valley State University (GVSU). His research interests are exploring the influences of technology on teacher knowledge, beliefs and practices, how adults learn in web-based or online environments, and how online learning can be facilitated to maximize instructional effectiveness. Dr Topper was a member of the editorial board of the Journal of Research on Technology in Education for four years, coordinates the graduate educational technology program at GVSU, and is actively involved in scholarship related to teaching with technology. References Buckley, K. M. (2003) Evaluation of classroom-based, web-enhanced, and web-based distance learning nutrition courses for undergraduate nursing, Journal of Nursing Education, 42(8), Dommeyer, C. J., Baum, P., Hanna, R. W. & Chapman, K. S. (2004) Gathering faculty teaching evaluations by in-class and online surveys: their effects on response rates and evaluations, Assessment & Evaluation in Higher Education, 29(5), Jackson, D. L., Teal, C. R., Raines, S. J., Nansel, T. R., Burdsal, C. S. & Force, R.C. (1999) The dimensions of students perceptions of teaching effectiveness, Educational and Psychological Measurement, 59(4), Kleiman, G.M. (2005) Meeting the need for high-quality teachers: e-learning solutions. U.S. Department of Education Secretary s No Child Left Behind Leadership Summit: increasing options through e-learning. Available online at (accessed 15 October 2006). Layne, B. H., DeCristoforo, J. R. & McGinty, D. (1999) Electronic versus traditional student ratings of instruction, Research in Higher Education, 40(2), Merisotis, J. P. & Phipps, R.A. (1999) What s the difference? Change, 31(3), Moss, J. & Hendry, G. (2002) Use of electronic surveys in course evaluation, British Journal of Educational Technology, 33(5), Phipps, R. & Merisotis, J. (1999) What s the difference? A review of contemporary research on the effectiveness of distance learning in higher education (Washington, DC, Institute for Higher Education Policy). Purcel, D. J. & Stertz, T. F. (2005) Effectiveness of and student satisfaction with Web-based compared to traditional in-service teacher education courses, Journal of Industrial Teacher Education, 42(1), Skylar, A. A., Higgins, K., Boone, R., Jones, P., Pierce, T. & Gelfer, J. (2005) Distance education: an exploration of alternative methods and types of instructional media in teacher education, Journal of Special Education Technology, 20(3), Spooner, F., Jordan, L. & Algozzine, R. (1999) Student ratings of instruction in distance learning and on-campus classes, Journal of Educational Research, 92(3), Topper, A. (2005) Evaluating the quality of online graduate educational technology courses: what can student course evaluations and participation tell us about our instruction?, paper

11 Instructional quality of online and face-to-face graduate courses 691 presented at the annual Society for Information Technology in Education (SITE) Conference, Phoenix, AZ, March. Tucker, S. Y. (2000) Assessing the effectiveness of distance education versus traditional oncampus education, paper presented at the annual meeting of the American Educational Research Association. New Orleans, LA: AERA 2000 Conference. Watt, S., Simpson, C., McKillop, C. & Nunn, V. (2002) Electronic course surveys: does automating feedback and reporting give better results?, Assessment & Evaluation in Higher Education, 27(4), Appendix A: University course evaluation items The instructor explained material well Course objectives were clearly presented The instructor is fair in evaluating students The instructor seemed to enjoy teaching this class The instructor demonstrated that it was important to learn about the subject The instructor was willing to expend energy and time to help students learn The instructor set realistically high standards for students The instructor communicated effectively with students The instructor s objectives were reflected in his/her teaching The work requirements and grading system were clearly explained The instructor communicated enthusiasm and desire for learning The instructor served as a role model for learning The instructor was sensitive to student needs The instructor was committed to developing academic excellence in students Notes: Items in italics were selected for analysis in this study. Likert scale items: 5 = Excellent, 4 = Good, 3 = Average, 2 = Below average, and 1 = Poor. Open-ended questions: students provide additional qualitative comments on the course.

12