A comparison of online versus face-to-face teaching delivery in statistics instruction for undergraduate health science students

Transcription

1 Adv in Health Sci Educ DOI /s A comparison of online versus face-to-face teaching delivery in statistics instruction for undergraduate health science students Fletcher Lu Manon Lemonde Received: 16 July 2012 / Accepted: 2 December 2012 Ó Springer Science+Business Media Dordrecht 2012 Abstract The objective of this study was to assess if online teaching delivery produces comparable student test performance as the traditional face-to-face approach irrespective of academic aptitude. This study involves a quasi-experimental comparison of student performance in an undergraduate health science statistics course partitioned in two ways. The first partition involves one group of students taught with a traditional face-to-face classroom approach and the other through a completely online instructional approach. The second partition of the subjects categorized the academic aptitude of the students into groups of higher and lower academically performing based on their assignment grades during the course. Controls that were placed on the study to reduce the possibility of confounding variables were: the same instructor taught both groups covering the same subject information, using the same assessment methods and delivered over the same period of time. The results of this study indicate that online teaching delivery is as effective as a traditional face-to-face approach in terms of producing comparable student test performance but only if the student is academically higher performing. For academically lower performing students, the online delivery method produced significantly poorer student test results compared to those lower performing students taught in a traditional face-to-face environment. Keywords Distance education Online Face-to-face Cognitive skills Interactive environments Undergraduate education Introduction Online delivery of courses have become increasingly popular due to several advantages for both education institutions and the students in terms of flexibility in scheduling, F. Lu M. Lemonde Faculty of Health Science, University of Ontario Institute of Technology, Oshawa, ON, Canada F. Lu (&) Faculty of Business and Information Technology, University of Ontario Institute of Technology, 2000 Simcoe Street North, Oshawa, ON L1H 7K4, Canada fletcher.lu@uoit.ca

2 F. Lu, M. Lemonde opportunity to reach remotely located students, economies of scale and application of nontraditional educational tools such as customized learning programs (Cook 2007). However, the utility of an online delivery approach is also predicated on the approach s effectiveness in producing comparable (or better) learning outcomes compared to the traditional face-toface approach (Ellaway and Masters 2008; Garrison and Kanuka 2004). Much of the early research that has been compiled by Russell (1999) had indicated that student performance is on average comparable using online recorded lectures compared to a traditional face-toface approach, which he called the no significant difference phenomenon. This result has been supported by numerous subsequent studies and meta-analyses both inside and outside the health education field (Campbell et al. 2008; Goodrich 2007; Riffell and Sibley 2005; Sitzmann et al. 2006). However, work by Ross and Bell (2007) has indicated that the earlier studies results may have only related to what Wieling and Hofman (2010) refers to as surface learning. Ross and Bell s (2007) work demonstrated that deeper levels of learning, requiring more abstract thinking, show a difference in test performance between students in a face-to-face environment versus those learning through strictly online asynchronous methods. The face-to-face students in their study performed significantly better than the online students in test questions requiring application/analysis/synthesis, while more surface learning problems dealing with knowledge and comprehension showed no statistically significant difference between the two student groups. Bloom s taxonomy of cognitive domains (Huitt 2011; Bloom 1956) that was revised by Krathwohl (2002) indicates that students must be able to achieve the lower cognitive skill levels such as remembering and understanding before being able to demonstrate higher cognitive skill levels such as being able to analyze, evaluate and create. Such higher cognitive processes requiring lower levels to be first achieved are thus, inherently more complex. It would thus appear from Ross and Bell s (2007) study that online asynchronous teaching is more challenging for instructing students on the more complex learning attributes of analyzing/ evaluating/creating. However, de Jong et al. s (2012) study on a graduate statistics class supported the no significant difference results on their graduate course, which required the higher cognitive skills of analyzing, evaluating and creating using a problem-based teaching approach. These studies (de Jong et al. 2012; Ross and Bell 2007) would indicate that although online instruction may have greater difficulty imparting higher levels of cognitive skills, it does not necessarily preclude it. One of the differences between the studies that may indicate a factor determining success in the higher cognitive skills level test performance would be in student aptitude to learn. The Ross and Bell study principally studied undergraduate juniors and seniors while the de Jong study tested graduate students. Graduate students generally have greater academic success as part of a requirement to enter graduate studies than the general undergraduate student as graduate students originally come from a subset pool of the undergraduate population pool that then continue on to graduate studies. Thus, under such reasoning, our research explores if scholastic aptitude (which we interpret as academic ability) is a factor affecting student test performance for a face-to-face delivered course compared to an online version of the same course and how it relates to cognitive skill levels. This paper presents a quantitative analysis of a quasi-experimental study on student test performance comparing a completely online teaching delivery approach against a traditional face-to-face teaching method for a health science course in undergraduate statistical methods. For the purposes of our study, the term online teaching refers to an approach where the delivery of all lectures, assignments and learning object tools may be accessed asynchronously by the students through the internet. No face-to-face interaction occurs between the instructor and online taught students, except during testing. Also for the

3 Online versus face-to-face teaching delivery purposes of this study, in order to determine if scholastic aptitude is a factor in online versus face-to-face student test performance, we quantify scholastic aptitude by dividing the students into academically higher performing students and lower performing students for each teaching delivery method based on assignment marks. In order to avoid skews due to students who missed assignments, the median grade on assignments was used to divide the students, with those students whose overall assignment averages that were above the median being classified as higher performing and those below the median being classified as lower performing. For those students categorized as higher performing, their test results replicated the results of the many past studies showing no significant difference in their test performance between online versus face-to-face teaching delivery. But the students categorized as lower performing demonstrated test results that were significantly poorer for those enrolled in the online delivery version compared against their lower performing counter-parts in the face-to-face delivery version. Thus, Russell s (1999) no significant difference phenomenon may be better clarified as a no significant difference phenomenon for lower cognitive skill levels or students of higher scholastic aptitude. Materials and methods We classify this study as quasi-experimental as it originated from a naturally self-selected group of undergraduate university health science students who were given the option to choose between an online statistics course and one taught in a traditional face-to-face manner. The student population consisted of those majoring in general health science studies, nursing, kinesiology and medical laboratory studies. In order to mitigate the effects of self-selection, such as those discussed in Collins and Pascarella s (2003) paper, which could result in other explanations for differences in test performance, the researchers evaluated the assignment performance of the students to determine if the two groups of students had any significant differences in their aptitude for the subject. The baseline assumption being that assignment performance is a measure of their general topic competency. Table 1 shows the results of that assessment showing the means, standard deviations, effect sizes and independent t test values of student assignment mark averages. The same assignments were given to all students: online and face-to-face. The results are shown for the overall group of students in the far right column as well as those categorized above and below the median assignment grade. There was no statistically significant difference in student assignment performance between the online and face-to-face students in both the overall mark (p = 0.29) and in the sub-categorization of above (p = 0.26) and below median grade students (p = 0.15). The effect size for each of the columns of Table 1 (uses Cohen s effect size method) indicates that if there is any difference in student scholastic aptitude between the online and face-to-face students it is small. The online class size was restricted by the administration to a maximum 30 in keeping with Qiu s (2010) general recommendations for the need for smaller class sizes for teaching online delivery effectiveness. The actual online class enrolment was 20 students with 72 students enrolled in the face-to-face delivery approach. For a more detailed analysis of the assignment performance differences see the Results section. Course and subjects The online version of the course began with 20 students with the face-to-face version beginning with 73 students. One student from the face-to-face version dropped the class

4 F. Lu, M. Lemonde Table 1 Comparing online versus face-to-face assignment grades Above median Below median Overall Mean online Mean face to face Standard deviation online Standard deviation face to face Effect size t statistic p value before the final exam. The course is a required subject for all students enrolled in a health science related program. The students majors were principally in nursing (20 %), general health science (60 %), kinesiology (10 %) and medical laboratory studies (10 %). The course is typically taken in second or third year of their programs depending on the student s major. It is a course in statistical research techniques designed specifically for students studying in a health science field. The learning objectives for the course are specified in the university s course outline and are listed as: To understand and critically appraise scientific and everyday literature on the statistical techniques used; To have a basic understanding of standard statistical techniques, applying these methods, their theoretical principles and conditions under which to apply them; To have an appreciation on how to write and read research studies. In order to test the hypothesis that poorer performing students test results will be significantly worse for an online delivery approach in contrast to a face-to-face approach, we will subdivide the students in each teaching delivery group into higher and lower performing students using their median assignment mark for each group. This approach is in keeping with Garavilia and Gredler s (2002) work, which indicated that assignment performance should be a strong predictor on subsequent future performance, in this case poor performance. The written assignment average was computed from five assignments given out during the course, with each assignment given approximately 2 week each to complete. The median assignment mark for each student was computed. Online student assignment median was 79.9 % while face-to-face students had an assignment median of 82.1 %. After the students were split into higher and lower performing students using their median test scores in each group we had 10 lower performing students in the online class and 36 lower performing students in the face-to-face class. Similarly we have 10 higher performing students in the online class and 36 higher performing students in the face-toface class. The student test performance on the subjects that we report is from the closed book final exam for the course which was administered 1 week after the last regularly scheduled face-to-face class meeting. Online delivery tools Both the face-to-face and online courses used the same online learning tools that are listed below. The only difference was in the lecture component. In the lecture delivery, the faceto-face class had the lectures presented in a traditional manner in a classroom by the instructor using PowerPoint slides. The online course had their lectures delivered via

5 Online versus face-to-face teaching delivery Camtasia studio software recording where the PowerPoint slides are video recorded with the lecturer s accompanying voice capturing all slide changes and mouse movements. The recordings could therefore be accessed by the online students asynchronously on their own schedule. The PowerPoint slides were available for download to both classes. Assigning and return of assignments as well as delivery of marks and discussions were all conducted through the internet using the system known as WebCT. Both classes had weekly 1 h tutorials where the online class interacted in real time with the tutor through the software system known as Adobe Connect which allows for audio and text conversations as well as posting of documents and virtual whiteboards to write out problems and solutions. The face-to-face class tutorial was conducted in the traditional face-to-face format using real whiteboards and documents. Several customized web-based learning tools were created by the instructor to teach concepts such as demonstrating probabilities with virtual die and choosing and applying statistical test techniques for medical/health studies. All the custom made online tools were made available to both classes. Student assessments Two forms of assessment were used in this health statistics course: written assignments and a closed book final exam. The assignments were delivered through WebCT and written submissions were handed in by students through WebCT for both the online and face-toface students. The closed book final exam was administered on campus with both groups of students writing in the same room. The assignments and final exam comprised of the same questions for both groups. Thus any differences in performance should not be attributable to any differences in the assessments construction. Also, marking of the assignments and final exam was conducted by the instructor and two teaching assistants with questions being divided among the markers so that one person marks all students for a given question. Thus, there should be a consistent marking for all students. The final exam consisted of 36 % multiple choice questions, 22 % short answer questions, and 42 % computational questions, by mark value. Among the cognitive skill levels, the multiple choice questions principally tested recall and recognition as the choices provide the correct answer and thus require the student to recognize which of the choices is correct tying into the course learning objectives of basic understanding of statistical methods and the literature. The short answer questions tested both recall and providing explanations, classifications and summarizations of statistical terms and methods. The computational questions required the application of the statistical methods as well as analysis, and evaluation of the results. Results An assessment of the reliability of the three test question forms was conducted using a Cronbach s alpha value computed from the test performance results taken from all 92 students who wrote the test. The multiple choice questions had a Cronbach alpha value of 0.71, the short answer questions had a Cronbach alpha value of 0.67 and the computational questions had a Cronbach of These results indicate that the computational questions had the highest internal consistency, followed by the multiple choice and then the short answer questions with borderline acceptable reliability that they are evaluating the same cognitive skills.

6 F. Lu, M. Lemonde We address the internal validity of our study assumptions with Figs. 1and 2 that illustrate the scatterplots and correlations between assignment and test performance for the online and face-to-face students. One of the basic assumptions in our study design is that a student s assignment grade could be used as a categorization method to classify a student s scholastic aptitude. Figures 1 and 2 show a correlation between assignment average grade and test performance had a correlation of 0.51 (df = 18) and 0.46 (df = 70), which has a significance level of 0.44 and 0.23 at a 5 % rejection level, respectively. Thus both online and face-to-face students have significant correlations between their assignments and test performance and can be used to categorize the likely test performance of students into higher and lower performing groups. The t tests of Table 1 in the Materials and Methods section also illustrate the validity that our test sub-groups of lower performing students in the online compared to the face-to-face group are not significantly different in their scholastic aptitude; similarly for the higher performing students comparing the online to the face-to-face higher performing students they are not significantly different in their scholastic aptitude. The last row of Table 1 shows the corresponding p values and illustrates that none of the assignment sub-category groups nor overall group of online versus face-to-face students reach a 5 % significance level. Similar to Tables 1 and 2 illustrates the means, standard deviations, effect size, independent t test and corresponding p values but for the student test performances rather than the assignments. Table 2 s columns breakdown the statistic values for each of the exam components: the Multiple Choice section (MC), the Short Answer section (SA) and the Computational Analysis section (CA) as well as providing the overall student test performance values in the far right column. It is important to note that a negative t value indicates that the online students were performing worse than their face-to-face counterparts, while a positive t value indicates that the face-to-face students were performing worse. The results of Table 2 provide support for some of the past conclusions of both Russell (1999) s no significant difference and Ross and Bell s (2007) cognitive skill level difference. For the overall group of students, without partitioning them into scholastic aptitude levels, there was no statistically significant difference between the online and face-to-face students for the multiple choice (p = 0.22) or the short answer test (p = 0.29) results. There was a statistically significant difference in the computational questions (p = ). This significant difference was also reflected in the overall final exam Fig. 1 Correlation of final exam grade with assignment grade for online students

7 Online versus face-to-face teaching delivery Fig. 2 Correlation of final exam grade with assignment grade for face-to-face students Table 2 Comparing online versus face-to-face final exam performance Final exam MC Final exam SA Final exam CA Final exam Overall Overall Mean online Mean face to face Standard deviation online Standard deviation face to face Effect size t statistic p value \ Above median Mean online Mean face to face Standard deviation online Standard deviation face to face Effect size t statistic p value Below median Mean online Mean face to face Standard deviation online Standard deviation face to face Effect size t statistic p value \0.01 \0.01 marks of the students, which had the CA section make up 42 % of the overall final exam mark, resulting in a significant difference between the online and face-to-face students (p = 0.01). Looking at the student performance of the subcategories of above median

8 F. Lu, M. Lemonde (higher performing) and below median (lower performing) students, the no significant difference effect is much more consistent for the higher performing students. None of the differing question forms reaches a 5 % significance level between the online and face-toface higher performing students, though we do see a widening of the difference in performance as the cognitive skill levels of the question types increase, with the computational questions getting the closest to statistical significance. Only with the lower performing students does the overall test performance between the online and face-to-face students reach the 5 % significance level. As with the higher performing students, as the cognitive skill level of the test questions increases the difference in test performance between the online and face-to-face students increases. In order to analyze the interaction between the changing cognitive skill levels of the questions and the scholastic aptitude levels of the students, we conduct a two-way ANOVA test on the overall test results as well as for each subcategory of question form (multiple choice, short answer and computational question) with the F statistic results illustrated in Table 3. The Teaching Delivery group refers to the online versus face-to-face groups, the Scholastic Level refers to the higher level and lower level student performance groups and the Interaction refers to how much the Teaching Delivery and Scholastic Level interactively affect each other. As each of the Teaching Delivery and Scholastic Level categories have only two groups, the between group degree of freedom is 1 while the within group degree of freedom is 88, corresponding to the number of student participants. At a 5 % significance level this corresponds to an F value of Table 3 demonstrates that at the lowest cognitive level relating to the multiple choice questions, not surprisingly, the student aptitude to learn and the teaching delivery mode each affects the student s performance but they do not significantly interact. As the cognitive level of the questions shift from short answer to computational questions, the student scholastic aptitude continues to have a strong effect on the student test performance, but the teaching delivery mode effect becomes stronger tied with that student s scholastic aptitude with the interaction effect between the two categories significantly affecting each other (F = 5.09) for the computational questions. These results illustrate that overall, the method of teaching delivery of either online or face-to-face can have an effect on test performance. How that effect gets expressed depends in part on both the cognitive skill levels required of the course and the student s own academic ability. The t-test results indicate that for lower performing students, the online teaching delivery method results in significantly poorer test performance especially for questions requiring higher cognitive skill levels. Discussion and conclusions Our study supports aspects of the Ross and Bell (2007) findings that as the cognitive skill levels being tested increase, student performance decreases in an online teaching Table 3 F Statistic values for a two-way ANOVA comparing grades with teaching delivery method and scholastic level of students Final exam MC Final exam SA Final exam CA Final exam overall Teaching delivery Scholastic level Interaction

9 Online versus face-to-face teaching delivery environment versus a traditional face-to-face teaching environment. Like Ross and Bell, we broke down the components of our final exam to analyze student performance with Multiple Choice (MC) exams requiring mostly recall and recognition thinking and thus a lower cognitive skill level of learning than the Short Answer (SA) questions which require more understanding (comparing, interpreting, etc.), while the Computational Analysis (CA) questions require the greatest level of cognitive skills. Our Table 2 results had online higher performing students producing better marks than their face-to-face counterparts in MC and SA questions. It was only in the CA questions that the online students performed worse, but not significantly worse. However, for lower performing students, there was a consistent gradual worsening of test performance demonstrated with each successively more cognitively challenging question form. The online students were always consistently under-performing compared to their lower performing face-to-face counterparts. These results indicate that for students who struggle academically, online courses with their lack of face-to-face synchronous interaction may generally produce worse test performances than if such students took the same course in a traditional face-to-face format. For such struggling students, as the cognitive skill level required for the course increases, test performance in the online class tends to degrade precipitously compared to their potential performance in a face-to-face version of the same course. For generally higher performing students, our study was not able to show any such consistency in test result degradation as cognitive skill level increased. In this study, we have not explored the potential reasons that lower performing students do worse in an online environment while their higher performing students do not show such a difference. Indeed, Campbell et. al. (2008) in their study showed that those of higher scholastic aptitude such as graduate students can even have better performance with an online or hybrid online course. Riffell and Sibley (2005) also had such better performance at the undergraduate level but was also able to show that the benefits were more pronounced with upper year students. Our study has some limitations due in part to the selfselected approach of the students choosing for themselves which form of the course they took. Thus, the possibility that some factor affects both their choice for taking an online course and their ability to apply higher cognitive skills may be at play. There are a number of possible explanations which the researchers would like to explore for future work. For instance, face-to-face classes generally provide more structure as a regular class meeting time must be coordinated (Ellaway and Masters). This increased structure due to regular meetings may force greater discipline that aids poorer performing students in their learning habits by reducing procrastination for example (Romano et al. 2005). Young s (1996) analysis indicated that those with poor self-regulated learning strategy ability tended to do poorer in an environment where the student had controls on the teaching delivery such as in an online environment. Thus, lower performing students may be poorer in self-regulated learning strategy. Low test performance of students was also shown for those frequently using an online teaching tool by Le et al. (2010), in a comparison study of students who had access to both the face-to-face form of lecture delivery and recorded online lectures. Le et al. s study illustrated that those using recorded lecture tools the most actually performed the poorest. They attributed the results to the type of surface learning that Ross and Bell indicated in their study. The belief is that those viewing a recorded lecture many times and/or pausing the recording many times most likely are having a difficult time understanding a concept and thus are failing to achieve the deeper level of learning needed to answer complex test questions of analysis and evaluation. This surface level of learning may also explain our

10 F. Lu, M. Lemonde study result of why poorer performing students would also do significantly poorer in a purely online learning environment instead of a face-to-face environment. In the face-toface environment concepts that a student is having difficulty with may be interactively explored where instructors may answer clarifying questions of the student. In a recorded lecture scenario, that interactive and synchronized approach is lost. The face-to-face environment also provides students with the opportunity to interact with the instructor during the actual lecture to bring up questions for clarification and elaboration on topics at the moment of the presentation of a topic. It could be that receiving the clarification at the moment when a topic is learned, especially one that requires a great deal of abstract thinking, helps avoid confusions that would not be otherwise easily clarified after the fact during, for instance, a tutorial. An avenue of future study would be exploring why this would affect lower performing students more than higher performing ones. From a practical stand-point, for students and educators, our study would indicate that offering online courses is a valid option for educating students that would produce comparable test performance but only for a segment of the student body that is generally higher performing. The traditional face-to-face format for educating is most acutely useful for educating those students generally struggling academically or those courses that require a heightened level of cognitive learning. For educators, especially those involved in allocation of resources and curriculum design, it would indicate that online delivery is best kept for those courses not requiring high cognitive skill levels. For students, the saying know thy self in terms of one s own scholastic aptitude for a topic may serve best in determining if a particular student should take an online course in that topic. References Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. New York: David McKay. Campbell, M., Gibson, W., Hall, A., Richards, D., & Callery, P. (2008). Online versus face-to-face discussion in a web-based research methods course for postgraduate nursing students: A quasi-experimental study. International Journal of Nursing Studies, 45, Collins, J., & Pascarella, E. T. (2003). Learning on campus and learning at a distance: A randomized instructional experiment. Research in Higher Education, 44(3), Retrieved July 20, 2008 from Cook, D. A. (2007). Web-based learning: Pros, cons and controversies. Clinical Medicine, 7, de Jong, N., Verstegen, D. M. L, Tan, F. E. S., & O Connor, S. J. (2012). A comparison of classroom and online asynchronous problem-based learning for students undertaking statistics training as part of a Public Health Masters degree. Advances in Health Science Education (Epub ahead of print). Retrieved June 26, 2012 from Ellaway, R., & Masters, K. (2008). AMEE Guide 32: E-learning in medical education, Part 1: Learning, teaching and assessment. Medical Teacher, 20, Garavilia, L. S., & Gredler, M. E. (2002). Prior achievement, aptitude and use of learning strategies as predictors of college student achievement. College Student Journal, 36(4), Garrison, D. R., & Kanuka, H. (2004). Blended-learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7, Goodrich, C. (2007). Using web-based software to enhance learning of analytical and critical skills. Journal of Educational Technology Systems, 36, Huitt, W. (2011). Bloom et al. s taxonomy of the cognitive domain. Educational Psychology Interactive. Valdosta, GA: Valdosta State University. Retrieved March 22, 2011 from active.org/topics/cogsys/bloom.html. Krathwohl, D. (2002). A revision of Bloom s taxonomy: An overview. Theory Into Practice, 41(4),

11 Online versus face-to-face teaching delivery Le, A., Joordens, S., Chrysostomou, S., & Grinnel, R. (2010). Online lecture accessibility and its influence on performance in skills-based courses. Computers & Education, 55, Qiu, M. (2010). A mixed methods study of class size and group configuration in online graduate course discussions. Dissertation for the Degree of Doctor of Philosophy Department of Curriculum, Teaching, and Learning, Ontario Institute for Studies in Education. University of Toronto. Riffell, S., & Sibley, D. (2005). Using web-based instruction to improve large undergraduate biology courses: An evaluation of a hybrid course format. Computers & Education, 44, Romano, J., Wallace, T. L., Helmick, I. J., Carey, L. M., & Adkins, L. (2005). Study procrastination, achievement, and academic motivation in web-based and blended distance learning. The Internet and Higher Education, 8, Ross, T. K., & Bell, P. D. (2007). No significant difference only on the surface. International Journal of Instructional Technology and Distance Learning, 4(7), Russell, T. L. (1999). The no significant difference phenomenon. USA: Office of Instructional Telecommunications, North Carolina State University. Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59, Wieling, M. B., & Hofman, W. H. A. (2010). The impact of online video lecture recordings and automated feedback on student performance. Computers & Education, 54, Young, J. D. (1996). The effect of self-regulated learning strategies on performance in learner controlled computer-based instruction. Educational Technology Research and Development, 44(2),