A Comparative Study of the Effectiveness of an Online and Face-to-Face Technology Applications Course in Teacher Education

Kuo, J., Song, H., Smith, R., & Franklin, T. (2007). A comparative study of the effectiveness of an online and face-to-face technology applications course in teacher education. International Journal of Technology in Teaching and Learning, 3(2), 85-94. A Comparative Study of the Effectiveness of an Online and Face-to-Face Technology Applications Course in Teacher Education Chia-Ling Kuo Kent State University Hongbo Song Yantai University Renee Smith and Teresa Franklin Ohio University An online and face-to-face technology applications course was examined for their effectiveness on technology attitude of college students and their technological application skills. A three-part survey was designed and administered to 62 education undergraduate students. Twenty-four out of 27 valid surveys were returned from the online group, while 34 out of 35 from the face-to-face group. The results showed that there was no significant difference between the online and face-to-face classes in terms of students attitudes toward technology before and after taking a technology applications course although their technological application skills were increased after both modes of course delivery. Keywords: effectiveness, online learning, technology applications course, teacher education BACKGROUND OF THE STUDY The Internet has become an increasingly important tool in education during the past decades. More and more colleges and universities provide online courses for their students via spending a great deal of time and effort in looking inward to analyze the efficacy of Chia-Ling Guo is an assistant professor at Kent State University. Hongbo Song is an associate professor in the School of Foreign Languages, Yantai University, Laishan, Yantai, Shandong, P. R. China. E-mail: hongbo_song@hotmail.com. Renee Smith is an adjunct teacher at Ohio University. Teresa Franklin is an associate professor of Instructional Technology, College of Education, Ohio University.

86 International Journal of Technology in Teaching & Learning learning through the use of technology in addition to making substantial investments in new technologies for teaching (Phipps & Merisotis, 1999, p. 9). Whether or not the learning outcomes are the same in both online and face-to-face learning environments remains a point of debate (Johnson, Aragon, Shaik, & Palma-Rivas, 2000; Phipps & Merisotis, 1999; Robertson, Grant, & Jackson, 2005; Sellers, Wright, & Wright, 2003; van Schaik, Barker, & Beckstrand, 2003). Furthermore, learners various perceptions of online courses concerning course quality, learning content, learner satisfaction, learning effectiveness, course assessment, interaction level, learning flexibility, course design, technical support, online resources, learner autonomy, and online pedagogy may largely contribute to the complexity of such a debate. PURPOSE OF THE STUDY This study examined pre-service teachers attitudes toward technology and their selfevaluation of technological application skills in a technology applications course in two teaching formats: online and face-to-face. The results of the study may help instructors to better understand whether the instruction of online or face-to-face technology courses is similarly effective in terms of attitude and learning outcomes. The results may provide reference points on how to design dynamic online courses to maximize learning opportunities and achievements in online learning environments, and on how to provide necessary assistance to online students to better achieve teaching/learning outcomes. REVIEW OF LITERATURE Distance learning is geographically separated, flexible in time, and operated through appropriate and corresponding media of technology (e.g., microwave system, compressed video system, two-way video and two-way audio videoconferencing) and non-technology (e.g., organizational management, administrative support, policy). In other words, distance learning is The acquisition of knowledge and skills through mediated information and instruction, encompassing all technologies and other forms of learning at a distance (http:// www.usdla.org, 3). Online learning as a subset of distance learning facilitates the learning transaction through the Internet. As a special form of educational process or practice, however, the uncertainties of online learning remain to be solved regarding how, and in what ways, technology can enhance the teaching/learning process, particularly at a distance (Phipps & Merisotis, 1999, p. 29). Further research in online learning is thus required because there is little research to accurately determine the benefits and pitfalls of online instruction, particularly when compared to the more traditional face-to-face learning environment (Johnson et al., 2000, p. 30). Or, to put it differently, do students who study online gain more knowledge, or at least the same amount, as they do in a traditional face-to-face setting? Undeniably, the effectiveness of teaching and learning is an indicator of being able to achieve desired results under certain conditions; consequently, the ultimate indicator of course effectiveness is the extent to which students reach the learning objectives (Johnson et al., 2000). In terms of the effectiveness of online learning, three broad measures are usually examined in the original research: (1) student outcomes such as grades and test scores; (2) student attitudes about learning through online instruction; and (3) overall student satisfaction toward online instruction (Phipps & Merisotis, 1999). In a study that compared a graduate online course (n = 19) with an equivalent course taught in a traditional face-to-face format

Teaching Effectiveness 87 (n = 19) by the same instructor and with the same content, activities, and projects, Johnson et al. (2000) found that the students in the face-to-face course held slightly more positive perceptions about the instructor and overall course quality; students had significantly more positive views about interaction and support although there was no difference between the course s format in outcome measures in terms of course projects, grades, and most of the student self-assessment of tasks. In another study that focused on a community and technical college system s online courses, however, Prince and Stern (2002) presented the survey results in a report that compared online students (n 22,000) with on-campus students (n = 291,000) from 1999 to 2001. In this report, Prince and Stern found that students satisfaction with their online courses and instructors remained consistently high although their satisfaction level with library resources, technical support, and advising was lower. More recently, in a quasi-experimental mixed research design, van Schaik et al. (2003) reported that there was no observable difference in either test performance or attrition between on-campus and online course delivery methods, although those who studied online (n = 36) appeared to appreciate the flexibility of this approach to study content more than did the on-campus group (n = 83). Similar findings appeared in Caywood and Duckett s (2003) study that revealed no significant differences in the measurement of initial learning or follow-up performance between 76 graduate students from multi-cultural backgrounds who started classes online compared to another 75 graduate students with similar backgrounds who started the same on-campus course in the special education teacher credential program. However, in an Internet-based graduate allied health program for two groups defined based on residential status, Sellers et al. (2003) reported that when the interaction of residential status and employment status was examined, off-campus students who were employed in full-time positions achieved higher Grade Point Averages (GPAs). Further, in a study that compared differences in the academic achievement of postgraduate business students performance (final grade) in nine units offered in both face-to-face (n = 1,085) and electronic learning (n = 316) modes over the course of two years, Ladyshewsky (2004) reported that students, on average, did better in the electronic learning mode although at the individual unit level there were minimal, if any, significant differences. Age and gender did not appear to moderate performance in any way except for those students under thirty-three years old who did better, on average, in the electronic learning mode. Moreover, based on the 92 student survey responses (20 for online courses and 72 for on-campus courses), Robertson et al. (2005) further confirmed that online students perceived their learning experience the same or significantly better than the classroom-based learning for graduate education courses. These studies, via diverse foci, using various media, different levels of participants and settings, focusing on individual courses or whole programs, and utilizing assorted research methods and designs, presented mixed findings in terms of the effectiveness of distance learning as compared to traditional face-to-face learning. Under such circumstances, attitudes from pre-service teachers toward online and face-to-face teaching formats are important because their perceived effectiveness of these two distinct instructional delivery platforms may generate potential impact on the pre-service teachers future use of technology in the classroom. DESIGN OF THE STUDY This study, based on a pretest-posttest non-experimental design, was to examine changes in technology attitudes and technological application skills of students majoring in education. Two types of research questions, descriptive and statistical, were used in this study:

88 International Journal of Technology in Teaching & Learning DESCRIPTIVE QUESTIONS 1. What is the background information of the participants? 2. What are the attitudes toward technology among students in the online course before and after taking the technology course? 3. What are the attitudes toward technology among students in the face-to-face course before and after taking the technology course? 4. What are the technological application skills among students in the online course before and after taking the technology course? 5. What are the technological application skills among students in the face-to-face course before and after taking the technology course? STATISTICAL QUESTIONS 1. Do students attitudes toward technology in the online class differ from those in the face-to-face class before and after taking the technology course? 2. Do students technological application skills in the online class differ from those in the face-to-face class before and after taking the technology course? The instrument utilized for this study was a three-part survey designed by the researchers after careful examination of the literature in the field. The first part of the survey asked for participants demographic information such as gender, major, year in school, a self-evaluation of technology ability, and different ways they use technology. The second part of the survey measured participants attitudes toward technology via implementing a five-point Likert scale of a 15 item survey with 1 as strongly disagree (SD) and 5 as strongly agree (SA). The third part of the survey was a 20 item technological application skill test with 1 as I don t know how to do it, and 5 as I know how to do it very well. These technological application skill items were designed based on the course syllabus, which covers technological application skills in using Microsoft Word, PowerPoint, Excel, Publisher, multimedia software, and Web design software. The purpose of the questions design was to help the researchers understand how much the students learned after having taken a technology applications course. The content and construction of the survey instrument was evaluated by three professors and three doctoral students in the programs of Instructional Technology, Research and Evaluation, and Physics. To further control instrumentation bias, reliability for both scales was tested. Two internal consistency, coefficient alpha and split-half coefficient expressed as a Spearman-Brown corrected correlation, were computed via SPSS for Windows 12.0. For the split-half coefficient, the scale was split into two halves using odd and even item entry method. The reliability of attitude toward technology scale and the technological application scale in the pre-test and post-test for both face-to-face and online teaching formats were computed accordingly. The reliability for both scales in the pre-test and post-test were found to be consistent. The values for coefficient alpha and split-half coefficient were all above.83 (see Table 1 on the next page). These values were satisfactory because a measure that has a coefficient alpha of at least.6 or.7 and preferably closer to.9 is considered helpful (Aron & Aron, 2002). Sixty-two undergraduate students majoring in education at a large Midwestern university in the United States participated in the pre-test during the spring and summer quarters of 2004. Among them, twenty-seven students were enrolled in the online technology applications

Teaching Effectiveness 89 class and the other 35 students were enrolled in the equivalent face-to-face class. Of the 27 online students, twenty-four (8 males and 16 females) participated in the post-test with the majority of the students majoring in Early Childhood Education. Of the 35 face-to-face students, thirty-four (5 males and 29 females) participated in the post-test with the majority of students majoring in Middle School and Young Adult Education. Those unreturned post-test surveys in both modes of course delivery could have been the result of voluntary participation. Blackboard Learning Management System 6.0 was used for the online class to deliver course-related materials, instructional interactions, quizzes, feedback, and scores. For the online class, students were taught asynchronously using discussion boards, email, assigned readings, and electronic presentations; yet, they were expected to come to the first class meeting, one class meeting in the middle of the quarter, and the last class meeting. The planned combination of online learning with traditional face-to-face teaching or a hybrid/ blended delivery mode was to provide online learners with the opportunity to meet face-toface with the instructor and classmates. The students in the face-to-face class met twice a week in a two-hour session each. The teaching content was the same for both classes and by the same instructor for both modes of course delivery. Table 1. The Reliabilities of Each Scale for Pre-test and Post-test Online and Face-to-Face Groups Pre-Test Post-Test Coefficient Split-Half Coefficient Split-Half Alpha Coefficient Alpha Coefficient Face-to-Face Attitude toward Technology Scale.92.95.88.88 Technological Application Skill Scale.87.93.91.96 Online Attitude toward Technology Scale.92.93.89.91 Technological Application Skill Scale.92.95.83.91 The identical survey was administered to participants in both classes during the first and the last class meetings. Quantitative methods via running SPSS for Windows 12.0 were applied in this study. Descriptive statistics was used to understand the background information for both sets of the participants. An independent t test was used to analyze the differences between pre-test and post-test scores within each type of class (e.g., online and face-to-face) as well as the differences between the pre-test and pre-test and between the post-test and post-test across the two types of classes. The effect size for the t test for independent means was computed to understand the magnitude of difference between the means of both online and face-to-face groups.

90 International Journal of Technology in Teaching & Learning RESULTS Data collected via the administered survey were keyed into SPSS for Windows 12.0 for further analysis. The results of the study were reported as follows: OVERALL RESULTS FOR BACKGROUND INFORMATION OF THE PARTICIPANTS The participants included two groups: online and face-to-face. In the online group, twenty-four valid pre- and post-test data from 8 males and 16 females were collected. Eleven participants were majoring in Early Childhood Education, 6 in Middle Childhood Education, 3 in Young Adult Education, 1 in Physical Education, and 3 in the Integrated Education with special areas that included the grades 7-12. Seventeen out of the 24 participants were seniors, 6 juniors, and 1 for the sole purpose of getting a teaching license. With respect to the self-evaluation for the technology ability, the results of the pre-test showed that 33% of the participants thought that they were in the categories of non-users and beginners, but the percentage dropped to 17% in the post-test. In the pre-test, 66.7% of the participants evaluated themselves as confident in their technology ability and capable of teaching others, but the percentage increased to 83.3% in the post-test. Regarding technology use, the pre-test and the post-test did not show large differences. Most participants used technology in email communication and as a supplementary means to extend classroom learning. Word processing programs, presentation software, email, the Internet, CD-ROM, and online resources were most frequently used by the participants. In the face-to-face group, a total of 34 valid data from 5 males and 29 females were collected. Six participants were majoring in Early Childhood Education, 7 in Middle Childhood Education, 4 in Young Adult Education, 9 in the Integrated Education with special areas of concentration that included grades 7-12, 1 in Physical Education, 2 in Special Education, 3 in Art Education, and 2 in Language Education. Six out of 34 participants were sophomores, 11 juniors, and 17 seniors. Regarding the self-evaluation for the technology ability, the results of the pre-test showed that 31.4% of the participants evaluated themselves as beginners in their technology ability, but the percentage dropped to 26.5% in the post-test. In regards to technology ability, 57.1% of the participants in the pre-test indicated the category of confidence, while in the post-test, 50% of the participants thought that their abilities were in the category of confidence. In the category of capable of teaching others, there was only 5.7% of the participants who thought that they were in this category, but in the post-test, the percentage of this category increased to 23.5%. With respect to technology use, there was no difference between the pre-test and post-test. Most participants used technology for email communication. Word processing and presentation software were the most frequently used programs. Spreadsheet, Web design, and multimedia application software were less used by these participants. Table 2 on the next page presented a summary of the participants background information. ATTITUDE TOWARD TECHNOLOGY AND TECHNOLOGICAL APPLICATION SKILLS AMONG ONLINE PARTICIPANTS The overall means (M) and standard deviations (SD) of the attitude toward technology (ATT) in the pre-test and post-test in the online class were similar (M ATT-pretest = 53.26, SD = 10.45; M = 57.46, SD ATTpretest ATT-posttest ATT-posttest = 8.99). The value of skewness was normally distributed. The changes were from Skewness ATT-pretest =.29 to Skewness ATT-posttest =.08. However, the means, standard deviations, and the values of skewness were very different

Teaching Effectiveness 91 in the technological application skills (TAS) between pre-test and post-test (M TAS-pretest = 33.33, SD = 15.63; M = 68.96, SD = 5.67). The value of skewness TAS-pretest TAS-posttest TAS-posttest changed from Skewness TAS-pretest =.99 to Skewness TAS-posttest = -.85. In other words, the means of technological application skills were doubled in the post-test, but the spread of scores was three times smaller than those in the pre-test. Regarding the values of skewness, scores were distributed from low in the pre-test to high in the post-test. Table 2. Background Information Online Group Face-to-Face Group Number 24 (Males: 8, Females: 16) 34 (Males: 5, Females: 29) Major Early Childhood Education 11 6 Middle Childhood Education 6 7 Young Adult 3 4 Physical Education 1 1 Integrated with Special Areas 3 9 Special Education 2 Art Education 3 Language 2 Participating Group Senior 17 17 Junior 6 11 Sophomore 6 For Teaching License 1 Self-evaluation of Technology Level Non-Users (Pre- to Post-test) 4% to 0% 0% to 0% Beginners (Pre- to Post-test) 33% to 17% 37% to 27% Confident Users (Pre- to Post-test) 52% to 66% 57% to 50% Teaching Others (pre- to Post-test) 11% to 17% 6% to 23% Technology Use Most for email and Most for email; as supplementary means Word processing & to extend classroom PPT frequently used; learning (e.g., Word Excel, Web design processing, PPT, and multimedia CD-ROM, online application software resources) less used ATTITUDE TOWARD TECHNOLOGY AND TECHNOLOGICAL APPLICATION SKILLS AMONG FACE-TO-FACE PARTICIPANTS The means (M) and standard deviations (SD) of the attitude toward technology (ATT) in the pre-test and post-test among face-to-face participants did not have a big difference (M ATT-pretest = 53.66, SD ATT-pretest = 10.58; M ATT-posttest = 58.65, SD ATT-posttest = 7.64). The value

92 International Journal of Technology in Teaching & Learning of skewness was also distributed normally. The changes were from Skewness ATT-pretest = -.18 to Skewness ATT-posttest = -.02. However, the means, standard deviations, and the values of skewness were very different in the technological application skills (TAS) between pre-test and post-test (M TAS-pretest = 33.03, SD TAS-pretest = 12.28; M TAS-posttest = 65.50, SD TAS-posttest = 8.28). This result was similar to the online case, although the online students showed slightly higher scores in means with a smaller distribution. The value of skewness changed from Skewness TAS-pretest =.11 to Skewness TAS-posttest = -.65, which indicated that most scores in the technological application skills were high in the post-test. COMPARISON OF ONLINE CLASS WITH FACE-TO-FACE CLASS IN THE COMPUTER ATTITUDE BEFORE AND AFTER TAKING A TECHNOLOGY COURSE An independent t test was conducted to evaluate the difference of computer attitude between the participants in the online class and in the face-to-face class before and after taking a technology course. The results of the t test showed that there was no significant difference in the computer attitude between the two different teaching formats before and after taking the technology course, t before (60) =.15, p =.88 (M face-to-face = 53.66, SD face-toface = 10.58; M online = 53.26, SD online = 10.45); t after (56) =.54, p =.59 (M face-to-face = 58.65, SD face-to-face = 7.64; M online = 57.46, SD online = 8.99). The effect size was -.14. COMPARISON OF ONLINE CLASS WITH FACE-TO-FACE CLASS IN THE TECHNOLOGICAL APPLICATION SKILLS BEFORE AND AFTER TAKING A TECHNOLOGY COURSE The technological application skills between the online class and the face-to-face class were also evaluated by an independent t test. The results of the t test showed that there was no significant difference in the technological application skills between the two different teaching formats both before and after taking the technology course, t before (60) = -.09, p =.93 (M face-to-face = 33.03, SD face-to-face = 12.28; M online = 33.33, SD online = 15.63); t after (56) = -1.77, p =.08 (M face-to-face = 65.50, SD face-to-face = 8.28; M online = 68.96, SD online = 5.67). The effect size was.49. A summary of these results was tabulated in Table 3. Table 3. Results Summary Pre-Test Post-Test M SD Skewness t p M SD Skewness t p Computer Attitude Online 53.26 10.45.29.15.88 57.46 8.99.08.54.59 Face-to-Face 53.66 10.58 -.18 58.65 7.64 -.02 Technology Skills Online 33.33 15.63.99 -.09.93 68.96 5.67 -.85-1.77.08 Face-to-Face 33.03 12.28.11 65.50 8.28 -.65 Note. M represents mean and SD standard deviation.

Teaching Effectiveness 93 CONCLUDING REMARKS This study showed that the attitudes of the students toward technology use in the face-toface and in the online classes were almost identical both before and after the course. Although no significant changes in terms of students attitudes toward technology use were found, their overall attitudes toward technology use in either type of class showed a very small increase after the course had been taken as compared to before the course. As far as students technological application skill levels were concerned, this study showed that their technological application skill levels for both online and face-to-face classes increased by about a factor of 2 on the post-test as compared to the pre-test. Although the pre-test scores for technological application skill level were similar for both online and face-to-face groups, the post-test scores were slightly higher for the online class as compared to the face-to-face class. To understand the mean difference across these two different groups, the effect size of a finding is its magnitude, generally regarded as an indication of its practical importance [italicized in the original] (Tuckman, 1999, p. 285). The effect size was.49 in the post-test and this indicated that a medium size of effectiveness in the technological application skills was found in the online class. The online teaching method results shown in this study were slightly more effective in increasing the technological skill levels. The results of this study may be elucidated from at least three perspectives. The first perspective concerns the students themselves. With the rapid development of technology as well as its diffusion into all walks of life, today s students have already fostered positive attitudes toward technology use. Therefore, neither method of teaching had much effect on increasing their attitudes toward technology use, but both methods had significant effect on increasing the technological skill levels. Given the same course materials to students in both classes, students in the traditional face-to-face setting may tend to rely more on teacher s assistance. However, students in the online class read the tutorials and related assignment instructions to complete their assignment problems. Such practices may have increased their confidence in technological application skills, and thus, be a reason why the online class showed slightly more effectiveness in terms of enhancing their technological skill levels. The second perspective is that online pedagogy must be fully considered and well designed to suit the requirements of the content and needs of learners (e.g., course planning, monitoring process, managing mechanism, evaluating criteria, learning resources, and learning activities). As such, Salmon s (2003) e-moderating model (i.e., access and motivation, socialization, information exchange, knowledge construction, and development) may serve as a planning tool for online instructors although the assessment and evaluation processes are not discussed in Salmon s e-moderating model. The third perspective relates to learning theory. From a social learning perspective, channels must be open for learner contact with instructors and peers and the provision of timely and critical feedback from both instructors and peers. Even in online learning, face-to-face meetings with instructors and learners are imperative to facilitate both social and academic interactions between student-instructor, student-student, student-content, and student-outside resources. Face-to-face meetings may be arranged to follow the at-the-beginning-middleending-course model. However, such a hybrid/blended learning modal may go beyond a ratio of delivery modalities; it may be perceived more as a pedagogical approach that combines the effectiveness and socialization opportunities of the classroom with the technologically enhanced active learning possibilities of the online setting in addition to its potentialities to cater to individual needs. At this point, socialization is indispensable in any educational mode. A natural occurrence of online socializing may be achieved through the affordances of technology, design, activity, and/or intervention. For example, content-focused learning

94 International Journal of Technology in Teaching & Learning can be achieved through browsing Web pages, tutorials, external links to online resources, or discussion boards, while social contact can be achieved through email, listserv, or chat. These processes, although time-consuming, are necessary steps toward ensuring successful online learning. REFERENCES Aron, A., & Aron, E. N. (2002). Statistics for the behavioral and social sciences: A brief course (2nd ed.). Upper Saddle River, NJ: Prentice Hall. Caywood, K., & Ducket, J. (2003). Online vs. on-campus learning in teacher education. Teacher Education and Special Education, 26(2), 98-105. Definition of Distance Learning (n.d.). Retrieved August 18, 2007, from http://www.usdla.org. Johnson, S. D., Aragon, S. R., Shaik, N., & Palma-Rivas, N. (2000). Comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments. Journal of Interactive Learning Research, 11(1), 29-49. Ladyshewsky, R. K. (2004). E-learning compared with face to face: Differences in the academic achievement of postgraduate business students. Australasian Journal of Educational Technology, 20(3), 316-336. Phipps, R., & Merisotis, J. (1999). What s the difference? A review of contemporary research on the effectiveness of distance learning in higher education. Washington, DC: The Institute for Higher Education Policy. Retrieved December 6, 2004, from http://www. ihep.com/pubs/pdf/difference.pdf Prince, D., & Stern, P. (2002). Online learning students: Characteristics, satisfaction, and implications for future planning. (ERIC Document Reproduction Service No. ED481385) Robertson, J. S., Grant, M. M., & Jackson, L. (2005). Is online instruction perceived as effective as campus instruction by graduate students in education? The Internet and Higher Education, 8, 73 86. Salmon, G. (2003). E-moderating: The key to teaching and learning online (2nd ed.). New York: Routledge Falmer. Sellers, R. G., Wright, K. E., & Wright, V. H. (2003). Attitudes and perceptions of on- and off-campus students of an internet-based graduate allied health program. (ERIC Document Reproduction Service No. ED478777) Tuckman, B. W. (1999). Conducting educational research (5th ed.). Fort Worth, TX: Harcourt Brace College Publishers. van Schaik, P., Barker, P., & Beckstrand, S. (2003). A comparison of on-campus and online course delivery methods in Southern Nevada, Innovations in Education and Teaching International, 40(1), 5-15.