1 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 115 Student Learning in Hybrid French and Spanish Courses: An Overview of Language Online 1 ABSTRACT N. ANN CHENOWETH University of Texas-Pan American EIKO USHIDA University of California, San Diego KIMMAREE MURDAY Indiana University-Purdue University Indianapolis This paper summarizes the assessment results of the Language Online project at Carnegie Mellon University. The study investigated the effectiveness of online language courses for students learning outcomes in four hybrid online language courses (elementary and intermediate levels) and their counterpart conventional (offline) courses from Spring 2000 through Spring Eleven teachers and 354 students were involved in this study, which included five semesters and 34 sections (13 online and 21 offline). Multiple measurements were used to compare learning between online and offline students in oral production, written production, reading comprehension, listening comprehension, grammar knowledge, and vocabulary. Student and teacher feedback, reflecting attitudes and experiences with the online courses, were used to better understand the comparative results. The results from this study indicate that the hybrid online language courses have been reasonably successful: the students in most online courses made progress in their L2 performance similar to that of the students in the equivalent offline courses. Statistical analyses identified two online courses in which the offline students outperformed the online students on several of the learning measures. The qualitative data suggest that students need instructor guidance and that both students and instructors need ongoing technical support for the successful implementation of online language courses. KEYWORDS Online Courses, Computer-assisted Language Instruction, Computer-mediated Communication, Assessment, Online Learning EMERGENCE OF ONLINE COURSES Over the past 10 years, the Internet has been transforming the nature of educational delivery as well as the expectations of education how, when, where, CALICO Journal, 24 (1), p-p CALICO Journal
2 116 CALICO Journal, Vol. 24, No. 1 and in what form we expect to access opportunities for learning (Eseryel, 2002). It is has been claimed that 84% of K-12 classrooms today are connected to the Internet and that more than 90% of K-12 teachers use the Internet in their teaching (reported in Bush & Browne, 2004). In particular, higher education has been moving into the digital age (Harley, 2001, p.10) to respond to societal change while confronting an uncertain future. Online learning activities such as , bulletin boards, database access, and the web are becoming common components in higher education (Kearsley, Lynch, & Wizer, 1995). For example, a special issue of the CALICO Journal in spring 2005 was devoted to computer-mediated communication (CMC) and included reports on a number of CMC-based activities and projects. According to the survey reported in Bush and Browne), 94% of the colleges surveyed use a web-based course management system such as Blackboard or WebCT. As a natural consequence, faculty interest in online teaching is also increasing (e.g., see Monaghan & Santiago, 2001). A number of universities have recently developed online courses and online degrees (for a partial list of these universities, see Appendix A). As we can see from the spring 2006 special issue of the CALICO Journal, devoted to online teaching and learning, we are still in search of an online pedagogy. BENEFITS AND DRAWBACKS OF ONLINE EDUCATION Some of the potential benefits of the use of the Internet in education reported in the US include increased student motivation, enhanced cooperation and collaboration among students, more balanced distribution of power between teachers and students, and increased attention to instruction which caters to individual students needs (Bowers, 2001; Carnevale, 2002; McGrath, 1998). The biggest inherent drawback of online courses is reported to be less interaction between teachers and students (Trotter, 2002). Unlike traditional classes in which teachers use various teaching strategies to engage their students in learning subject matter, online course structures seem to limit the use of such teaching strategies. They rely instead on students self-discipline and responsibility (Gilbert, 2001). That is, it is the students who determine how to study, what to study, and how much time to spend on studying, among other things that they may decide to do (Carnevale, 2002). As a result, students who do not possess, or have not developed sufficient self-discipline or self-regulation in general, do not always take full advantage of participating in online courses. Therefore, the importance of human interaction has been repeatedly emphasized for successful online course delivery regardless of the subject (Gilbert, 2001; Hiss, 2000, Lewis, 2000; White, 2000). An interesting consensus of more experienced online course designers and teachers is that properly designed hybrid courses consisting of both in-class time and online time provide the most beneficial results (Presby, 2001) by combining the advantages of both types of instructional delivery. Such hybrids of online and traditional in-class instruction have become common in university-level online classes such as the Language Online courses investigated in this study. Hybrid courses are believed to be more effective than completely
3 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 117 online courses by providing more guidance and helping students stay focused on their learning, rather than depending entirely on students self-regulation. LITERATURE REVIEW Online and hybrid language courses have become increasingly common as language educators have integrated the various online learning opportunities into their courses. Chenoweth and Murday (2003) describe the characteristics of the online language courses as a unique combination of CALL, CMC, and distance learning environments. Students use the computer to learn course content, as in CALL; communicate with one another and with the instructor both asynchronously and synchronously [using] a wide range of CMC activities; and participate from independent locations, as in distance learning environments. All three elements must be considered to get a full and accurate view of the process of learning that takes place. (p. 291) The following section reviews literature on the development and evaluation of various online language courses. One of the earliest studies (Cahill & Catanzaro, 1997) evaluated the online language course developed at Christopher Newport University. They reported that the students who took the first-year online Spanish course outperformed the students in traditional Spanish courses on writing essays. Flinders University in Australia has offered beginning levels of Italian and French courses online since 2000 (Strambi & Bouvet, 2003). These online courses used both CD-ROM and web-based materials to meet two major challenges associated with distance language learning: (a) sustaining learners positive attitudes and motivation despite the many difficulties they face and (b) maintaining high levels of interaction in the learning environment despite the limited personal contact (p. 85). Despite encountering some technical difficulties, students comments were positive about the use of the online courses. The authors intend to incorporate lessons learned from the pilot implementation into the future development and to conduct further research on students learning outcomes. Blake (2004) described the first implementation of Spanish Without Walls, a first-year Spanish distance-learning course offered through the Extension office at the University of California, Davis. The course uses a combination of materials such as multimedia CD-ROMs, content-based web readings, and an audio chat tool. Blake reported in the results of the preliminary evaluation that students in the distance Spanish course performed similarly to those in traditional Spanish classrooms on grammar tests. The development of Unicode which supports non-roman scripts for multiple platforms has motivated scholars who teach less commonly taught languages to explore online delivery. The National Foreign Language Resource Center (NFLRC) at the University of Hawaii has been conducting distance education and distributed-learning projects since 1995, offering advanced web-based distance courses in East Asian Languages (i.e., Chinese, Japanese, and Korean) not only for students
4 118 CALICO Journal, Vol. 24, No. 1 but also for other individuals, institutions, and businesses. The NFLRC s initial projects attracted professional attention because they offered a prototype for developing similar courses for other less commonly taught language courses. The NFLRC s courses target advanced-level instruction thus far, focusing on skills other than speaking due to technological limitations. However, the development of beginning-level courses has been under consideration with the hope that more advanced technology could offer live distance instruction. The NFLRC continues the development of online language courses and has disseminated information about online projects (see In a similar vein, the special issue of the CALICO Journal in spring, 2004 devoted to Hebrew and Arabic included six out of seven articles on the development of web-based CALL (Bush & Browne, 2004; Corda & Stel, 2004; Foster, Harrell, & Raizen, 2004; Hopp & Hopp, 2004; Nissim, 2004; Pintel, Raizen, Shemer, & Strassberg, 2004). Bush and Browne (2004) described in detail how recent advances in online technology have helped CALL developers to solve various problems such as choices in hardware (PC vs. Macintosh), particularly in dealing with non-roman scripts like Arabic. The authors emphasized the importance of integrating what we have learned in the past about CALL and pedagogical considerations for instructional material development for web-based CALL development. Zhang (2002) discussed the development of a Business Chinese online course at the University of Illinois. As Zhang pointed out, the advantages of web-based instruction for a course on language for specific purposes (LSP) include the fact that materials can be updated regularly using many different resources in response to constant changes in socioeconomic life. In addition, these courses can be offered to other institutions that wish to have a LSP course to supplement their regular language courses. As the NFLRC predicted, the technological advances have made live online instruction a reality. For instance, Wang (2004) found the use of NetMeeting to be the most appropriate tool for interactive language learning in distance mode. She pointed out that sufficient interaction (p. 375) was what has been missing in distance language learning, exploring the issue of oral-visual interaction in distance language education with a creation of a new taxonomy of interaction in CMC. Wang did not mention the pedagogical issues concerning how the NetMeeting tool should be incorporated into the curriculum to create an optimal language-learning environment but instead concluded that whether and how to take advantage of these tools will be largely determined by the requirements of teachers and learners (p. 392). The implementation of a similar tool in an online language course was reported by Hampel and Hauck (2004). The Open University, UK, began implementation of Lyceum, the internet-based audiographic conferencing system for an online German course in 2002 in order to increase opportunities for more flexible speaking practice. After the first implementation, they concluded that the remaining major challenges were on the technical side of online teaching and learning, such as the improvement of audio quality and requirements for computer equipment.
5 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 119 In his preliminary meta-analysis of CALL literature, Zhao (2003) found some empirical evidence that modern technology can help enhance the quality of input and the authenticity of communication and can help provide more relevant and useful feedback. While he found that the current uses of technology in language education are rather fragmented and isolated, implementing online language courses entails a comprehensive and systematic development of curriculum and content to serve the whole course. During the development process, online course designers determine which technologies are more suitable than others for each learning task for the target learners. Zhao emphasized the need for research concerning appropriate uses of technology. Lastly, he pointed out a lack of systematic empirical evaluation efforts to assess the effectiveness of large-scale, comprehensive uses of technology to support language learning, concluding that we cannot ignore the practical question of how and in what ways technology uses are effective in improving language learning (p. 23). As we can see from the above, most published articles on online language courses describe development processes such as background and rationale, product details, and initial implementation results. Some include preliminary evaluation results, obtained mostly from surveys related to participants reactions to the online courses. Although such comments are invaluable for improving the existing courses, it remains unclear how effective online delivery can be in terms of students language learning over time. While Burston (2003) criticized the use of comparative studies to seek significant difference between CALL-based courses and traditional courses, he stressed the need to evaluate how the technology contributes to pedagogical aims in association with measurable outcomes. However, few studies to date have empirically examined the effectiveness of large-scale online language courses focusing on students learning outcomes. This study, therefore, helps address this gap by investigating the effectiveness of online language courses on students learning outcomes using data collected from four hybrid online and conventional offline language courses (elementary and intermediate levels of French and Spanish) from Spring 2000 through Spring The study addresses the following research questions: 1. Are there differences among the groups of students enrolled in online and offline courses in terms of gains in language skills? 2. Are there differences among the groups of students enrolled in online and offline courses in terms of satisfaction? 3. Are there differences among the instructors of the online and offline courses in terms of satisfaction? In this article we focus on the first research question concerning linguistic gains. THE COURSES IN BRIEF Carnegie Mellon University has been delivering Language Online courses in elementary and intermediate French and Spanish classes since spring The Language Online courses attempt to avoid the constraints of time and space associated with traditional instruction, making it possible for students whose sched-
6 120 CALICO Journal, Vol. 24, No. 1 ules are often occupied with laboratory, project, or studio courses to take lower level foreign language courses. The same basic framework is used in all of the Language Online courses and has evolved slightly over the course of the development process. Each class meets face to face for 1 hour one evening per week. In addition, students are required to meet face to face with their instructor or a language assistant (typically an upper division undergraduate native speaker or language major) for 20 minutes each week on a rotating basis. Students are also required to participate in a weekly chat session scheduled at a time convenient for them, usually late evenings. These chat sessions began as 1-hour sessions which were typically moderated by the language assistant (and less often by the instructor), but these sessions have generally become more task based so that, when the participants finish an assigned task, they are often free to log off. A typical chat session in this new format may not even last 20 minutes but is more focused on the task at hand than the earlier ones, during which students often put in their time while eating dinner, watching TV, or doing other activities. All course materials are provided online, with self-check exercises (developed with Hot Potatoes), and written assignments to be turned in via or posted to the course bulletin board. The quizzes, chats, and bulletin board postings are managed through WebCT (for a more complete description of the courses, see Chenoweth & Murday, 2003). The conventional elementary (offline) courses meet 4 days per week, and the intermediate offline courses meet 3 days per week, 50 minutes per class session. These classes typically require students to buy a textbook, instructors make frequent homework assignments, but there are no regularly scheduled meetings outside of class with the instructor, language assistant, or classmates. 2 The syllabus for the online and offline courses is essentially the same; however, teachers wanted some freedom in designing their courses, so they were able to choose the topics which made up the cultural component of their course. ASSESSMENT: ARE THE ONLINE COURSES WORKING? To test the design and implementation of the Language Online courses, we compared L2 learning in the online courses with L2 learning in the conventional (offline) courses. Although the assessment of Elementary French I indicated that the students in the hybrid online course made sufficient and comparable progress in overall language skills compared to their counterparts in the conventional course (Chenoweth & Murday, 2003), we continued the assessment throughout the entire active development phase of the Language Online project. Eleven teachers and 354 students across 34 sections were involved in this study which took place over five semesters (see Table 1 for the timeline and the number of sections included in each semester of data collection). We also investigated student and teacher attitudes and experiences with the online courses to help us better understand the results of the comparisons of L2 learning. In this article we focus on the assessment of student learning and draw on part of the data from the focus groups and interviews to help us interpret those results.
7 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 121 Assessment Design To make the L2 learning comparisons between the online and offline sections, we developed our own measures which better reflected the course content and had higher face validity among the teachers involved in the project than would measures developed in other contexts for other purposes. We created final exams for individual sections in common in order to compare student learning in listening comprehension, reading comprehension, grammatical knowledge, vocabulary, and written production. Teachers in the courses at each level and each language collaborated on developing the test items, which were then checked by a member of the assessment team for validity and consistency. 3 The final exams were initially graded by the course instructors so that they could calculate the course grades, but, for the purposes of the course assessment, they were scored again by members of the assessment team to insure consistency in scoring across sections. Table 1 Course Implementation and Assessment Timeline Elem. French I Elem. French II Int. French I Int. French II Elem. Spanish I Elem. Spanish II Int. Spanish II Int. Spanish II Spring 2000 Fall 2000 Spring 2001 Fall 2001 Spring 2002 On Off On Off On Off On Off On Off Note: on = online; off = offline The essays from the final exam were given to independent raters (after the ones written by the offline students had been typed) for scoring using a scale developed in consultation with the teachers. We developed our own 5-point rating scale (Chenoweth & Murday, 2003) based on criteria that the teachers ordinarily used in grading student essays: topical relevance, overall development, organization/ ordering principle, the use of transitions/cohesive devices, vocabulary, syntactic complexity/variety, grammatical accuracy, mechanics (e.g., spelling, punctuation, accents, and capitalization), and length. This rubric had been successfully used
8 122 CALICO Journal, Vol. 24, No. 1 to detect differences in student writing in the evaluation of online Elementary French I (Chenoweth & Murday, 2003). Oral production was compared at two different points of the semester: one near the beginning (in week 3 for most levels, after the enrollment period for the semester, except Elementary I, which was conducted in week 5 to allow first semester students oral language skills to develop) and the other at the end of the semester. Pairs of students met with their teacher and completed two or more simple tasks such as role plays or picture descriptions. These sessions were audio taped and later evaluated by independent raters using the oral production rating scale developed by Payne and Whitney (2002), which measures comprehensibility, fluency, vocabulary usage, syntax and grammar, and pronunciation. The early interview/role play was used as a baseline of student capabilities, and the final interview/role play was used as a measure of oral language gain during the semester. Because we could not randomly assign students to the online or offline sections, we collected demographic data at the beginning of each semester. If one group of students outperformed the other on the learning comparison measures, we wanted to be able to control for outside factors which could have had an impact on learning such as previous experience with the language, a high degree of motivation to learn the target language, and so forth. We developed two questionnaires for this purpose: a general background questionnaire, which asked questions about students previous language-learning experiences and reasons for taking the course and a technology background questionnaire, which asked students about their experiences and use of computers and the Internet (see questionnaires in Appendix B). Finally, we obtained students SAT verbal scores from the Registrar s office. We were also interested in students and teachers satisfaction with the online course, for both formative and evaluative purposes. We used the university s official faculty course evaluations, a modified version of the foreign language department s supplemental course evaluations, and conducted interviews and focus groups (Patton, 1990) with the teachers and with groups of students to gain insight into their experiences in the courses. The Students The information from the Registrar s office and the background questionnaires revealed that the students enrolled in the sections had similar characteristics: the mean age of the students was 21, they were experienced language learners (73% of the students reported having learned or studied another language), they had similar SAT verbal scores (mean of 623), and similar proportions of undergraduates, graduate students, and staff members were enrolled in each section. The undergraduate students carried similar course loads (mean of 49 units [about five classes]). Females made up 41% of the online courses and about 53% of the offline courses. The most common reasons students reported for taking a language course were I like learning languages (61% offline and 55% online) and to study abroad (37% offline and 34% online); equal proportions of offline and
9 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 123 online students reported taking the courses to fulfill requirements for their majors (32% offline and 32% online). Most online students reported that they had enrolled in the online course due to schedule conflicts (57%) and because they would not be expected to attend class 4 days a week (61%). This result is not surprising given the need for more flexible scheduling which had necessitated the development of the online courses. Most offline students reported that they chose the traditional format for the course because they wanted more guidance from the teacher (69%) and because they like talking with people in class (56%). Most students (66%) reported being comfortable working on computers, and 73% had their own. The offline and the online students reported spending about the same amount of time on computers per day (66% reported 4 hours or less on average, and another 20% reported 5 to 6 hours per day). COMPARISON OF LEARNING We compared students performance on the common sections of the final exams (i.e., written production, reading comprehension, listening comprehension, control of grammar, and, in the Spanish courses, vocabulary knowledge 4 ) and the oral production measures to determine whether the online courses were as effective as the offline ones. Although we found a few statistically significant differences between scores, the overall finding which emerges from analysis of the final exams and oral production measures is that student learning in the online and offline courses appears to be comparable, even for oral communication, the area we were most concerned about. Summary tables of the assessment results are provided in Appendix C. Students performance on selected measures is described below. Oral Production Students were interviewed in pairs for the oral production measure in order to put students more at ease and to reduce the time demands on the teachers. For each level, the French and Spanish students were given the same task. The Elementary I students were asked to complete three tasks. The first was a role play in which students assumed the roles of meeting a new friend at a party and engaged in a conversation to get to know each other. The second task directed the students to talk about a picture together; each pair of students was given the same picture, and the students were told they could invent information about the people in the picture if they wished. The third task was to tell the story of what Jean-Claude [Javier] does on a typical summer day, again using a picture as a basis for the conversation. The Elementary II students had to complete two tasks together. The first involved discussing a situation (such as comparing whose roommate was more problematic), and the second was to discuss a picture (similar to the second task for the Elementary I students). The Intermediate I students were asked to do two tasks. The first asked them to make recommendations concerning what a mutual friend who was suffering from stress should do and then evaluate and defend their recommendations when they conflicted with their partner s recommenda-
10 124 CALICO Journal, Vol. 24, No. 1 tions. The second task was a role play in which students were asked to describe themselves, either to a headhunting company or a telephone dating service. The Intermediate II students had to complete two tasks: the first was to talk about and compare their experiences in high school with their experiences at Carnegie Mellon, and the second was to talk about what other people want them to do with their lives, what they want to do, and how they might deal with any conflicts between their own desires and the outside influences. These interview/role plays were tape recorded and later analyzed by independent raters who were not aware of which point in the semester the tape had been made nor whether the students were in the online or the offline course. Each student s performance (students identified themselves at the beginning of the tape) was rated on separate 10-point scales along five dimensions (comprehensibility, fluency, vocabulary usage, syntax and grammar, and pronunciation) (Payne & Whitney, 2002). Due to the small sample size and because we could not assume a normal distribution, these data were analyzed using Mann-Whitney U (see Table 1 in Appendix C). The only statistically significant differences were found in the comparison of the oral production scores of the online and offline students at the end of the semester for the Intermediate Spanish I class in Fall 2001 (see Figure 1). Figure 1 Students Oral Production Profile (Fall 2001, Intermediate Spanish I) pronunciation online syntax offline vocabulary fluency comprehensibility The overall median score was 37 for the online students and 31 for the offline students; the online students outperformed the offline students in fluency, comprehensibility, and control of syntax and grammar. No other statistically significant differences were found for the beginning or end-of-semester scores, nor for beginning versus end-of-semester scores, for any of the courses involved in the study. A more representative sample of the results (from Elementary French II, Spring 2001) is shown in Figure 2.
11 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 125 Figure 2 Students Oral Production Profile (Spring 2001, Elementary French II) pronunciation online syntax offline vocabulary fluency comprehensibility The median score for the offline course was 24, compared to 27 for the online group. The online students had higher scores than the offline students in comprehensibility, vocabulary, syntax and grammar, and pronunciation. Analysis of students performance on the oral production tasks at the beginning of the term revealed that the two groups had very similar skills and abilities (the median scores for fluency were the only scores that varied; the offline students were rated at 5, while the online students were rated at 4.5). The only improvement from the beginning of the semester for the offline students in Elementary French II was in pronunciation, where they went from a median score of 4.5 to 5; the online students on the other hand, showed a slight improvement in all areas. Written Production The students were asked to write short essays on their final exams. In Elementary French I, students wrote a short essay introducing themselves and their family to a future French exchange student. Elementary Spanish I students wrote a three paragraph essay about vacations: past, present, and future. In Elementary II (French and Spanish), students were asked to write about what they thought their life would be like in 20 years how old they would be, what they would be doing, where they would be, and so on. Spanish students also wrote a second essay giving advice to an incoming college student. In Intermediate French I, students were asked to write a letter to (future) home-stay parents, introducing themselves, talking about their plans for the semester, and what they were interested in doing in the particular region of France they had selected to go to. Data for the Intermediate Spanish I students were not available. In Intermediate French II, students summarized a film they had seen that semester which had had an impact
12 126 CALICO Journal, Vol. 24, No. 1 on them and discussed what the impact was and why the film had affected them in that way. Intermediate Spanish II students wrote about the future experience of a student attending college in 10 years. Students responses to these prompts were analyzed by raters who were unaware of whether the student was enrolled in the online or offline course (which meant that we first had to type the offline students responses and print out the online students responses). The raters used 5-point scales to rate the essays on the following dimensions: relevance to topic, overall development, organization, use of transitions and cohesive devices, use of vocabulary, syntactic complexity and variety, grammatical accuracy, mechanics, and length. Again, because of the small sample size and because we could not assume a normal distribution, the raters judgments were analyzed using Mann- Whitney U (see Table 2 in Appendix C). Statistically significant differences were found in the comparison of the written production of the online and offline students for the Elementary French I students in Spring No other statistically significant differences were found. In a study investigating the first iteration of French Online in Spring 2000, Chenoweth and Murday (2003) found that the online students scored significantly higher in the essays than the offline students. The authors speculated that result could be due to the practice that the online students were getting in the chat sessions which trained them to write French quickly in order to keep up with the flow of the chat conversation. However, this reasoning does not explain why the students in subsequent semesters did not receive higher scores than their offline counterparts. Figure 3 shows the written production profile of the students who took Elementary French II in Spring Figure 3 Students Written Production Profile (Spring 2001, Elementary French II) length mechanics accuracy syntax online offline vocabulary transitions organization development on topic
13 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 127 The median score was 33.5 for the online group and 34.0 for the offline group. The essays written by the offline students were judged to be somewhat more on topic and their use of vocabulary grammatically was judged to be somewhat better, while the online students were rated more highly on their use of transitions. Reading Comprehension For the reading comprehension section, students typically read a short passage and responded to short-answer and multiple-choice questions to measure their comprehension. Data are not available for the Elementary French I course in fall 2001 because the reading comprehension sections of the two finals were not equivalent (some additional context was added to the online version when it was programmed into WebCT), nor are they available for the Intermediate Spanish courses in fall 2001 or spring 2002 (see Table 3 in Appendix C). Statistically significant differences were found in one of the two iterations of Elementary French I (t = , p =.006) and in one of the two iterations of Elementary Spanish II (t = 3.252, p =.003); in both cases, the offline students outperformed the online students. Listening Comprehension In the listening comprehension sections, in Elementary I for both languages, students responded to short-answer questions based on a dialogue between two people and also answered oral questions based on pictures or short personal questions and questions based on a short passage. In Elementary II French and Spanish, students listened to several descriptions and chose the one that best described a picture, and they also wrote answers to orally presented questions about themselves or listened to a dialogue and answered true-false and short-answer questions. In the Intermediate courses for both languages, students typically listened to a passage and answered multiple-choice questions and also wrote answers to orally presented questions about themselves (see Table 4 in Appendix C). The offline students outperformed the online students on these sections in two of the Spanish courses: the fall 2000 iteration of Elementary Spanish I (t = 3.109, p =.005) and the spring 2001 iteration of Elementary Spanish II (t = 2.210, p =.036). In other iterations of these courses and in the other intermediate and elementary courses, students in the online and offline courses performed at about the same level on these sections. Grammatical Knowledge In Elementary I French and Spanish, students knowledge of grammar was measured by analyzing the grammatical accuracy of their responses to a set of written questions. Responses included negation, agreement, verb forms, verb tenses, and so forth. In Elementary I Spanish, responses also required use of preterit versus imperfect. In Elementary II French and Spanish, the grammar section had two parts. In the first part for Elementary II French, students were presented with multiple-choice questions which tested their knowledge of pronouns, verb tenses,
14 128 CALICO Journal, Vol. 24, No. 1 and verb usage. In the second part, students read a passage and put present tense verbs into either the imperfect or the passé composé depending on the context. In Elementary II Spanish, students completed hypothetical phrases with the past subjunctive or the conditional in the first part and then wrote formal and informal commands in the second part. In the Intermediate French classes, students were tested on their control of pronouns while answering questions; differences between the passé composé and the passé simple; and use of the conditional, passive voice, and indirect speech. Intermediate Spanish II students completed sentences that tested their control of various tenses and conjugations, including present, subjunctive, conditional, past subjunctive, and the infinitive (see Table 5 in Appendix C). The offline students outperformed the online students on these sections in three of the Spanish courses: the fall 2000 iteration of Elementary Spanish I (t = 3.163, p =.004), the spring 2001 iteration of Elementary Spanish II (t = 2.297, p =.030), and the spring 2002 iteration of Intermediate Spanish II (t = , p =.001). Students in the other online and offline classes performed about equally well on these items (although data are not available for Intermediate Spanish I). These results are consistent with the raters judgments on the written essays in which the students control of grammar was rated about the same for online and offline students. Vocabulary The Elementary Spanish students were assessed on their vocabulary knowledge. 5 Elementary Spanish I students answered questions based on emotional state vocabulary as well as common objects found in various places. Elementary Spanish II students listed food items for a restaurant menu and symptoms of and cures for the flu (see Table 6 in Appendix C). The offline students in the fall 2000 iteration of Elementary Spanish I outperformed the online students (t = 2.196, p =.037), but the students in the other semesters and the other classes performed equally well on the vocabulary sections. Summary of Findings The comparisons of students performance on common sections of the final exams and on the oral production tasks need to be interpreted carefully because this was not a controlled study in which we could randomly assign students to courses (conditions) and have the same teacher teach both sections (a way to control for teacher effects). Additionally, class size, particularly for the online courses, were generally small. Nevertheless, the results serve as an indication that we have been reasonably successful in creating online courses: we found that student performance on most measures was generally comparable in the online and offline courses. Further, teachers have not reported any differing abilities of the online students abilities in classes in subsequent semesters. Most of the statistically significant differences between students scores occurred in the Elementary Spanish I courses in fall 2000 and the Elementary Spanish II courses in spring Since the demographic data collected at the beginning of the semester did
15 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 129 not reveal significant differences between the online and offline students which would help us understand these differences, we turn now to the focus group and interview data to try to account for these results. The discussion which follows focuses on the Spanish courses as a window on the differences in the performance of some of the Spanish students. A fuller treatment of the qualitative data from both the French and Spanish courses will be forthcoming. STUDENT FEEDBACK Six students were enrolled in Elementary Spanish I in fall 2000; two of these students continued the next semester in Elementary Spanish II and were joined by four other students new to the online environment. When we sifted through the comments both favorable and unfavorable that students made in the focus groups, what emerged was that for these students in Elementary Spanish I and II, the transition to learning in an online course was a challenge. The students in Elementary Spanish I found using the online materials frustrating, particularly in the beginning of the semester; they wanted a textbook and found the work plans (intended to guide their studies) confusing. For the students in Elementary Spanish II, major factors seem to be related to study materials and practices. For the Elementary Spanish I students, frustrations arose mostly in conjunction with technological problems, as the following quotations from students in the focus group session held around midterm illustrate: Some people can t type the accents. That s coming up on the tests, because the computer grades it, and in a real class environment you wouldn t get a whole question wrong if you write a whole sentence and you miss one accent. That bother[s] me. That happens with the exercises too. It s frustrating. You re trying to learn, and [the computer] keeps saying incorrect, incorrect, and you think you ve got everything, and it s the period, or you don t need the period [at the end of a sentence]. It seems like the modern languages world is Mac based, but the rest of the world isn t. So our recording software doesn t work right. I can t use the format so my assignments will probably never be graded. [...] For example, I have to make my recording five times, and it s not because the Spanish is wrong, it s because the recorder [malfunctions]. If I was in class I could just say it once. These issues became less problematic as students and the teacher became more familiar with the web site and were even less of a problem the following semester when the teacher was already familiar with the materials and web site before the beginning of the semester. Another particularly salient challenge for students involved the switch from a traditional textbook to electronic hypertext: they missed the convenience of having a textbook in hardcopy format. The electronic materials were very rich with images, audio files, roll-over translations, and interactive exercises which could
16 130 CALICO Journal, Vol. 24, No. 1 be used in very flexible ways. However, students missed the familiar structure of a traditional textbook that they could open to a chapter and work through in the order laid out for them. They perceived the web site to be relatively unstructured compared to a textbook. The following comments from students in Elementary Spanish I reflect their initial problems engaging with course materials: I didn t really know what to do first, so I started with vocabulary and then went on to the grammar. It took me a while to realize that it s better to start with the grammar and just use the vocabulary list as a reference. Yeah, a textbook would be organized; but hypertext is unstructured. It s difficult to know where to start. Most students found it necessary to print out a large part of the course materials so that they could refer back to them more easily, particularly during chat sessions when it was inconvenient to open another window in order to access a phrase or grammatical structure they wanted to use. Students also wanted to be able to highlight important sections and to study them outside the class on the bus or other places where computers were not available. They requested printer-friendly versions of the materials for these reasons, even though they would lose some of the learning support (glosses and interactive exercises) in the printed version of the materials. Because of the relatively unfamiliar electronic learning environment and the relatively infrequent class meetings, students consistently reported that they needed more help from their teachers and the web site to be able to figure out what was due when. One student commented that the work-plan doesn t connect lessons with exercises. In a class that meets three or four times a week, there would be frequent reminders from teachers about assignments, but, because of fewer contact hours, this was not always the case with the Language Online courses (although some teachers did post messages to the class bulletin boards or send reminder messages to students). At the early stage of the courses, the work plans tended to be general, with assignments listed per content module or by the week. Even though the assignments that students were to complete were described in the modules, students either could not find them or did not access them, or they failed to get an overall sense of what they needed to study. Students wanted, and needed, a much more specific work plan with specific due dates, both to guide them in their study of the materials and to help them maintain a certain pace so that they would regularly engage with the course and not fall behind or have to rely on cramming to prepare for an exam. Many students stated that they usually ended up studying Spanish right before something was scheduled: an exam, a chat session, meeting with the instructor. I usually cram right before I have to meet with our professor, for that 15 minute chat. I read what we have to talk about it, and for the minutes beforehand I cram. So I figure out all the -ar verbs I need to know in order to speak like a normal human being, and then I ll study before a test. But besides that, that s about it.
17 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 131 I actually listen to the dialogs a lot. I just listen to them, look at things. I don t really focus on a lot. I haven t done most of the exercises, even though I ve been to the web site and played around with a lot of different things. But again, it s the same as [for a classmate], I ll just do them until I get them right before the exam. I just kind of relax and go through it. It s pretty regular. You have to be ready for the chat, and for the class. One of the things people think of when you think about stuff being available on the internet is that you can get to it anywhere. But you can t get this sort of anywhere. It would be ideal if we had a recommended cassette and you could play it in your walkman when you re walking between classes, but you have to actually set aside time when you think ok, I m going to sit down and do this Spanish stuff. In a regular classroom, you go in for an hour a day, and you re immersed in it. With more tailored and specific work plans, students could be guided to distribute their studying throughout the week, and not just before something was due. One student reported that he did not know what was required and asked are there due dates? He suggested making a checklist of assignments and their due dates so that it would be clear what was due when. Another student agreed but added that while specific assignments in a specific order would be helpful, some flexibility on the due dates would also be helpful. Because these courses do feature some flexible scheduling, replacing in effect self-paced courses, a structured schedule with some built-in flexibility seems consistent with student expectations and perceptions. In addition, such a schedule could also help to mitigate the problems with navigation and study strategies mentioned above that come with the unfamiliar hypertext learning environment. However, even with improved work plans, and printed materials, students will still need to learn to adapt to studying online; one of the Elementary Spanish II students reported that there are lots of distractions when using the web compared to studying with books. TEACHER FEEDBACK The teachers also had to make substantial adjustments to teaching the Language Online courses. First, they had to learn what was offered in the materials, how to navigate them, how to keep records, and how to do basic things like post messages on the bulletin boards and access the chat rooms. These introductory topics were covered in an orientation session held before the semester began, and support staff were available for trouble shooting throughout the semester. The teachers also had to figure out how much of the materials to cover and how best to use the weekly class meeting: to review, to present new or supplemental material, to get students to interact with each other, to remind students about what to do, to schedule appointments, or to teach students strategies for studying a foreign language online? Teachers still confronted these issues the second or third time that they taught online, but each time they reported being more comfortable with the materials and the technological aspects of the course. However, in fall 2000 and spring 2001, both of the teachers were teaching the online courses for the
18 132 CALICO Journal, Vol. 24, No. 1 first time (and while one teacher was very familiar with the web site and technologically proficient at the beginning of the semester, the other teacher, who taught Elementary Spanish I, was not and reported that she learned WebCT as she went). The Elementary Spanish I teacher also reported that she did not feel like she was in charge of the course. She lacked the technological expertise to modify the work plans, and she felt obligated to assign all the material on the web site since it had taken so much effort to create. These are problems that never arose for her when using a textbook. The Elementary Spanish II teacher seemed to have fewer problems adapting to the Online course but reported that there seemed to be an insufficient number of exercises on the web site, which could definitely have impacted students learning. Students themselves commented that they wanted more interactive exercises and more links to the material covered in the Elementary Spanish I course, lending weight to the teacher s perceptions. The feedback from the teachers and students was passed along to the course developers at the end of each semester. In subsequent iterations of the courses, changes were made to the work plans, materials, and exercises to address some of the problems that had been identified. The courses are thus continually being modified and improved. Implementing the online courses has been a complex process, replete with developing understanding of instructional and technical issues and continually evolving goals. DISCUSSION AND CONCLUSION The results of this study indicate that the students in the hybrid online courses made similar progress to the students in the equivalent offline courses. These results are extremely encouraging since they provide support for alternative delivery methods for courses that often are unavailable to students with schedule conflicts. The issues in the Elementary Spanish courses (and to a lesser extent in the French courses), however, highlight some important issues that must be considered when implementing online language courses. The first issue involves technical problems. While some technical problems could be considered a natural part of a newly created program, the fact remains that it is never easy to maintain completely reliable technology. It is critical for student adaptation and satisfaction that adequate training and support be provided not only at the beginning of the semester, but also throughout the course. Technology support must also be provided for teachers; the instructor must be comfortable with the materials, both in terms of content and technology. Teachers are generally the first point of contact for students when problems arise and must be able to address their problems and concerns as much as possible. The second issue is the need for general guidance. For example, the students in the first iterations of Elementary Spanish reported not always knowing what assignments were due when. As a result of their confusion, steps were taken in subsequent iterations of both online Spanish and French courses to make sure that the work plans were clear and that students understood where to look for assignments and how to keep track of due dates. Teachers must make more of an
19 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 133 effort than might be necessary in a traditional class to make sure that students are studying the proper materials and keeping up with assignments. Students must also understand that, when signing up for a hybrid course, they will need to make an effort to study regularly and not allow themselves to fall behind. There are limitations to the generalizability of the results of this study. In addition to the fact that students were not randomly assigned to courses, the students at Carnegie Mellon University are particularly known for their comfort with cutting edge technology. These students might be unusually quick to adapt to using new technology in a course. The results may have been different at a less technocentric university. Further research is needed with hybrid courses to establish their appropriateness in different contexts (e.g., community colleges, large and small universities, etc.) and with different hybrid conditions (e.g., more or less class time, different assignments, etc.). Future study designs might also include a follow-up retention study in order to measure what students remember several months after they have completed the course. One of the most critical aspects to such research is the need to collect both qualitative and quantitative data. In our study, the quantitative data would have been much more difficult to interpret without the perspective of both the students and the teachers involved. In addition to using multiple methods of data collection, the assessment measures were designed to be minimally intrusive and pedagogically sound with high validity for both instructors and students. Since assessment was an integral part of the project from the beginning, the results could be used from the very first iterations to improve the materials and implementation of courses. This effort, we believe, is key to the long-term success of the hybrid online language courses. NOTES 1 This study was supported, in part, by a grant from the Andrew W. Mellon Foundation to Christopher M. Jones and G. Richard Tucker of the Department of Modern Languages at Carnegie Mellon University. 2 Although some offline courses, particularly at the intermediate level, make use of a writing assistant (typically an upper division undergraduate native speaker or language major), students in these courses typically meet with their writing assistant no more than once or twice during the entire semester. 3 As the materials for each section were not exactly the same, the teachers were free to add additional materials to their own class s final exam to reflect areas that might have received more emphasis in their section 4 The French teachers decided that students vocabulary knowledge was adequately assessed by the oral and written production measures for their sections. 5 The French teachers decided that vocabulary was adequately assessed by the oral and written production measures.
20 134 CALICO Journal, Vol. 24, No. 1 REFERENCES Blake, R. (2004). Evaluating Spanish language teaching at a distance. Paper presented at CALICO 2004, Carnegie Mellon University, Pittsburgh, PA. Bowers, P. (2001). Discovery-based learning: Lessons in wireless teaching. Syllabus, 14 (6), Burston, J. (2003). Proving IT works. CALICO Journal, 20 (2), Bush, M. D., & Browne, J. M.(2004). Teaching Arabic with technology at BYU: Learning from the past to bridge to the future. CALICO Journal, 21 (3), Cahill, D., & Catanzaro, D. (1997). Teaching first-year Spanish on-line. CALICO Journal, 14 (2-4), Carnevale, D. ( 2002). Online students don t fare as well as classroom counterparts, study finds. The Chronicle of Higher Education. Retrieved February 25, 2002, from Chenoweth, N. A., & Murday, K. (2003). Measuring student learning in an online French course. CALICO Journal, 20 (2), Corda, A., & Stel, M. (2004). Web-based CALL for Arabic: Constraints and challenges. CALICO Journal, 21 (3), Eseryel, D. (2002). A framework for evaluation and selection of e-learning solutions. (ERIC Document Reproduction Service No. ED47954) Foster, J. Q., Harrell, L. F., & Raizen, E. (2004). The Hebrewer: A web-based inflection generator. CALICO Journal, 21 (3), Gilbert, S. D. (2001). How to be a successful online student. New York: McGraw-Hill. Hampel, R., & Hauck, M. (2004). Towards an effective use of audio conferencing in distance language courses. Language Learning & Technology, 8 (1), Retrieved April 9, 2006, from Harley, D. (2001). Higher education in the digital age: Planning for an uncertain future. Syllabus, 15 (2), Hiss, A. ( 2000). Talking the talk: Humor and other forms of online communication. In K. W. White & B. H. Weight (Eds.), The online teaching guide (pp ). Boston: Allyn and Bacon. Hopp, M. A. & Hopp, T. H. (2004). newslate: Building a web-based infrastructure for learning non-roman script languages. CALICO Journal, 21 (3), Kearsley, G., Lynch, W., & Wizer, D. (1995). The effectiveness and impact of online learning in graduate education. Education Technology, 35 (6), Lewis, C. (2000). Taming the lions and tigers and bears: The WRITE WAY to communicate online. In K. W. White & B. H. Weight (Eds.), The online teaching guide (pp ). Boston: Allyn and Bacon. McGrath, B. (1998). Partners in learning: Twelve ways technology changes the teacherstudent relationship. T.H.E. Journal, 25 (9), Monaghan, J., & Santiago, R. S. (2001). Critical examination of the use of online technologies in diverse courses at a large comprehensive university. (ERIC Document Reproduction Service No. ED470104)