Irony and Asynchronicity: Interpreting Withdrawal Rates in E-Learning Courses

Transcription

1 Irony and Asynchronicity: Interpreting Withdrawal Rates in E-Learning Courses Norbert Elliot New Jersey Institute of Technology Newark, New Jersey Robert S. Friedman New Jersey Institute of Technology Newark, New Jersey Vladimir Briller New Jersey Institute of Technology Newark, New Jersey Abstract: This paper presents findings from two E-Learning studies conducted at New Jersey Institute of Technology. The first, a large-scale quasi-experimental study, centers on student performance in E-Learning and traditional courses offered within the College of Computing Science. The second, a small-scale case study, centers on student performance in an E-Learning course in world literature. Although the content of the courses rests in different disciplines, similarities in student performance are identified regarding academic preparation and perseverance of students. To provide a framework for interpretation of these results, the authors turn to literary criticism and the interpretative framework of irony. Introduction: E-learning at New Jersey Institute of Technology New Jersey Institute of Technology has long been involved in E-learning. With 5,366 undergraduate students and 2,883 graduate students enrolled in the fall of 2004, NJIT serves the state as one of New Jersey's three research universities involved in the design and delivery of asynchronous networks. While the state's other two research universities Rutgers, The State University of New Jersey and the University of Medicine and Dentistry of New Jersey engage in E-Learning, NJIT has identified both a division (The Division Continuing Education) and an evaluative component (The Office of Institutional Research and Planning) as infrastructure to support its efforts. As a result of its intensive involvement in e-learning, NJIT offers five complete undergraduate and graduate degree programs that can be taken in an E-Learning (EL) environment, in addition to fourteen graduate certificates. Over 150 EL courses are offered by NJIT faculty to support these programs. The College of Computing Sciences hosts the largest cohort of EL courses on campus. The College of Science and Liberal Arts hosts a smaller set of courses. Both colleges are committed to delivering high quality instruction, and both are committed to an empirical base for decision-making regarding the evaluation of these courses. A key concern for administrators and faculty in both colleges centers on the existence of lower passing rates and, subsequently, lower time to completion and graduation rates for students enrolled in EL courses. Nationally, retention is a recurring issue, a master trope in discussions of the EL. Low perseverance rates are often noted and often found to fall in disproportional ranges when compared to traditionally taught courses (Institute for Higher Education Policy, 1999). A recent study regarding perseverance in an EL introductory computer science course, for example, noted that the "dropout rate" (i.e., a student who submitted some work and then either withdrew from the course of failed to take the final examination) was 42%, a rate much higher than the 12% and the 26% for the same class, taught in prior years by the same instructor in a face-to-face format (Mock, 2003). While it is true that current studies using meta-analysis techniques demonstrate that e-learning does not diminish the level of student satisfaction, it is also true that such studies tell us little about the performance of students in these classes (Allen, Bourhis, Burrell & Mabry, 2002). And it is that performance that most concerns educators. Study Design: E-Learning and Face-to-Face Courses in Two Disciplines

2 Quasi-Experimental Design: Student Performance in the Computer and Information Science Program During the spring of 2003, NJIT's Office of Institutional Research and Planning extracted the records of all students then enrolled in courses offered by the Department of Computer Science and the Department of Information Systems. Administrators in the College of Computing Sciences, where the two departments reside, had developed the perception that students in their EL courses had lower passing rates and, thus, lower time to completion and graduation rates, than students taking traditional face-to-face (FTF) courses. Student records were extracted from fall of 1996 to spring of The total sample of students for the FTF courses were 15,468; the total sample of students for the EL courses were 2,544. Because students could not be randomly assigned to either EL or FTF courses, the study is best classified as quasi experimental in design. The study asked six questions: (1) Is there a difference in academic preparation between students who enrolled in EL and face-to-face (FTF) modalities? (2) Is the withdrawal rate from the DL courses different from FTF courses based on cumulative GPA and SAT scores? (3) Is there a relationship between students GPA and their performance in EL courses? (4) Are students who failed the CIS EL courses more likely to pass if they repeat those courses in EL or FTF format? (5) What would be the impact of policies establishing required cumulative GPAs for enrollment in CIS EL courses? (6) Can we determine any criteria for success from comments given by students for dropping EL courses? The average SAT score for the students who enrolled in the EL courses was 1097, and the average score for students who enrolled in the FTF was Hence, with the acknowledged limits of using SAT scores as a measure of academic preparation, no difference was found. There was, in addition, no difference in SAT scores for students who withdrew from the EL courses (average score: 1095) and who complete the EL course (average score: 1099), although the SAT scores of students who completed the FTF (average score: 1131) appeared to be slightly higher than those who withdrew (average score: 1098). If the academic preparation was similar in a comparison of EL and FTF undergraduate courses, the withdrawal rate revealed differences, as Table 1 demonstrates: CIS Courses G P A Total < < < 1.5 Seats < < < Total enrollment Number withdrew Percent withdrew 10.5% 23.2% 27.1% 18.2% 15.2% 11.2% 7.5% 3.8% EL enrollment Number withdrew Percent withdrew 16.5% 66.7% 70.0% 29.1% 23.6% 16.5% 13.7% 7.6% FTF enrollment Number withdrew Percent withdrew 9.6% 16.7% 21.3% 16.6% 14.1% 10.3% 6.5% 2.9% Table 1: Enrollment and Withdrawals from CIS courses by current GPA range by Mode of Study Overall, the withdrawal rates from the EL courses were higher (16.5%) than from the FTF courses (9.6%). In addition, the withdrawal of students with GPA s of 2.5 and below was markedly higher for EL courses and markedly higher than for the students with GPA s above 2.5. (It is, of course, important to note that the numbers of students with GPA s below 2.5, taking EL courses, is too small in this sample to allow conclusions to be drawn. However, results suggest that it is important to monitor this indicator.) The withdrawal rate, in addition, cannot be linked directly to the student GPA; one possible explanation for withdrawals by the students with high GPA s could be that they believe that their grade is not going to be as high as they expected and could damage their GPA. Answers to the third research question revealed that there was a statistically significant moderate-to high correlation between student GPA and grades in both EL (.542, p<.0001) and FTF (.532, p<.0001) CIS courses. Withdrawal and

3 failure rates at all GPA ranges were, again, found to be higher for DL students than for FTF students. For the students at all GPA ranges who successfully completed EL and FTF courses, the grade distribution was similar. Are students who failed the CIS EL courses more likely to pass if they repeat those courses in EL or FTF format? As Table 2 demonstrates, students who fail EL courses are much more likely to pass them if they repeat those courses in the FTF format. Distribution of Outcomes for Students who Withdrew from EL Courses Total, withdrawals from DL: 422 Of those, repeated the course: 342 (81%) Repeated the course in DL mode: 102 (30%) Of those, passed: 58 (57%) Repeated the course in FTF mode: 240 (70%) Of those, passed: 132 (55%) Distribution of Outcomes for Students who Failed in EL Courses Total, failures in DL: 200 Of those, repeated the course: 103 (52%) Repeated the course in DL mode: 30 (29%) Of those, passed: 18 (60%) Repeated the course in FTF mode: 73 (71%) Of those, passed: 61 (84%) Table 2: Outcomes for Students Who Withdrew and Who Failed EL CIS Courses Were policies to be established required cumulative GPA s for enrollment in CIS EL courses, what would be the impact of that policy? Table 4 presents a series of cut-off average GPAs and the impact of such a policy? Would Not Have Been Allowed to Enroll Cut off GPA Percentage (Based on sample EL Number enrollment of 2554 in % % % % % Table 3: Student in the CIS Sample Who Would Not Have Been Allowed to Enroll in the EL courses in if the Cut-Off GPA Scores Were Implemented Clearly, the establishment of cut-offs would result in substantial EL enrollment seat decrease, with substantial seat decrease as cumulative cutoff increases. While the study of student records did not reveal any particular criteria for success, results of a small survey of 30 students who withdrew from these EL courses reported that the students who withdrew studied from home, in the evening, and typically worked more than 30 hours a week. Thus, the criteria for success might be due to environmental issues not captured in studies of student records. Case Study: Student Performance in a Course in World Literature A second study conducted with NJIT's Office of Institutional Research and Planning extracted the records of all students enrolled in two junior-level courses, World Literature (Lit 330 and Lit 331), offered as electives to all NJIT undergraduates to fulfill the required Humanities General University Requirements. Hosted by the Department of Humanities in the College of Science and Liberal Arts, World Literature I and II are courses specifically developed to be offered in an EL environment. Lectures in the course were developed by the English faculty in the Department of Humanities and produced by the Division of Continuing and Professional Education; the courses were delivered in a WebCT format, with full use of well-articulated assignments leading to researched essays and group projects. The study design identified three independent (predictor) variables hypothesized to be associated with successful student work in the two EL courses: increased student-instructor contact (defined as instructor entry into the WebCT portal at least once every 48 hours); increased assignments (defined as specific tasks for completion, bibliographic

4 search in support of researched essays, and group projects); and increased formative assessment (defined as opportunities for revisions of assignments in order to increase point values and grades). The dependent (outcome) variables associated with this success were defined as the students' average grade, average cumulative GPA, and level of student satisfaction. In order to test the hypothesis of this variable relationship, we examined students enrolled in both World Literature courses taught by the instructor who designed and most often taught the courses (the experimental group) in 2001 and 2002 (n=63)and students enrolled in other EL courses (the comparison group ) taught in the Department of Humanities from (n=384). The study asked three questions: (1) Is there a difference in academic preparation as measured by Grade Point Average? (2) Was there a difference in the average grade between the experimental EL class and the comparison EL classes? (3) What was the retention rate in the experimental EL class? As illustrated in Table 4, the average GPA (recorded from the semester before the Lit courses were taken) was somewhat higher in the experimental group, and women in that group had a higher GPA. Yet the academic preparation appears to be more similar than not. An additional study revealed that 128 students who took the two semester World Literature course from 1994 to 2001in a traditional FTF environment had an average GPA of Hence, the academic preparation appears to be the same across all groups. Experimental Group Compartive DL Group Female Male All Female Male All Asian Black Hispanic Native Amr White Not Known Table 4: Grade Point Averages in the Department of Humanities EL Study: Experimental and Comparison Group Results As illustrated in Table 5, however, the average grade was higher in the experimental group than in the comparative EL group. Experimental Group Compartive DL Group Female Male All Female Male All Asian Black Hispanic Native Amr. White Not Known Total Table 5: Average Grade in the Department of Humanities DL Study: Experimental and Comparison Group Results Table 6 provides the complexities involved in retention in this course. The EL experimental courses began with a total of 115 students, an average of some 38 students per section. In that these courses were capped at 30, many students either added or withdrew before the formal registration period ended. In fact, within the first two weeks of the course, 52 students withdrew from the course. The Registrar's deadline for withdrawal, occurring just after the midpoint of our 15 week semester, witnessed the loss of 26 students who had remained in the course after the first two weeks. By the end of the course, only 63 students were awarded final grades in the course. Thus, there was a 42.5% loss of students.

5 Total Number of Students listed in WebCT for Lit 330 (Fall 2001, Summer 2002) and Lit 331 (Spring 2002) Total Number of Students who were assigned final grades (Recorded on the Registrar s Roster) 115 students 63 students (42.5% loss) Withdrawals in the First Two Weeks of Courses (Not recorded on the Registrar s Final Class Roster) 52 students (54.8% loss) Withdrawals by the Registrar s Deadline (Recorded on the Registrar s Final Class Roster and Received a W ) 26 students (22.6%) Table 6: Retention Rate in the Experimental EL Classes At the present writing, the documented pattern endures. In the fall of 2004, 21 students remained in the class after the first two weeks. The average grade in the course was Seven students, 33.3% of the total, withdrew. Results: Where Did All the Undergraduates Go? There are obvious differences in the design of both studies. The first is a large-scale, quasi-experimental study, consisting of 18,030 students; the second, a case study, includes a mere 63 students in the experimental group. While little is formally known about the thirty EL undergraduate CIS courses analyzed in the first study, a great deal is known about the design and execution of the two World Literature courses involved in the second study. Taken together, the two studies are evocative, and it may be useful to think of the CIS study as the larger population that is representative of E-Learning at NJIT. The Lit courses, while in another discipline, may be considered part of that larger world. The similarities between student performance in the two worlds are noteworthy. First, there seems to be little difference in the academic preparedness of those who elect to take E-Learning and traditionally-taught courses in CIS. A similar conclusion can be drawn from the study of the Lit courses. Hence, it appears safe to assume that the students who elect to take EL courses at NJIT have similar academic backgrounds to those who do not. Second, even with similar academic preparation, there are striking differences in the performance of the EL students. In the EL classes in both disciplines, the perseverance rates (or what may dimly be called the mortality rates) are unacceptably high. And if a policy is enforced that would allow only the very best students to enroll in EL courses say, those with a GPA of 3.0 or above to enroll there would be a 57.2% decrease in enrollment. While that may be the necessary policy, it seems counter-intuitive to restrict asynchronous courses to those who most need them: students who must study from home, in the evening, and typically work more than 30 hours a week. Interpretation: The Presence of Irony What, exactly, is the nature of the brave, new world that we have created? At its worst, it is little more than an electronic mailbox in which students struggle in isolation; at its best, it is a community of scholars in which students participate in a community that strives to ensure the success of all. Have the students in this study received the poorest or the best instruction, or something in-between? How can we tell? The gaps in research that require further study the need for more emphasis on academic programs rather than on individual courses, the need to know more about the students who take these courses and how these students relate to the technologies used, the need for adequate explanations of drop out rates, the need for more astute theoretical frameworks, and the need for further study of the impact of digital libraries (Institute for Higher Education Policy, 1999) are, in reality, the gaps in all educational research. Granted, true experimental design requires randomization of sufficient numbers of students into comparative treatment groups. But, truth be told, how much do we know about the traditionally-taught courses that served the 15,486 students pouring into CIS classes in two semesters? Or about the teachers who hosted the 128 students who took World Literature from 1994 to 2001? Or about the students studying in either discipline? Do we know enough to randomize with any confidence? It is difficult to ignore the presence of irony. The world of E-Learning, associated with convenience, has proven to be just the opposite. The more we learn about E-Learning, the more we realize how little we know about the traditional world of the face-to-face classroom. Where, exactly, would the NJIT students be better served? Has the

6 expense of E-Learning been worth it if the student it was designed to serve the one working to pay tuition during the day, studying materials provided in WebCT during the evening, and listening to streamed lectures into the night winds up withdrawing from the course? Rather than creating a true asynchronous learning environment, have we instead created a revolving door for students most needing access to higher education? Implications of the Presence of Irony In a classic essay, "Irony as a Principle of Structure," the literary critic Cleanth Brooks proposed that literature has its own special language that is different from the language we find in ordinary experience. As John Carlos Rowe has astutely pointed out, for formalist critics such as Brooks, literary structure was achieved by negation of the empirical world, what Brooks called the "obvious warping of a statement by the context" (758). Building on this concept of irony as a negation of absolute value, the contemporary philosopher Richard Rorty has developed the definition of an ironist as one who has radical doubts about any final vocabulary, including one's own. Hence, the opposite of irony becomes common sense, those practices that assume that there is a referential world in which signifiers unfailingly match things signified. The application of the concept of irony to E-Learning is useful because there never was and never will be a final vocabulary to the world of education. Because there are simply more variables in humans than there are in falling objects, there is no model in the natural world that can be used to understand the complexities of human behavior (Hollis, 1994). What is called for, then, is further exploration of the contexts in which E-Learning and what we vaguely refer to as face-to-face learning occur. The presence of irony, in addition, suggests that each world whether existing in a room with desks or on a computer linked to a broadband connection is unique. Whether the topic is machine and assembly language programming or the novels of Gabriel Garcia Marquez matters less than the way those subjects are warped by the context in which they are presented. There is no final vocabulary that we can share, no common sense set of best practices that will work across time and circumstance. There is no absolute value. The problem, of course, is that while irony is invaluable in forming private transactions, it is rather useless when it comes to the public world of politics. Hence, it would seem that our application of irony to the world of E-learning is at an end. Administrators want to know about the effectiveness of these delivery systems, not about findings based on an interminable set of narrative case studies with numbers too small to be analyzed with inferential statistics. Recognizing such limits, Rorty suggests that we attend not to systems but to individuals, to finding avenues of commonality among us all. Such a frame of reference would require a massive perceptual shift if applied to educational research. Yet, since we are not, after all, going to know much about the 18,030 CIS students in this study, or even about the mere 63 students in the E-Learning World Literature course, perhaps what we need is a greater sense of the particular ironies of education in the 21st century and the unique ironies of EL. Then, at least, we would be getting at a kind of truth, if only the way that it is warped and rendered wonderfully particular. References Allen, M., Bourhis, J., Burrell, N., & Mabry, E. (2002). Comparing student satisfaction with distance education to traditional classrooms in higher education: A meta-analysis. The American Journal of Distance Education 16 (2), Brooks, C. (1951, 1998). "Irony as a principle of structure." In (Ed. D. H. Richter) The critical tradition: Classic texts and contemporary trends (pp ) Boston: St. Martin's Press. Hollis, M. (1994). The philosophy of social science: An introduction. Cambridge: Cambridge University Press. Institute for Higher Education Policy. (1999). What's the difference? A review of contemporary research on the effectiveness of distance learning in higher education. Washington, DC: Institute for Higher Education Policy. Mock, K. (2003). The development of a CSO course for distance delivery. The Journal of Computing Sciences in Colleges, 19 (2), Rorty, R. (1994). Contingency, irony, and solidarity. Cambridge: Cambridge University Press.

7 Rowe, J. C. (1995). Structure. In F. Lentricchia & T. McLaughlin (Eds.), Critical terms for literary study (pp ) Chicago: University of Chicago Press.