BALANCING RETENTION AND ACCESS IN ONLINE COURSES: RESTRICTING ENROLLMENT... IS IT WORTH THE COST?

Transcription

1 J. COLLEGE STUDENT RETENTION, Vol. 15(1) 9-36, BALANCING RETENTION AND ACCESS IN ONLINE COURSES: RESTRICTING ENROLLMENT... IS IT WORTH THE COST? ALYSE C. HACHEY, PH.D. CLAIRE W. WLADIS, PH.D. KATHERINE M. CONWAY, PH.D. Borough of Manhattan Community College City University of New York (BMCC-CUNY) ABSTRACT Open access is central to the Community College mission. For this reason, any restriction in online enrollments should not be undertaken lightly. This study uses institutional data gathered from a large, urban community college to examine a policy aimed at increasing student retention in online courses by restricting those eligible to enroll based on Grade Point Average (GPA) The data, counter to expectations, show that the policy did not significantly impact attrition rates. Further analysis reveals that a high GPA cut-off (3.0) is needed to significantly affect attrition rates; however, this would severely restrict those eligible to enroll. The data indicate that students in the middle GPA range ( ) have the highest proportional difference in attrition between online and face-to-face courses. The results suggest that rather than focusing on GPA restrictions, community colleges may be better served by addressing research and interventions targeted toward other factors to increase student retention in online learning. 2013, Baywood Publishing Co., Inc. doi: 9

2 10 / HACHEY, WLADIS AND CONWAY INTRODUCTION For over 100 years, community colleges have provided access to higher education in the United States. Currently, community colleges have a dual mission: workforce development and transfer to senior colleges. This dual mission operates within an overarching framework of open access for minorities, students of lower socio-economic status and those students not served by traditional four year colleges and universities (Shannon & Smith, 2006). As part of fulfilling their mission, community colleges endeavor to be sensitive to the changing educational and life-style needs of their students. The current student body entering community colleges today has considerably more need for experiences with technology than previous generations. Called Millennials or Net Generation students, they come with heightened expectations for access to digital resources and technology support. Striving to blend technology with higher education, online learning courses have become a core feature of services offered at most colleges and universities (Downes, 2005). However, with the rapidly growing popularity of online learning at the community college level, this instructional delivery method poses a continued challenge for meeting higher education standards of accountability while still maintaining the ideal of open access. LITERATURE REVIEW Historical Role of Community Colleges in Providing Access to Higher Education Since 1947 and the publication of the Truman Report, the community college has been the fastest growing segment of post-secondary institutions. Community colleges grew parallel to the overall growth in college enrollments. In the post-war years, the passing of the GI bill and increased demand for technical and clerical skills led to growth in college enrollments (Bowles & Gintes, 2003). By 1960, almost one out of every four first time freshman was enrolled in a community college (Higbee, 1973), but the greatest growth was still to come. The accepted belief that higher education led to increased social capital (Brick, 1963) combined with broadening civil rights, led to demand from new groups of students, who largely had been denied access to higher education in the past (Bowles & Gintes, 2003). Additionally, there was a growing belief that the measures used to determine college admissions in the past might not be valid for certain groups of students (Sue & Abe, 1988). These issues made way for open admissions and the resultant explosion in community college enrollments. In the decade ending in 1970, more than 400 new community colleges opened, an increase of slightly more than 60% (Cohen & Brawer, 1996) in just ten years. By 1970, 45% of all college students between

3 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 11 the ages of were in community colleges, compared to less than 5% at the turn of the century (Trow, 1988). A 1972 study defined a mature system of community colleges as one in which 90-95% of the population resided within a reasonable distance (25 miles or less) and in fact, by 1994, that model was in place, with a system of 1,094 community colleges established throughout the United States (Cohen & Brawer, 1996). The idea of universal education through grade 14 that began with Truman continues to the present. President Obama, in a speech given in July 2009, called for the establishment of the American Graduate Initiative: It will reform and strengthen community colleges from coast to coast so that they get the resources students and schools need and the results workers and businesses demand. Through this plan, we seek to help an additional five million Americans earn degrees and certificates in the next decade. Today, community colleges enroll the largest number of students among American institutions of higher education. Half of all first time freshmen begin at a community college (Community College Students: Goals, Academic Preparation & Outcomes, 2003). Growth of Online Learning in Community Colleges Community colleges have risen in response to the public demand for access, and with the President s call for more college graduates and enrollment exploding today, technology in the form of online learning may provide a low cost and convenient method of delivery. This is borne out by the fact that public community colleges lead the way in offering distance education, with the most popular offering being the asynchronous online course. In the academic year, 97% of community colleges offered some form of distance education, with enrollment in distance education programs exceeding 12 million and accounting for half of all college-level credit-granting courses that year (Parsad & Lewis, 2008). Moreover, over two-thirds of the institutions surveyed by the National Center for Educational Statistics (NCES) cite the desire to meet student demand for flexible scheduling and the goal of providing access for students who might not otherwise be able to attend college (e.g., because of geographic, family or work-related issues) as the primary reasons for offering online courses (Parsad & Lewis, 2008). NCES indicates that more than 20% of all higher education students are now taking at least one online course and that this trend is expected to keep growing (Parsad & Lewis, 2008). The current generation of college students has embraced technology as evidenced by their high levels (78%) of Internet usage for research during high school (Educause Center for Applied Research, 2003) and lifestyles that incorporate a variety of mobile technologies (Oblinger & Oblinger, 2005). A report from the Sloan Foundation (2008) affirms that online

4 12 / HACHEY, WLADIS AND CONWAY learning has been growing at a substantial rate, far exceeding the growth of higher education in general (Allen & Seaman, 2008). Overall, the research suggests that community colleges have embraced online learning as means of helping to meet their overarching purpose of open access. Quality and Online Learning With the astounding growth of online learning in higher education over the last decade, one of the main focuses has been the quality of education provided by this new instructional method. Student outcome, as measured by attrition rates, is a primary measure of the quality of education provided by an institution, and specifically, the effectiveness of an online learning program (Abel, 2005; Moody, 2004; Willging & Johnson, 2004). Nationally, attrition rates in online courses range from 30 to 40% (Tyler-Smith, 2006). Moreover, dropout rates for online courses have been consistently higher nationally than those for face-to-face classes and have been linked by some researchers to academic non-success (Diaz, 2002). Carr (2000) addresses the issue in terms of persistence and retention, with student retention in online learning programs often percentage points lower than campus-based programs (Angelino, Williams, & Natvig, 2007). It is important to note that attrition and persistence/retention are two sides of the same coin one from the perspective of the institutional manager (attrition) and one from the student perspective (persistence/retention) and that they can be used interchangeably. Regardless of perspective, a growing concern, and potential policy driver at community colleges, is identifying factors that affect student persistence and retention in online courses. Factors Contributing to Student Persistence In the mid-seventies, Tinto developed a model of student departure that built on the ideas of Spady and Durkheim (Tinto, 1975, 1986, 1993). Tinto s Model of Student Integration is predicated on the student s ability to academically and socially integrate. If the student cannot identify with, or assimilate into some aspect of the educational setting, dropout occurs. Tinto stated that students enter college with a variety of background characteristics, which include personal attributes, family demographics and academic preparation, as well as intent to attend and graduate from college. Over time, these intentions change as students interact with others in the academic setting, as well as with the institution itself (Tinto, 1975, 1986, 1993). Positive interactions lead to greater integration and thus persistence, while negative interactions lead to lesser integration, lesser commitment and ultimately dropout. Tinto s work has been widely tested and validated at the four-year institution (Pascarella, 1980; Pascarella & Chapman, 1983; Pascarella, Smart, & Ethington, 1986; Pascarella & Terenzini, 1991; Stage, 1989) and for non-traditional students attending two-year institutions (Kraemer, 1993; Nora, 1987; Nora, Attinasi, & Matonak, 1990; Nora & Rendon, 1990).

5 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 13 While Tinto s initial work only viewed external forces indirectly, he later stated that for those students whose affiliation with the institution may be weaker, such as commuters, or older students with family and work responsibilities, the effect of external forces may be greater (Tinto, 1987, 1993). Online learners may have the most tenuous affinity with the learning institution, and may have placed external responsibilities ahead of educational goals, thus making an online environment the only viable option for continuing their education. Later research, drawing on models of organizational turnover, gave greater weight to external forces, but did not explore the online learner specifically. Given that commuter students spend less time on campus, the impact of external forces is more significant in the Student Departure Model (Bean & Metzner, 1985; Metzner & Bean, 1987). Several studies have been done, validating the use of the Student Departure model for non-traditional students (Stahl & Pavel, 1992). While the Student Integration model and the Student Departure model have emphasized different dynamics, there are also similarities between the two (Cabrera, Castenada, Nora, & Hengstler, 1992; Cabrera, Nora, & Castenada, 1993). Nora and Cabrera developed a Student Adjustment Model (1996), based on the similarities between the earlier models and including issues such as employment (Cabrera et al., 1993), financial aid (Cabrera et al., 1993) and encouragement from friends and family (Nora, 1987). Predicting Persistence in Online Learning Existing models of persistence, retention and/or integration may not be applicable to the online learning environment because of an emphasis on social integration, a process which is very different for the online education student in comparison to the traditional residential or commuter students studied by Tinto, Bean or Nora and Cabrera. Tinto (1982) cautioned against application of his model in nontraditional settings, and similarly, later models which examined commuter students, part-time students, and so forth, may not apply to the online learner. To date, an archetypical model of persistence like that developed by Tinto for the traditional student does not exist for the online learning student. However, Kember (1995) has developed a two-prong model of persistence for adult learners in online education courses. According to this model, persistence in online courses is positively impacted to the extent that academic and social integration result in good academic performance and persistence is negatively impacted to the extent that external demands detract from academic performance. Social integration refers to a student s ability to integrate their education with other aspects of their lives (work, family, friends, etc.). Kember found that the correlation between grade point average and persistence was higher in online courses than in other similar models of on-campus students, but stated that there is a significant portion of the variance in student persistence that cannot be explained

6 14 / HACHEY, WLADIS AND CONWAY by GPA alone, which suggests that there are other factors not included in Kember s model which likely affect student persistence. Tinto s work, along with several of the other models noted above, focuses on factors which influence student success after entry into a program of study. The goal of this research, however, was to explore whether a pre-entry characteristic might impact student success. As noted earlier, one of the criticisms of distance education is its lower rates of persistence versus the traditional classroom environment (Noble, 2001), resulting in a quandary for college administrators: how to increase accessibility to higher education via online education without negatively impacting the main quality indicator of attrition rates. This is a critical concern, as success, measured by course completion, graduation and transfer rates to four-year programs, are the primary indicators used by accrediting agencies in determining performance-based funding for community colleges (Ewell, 1994). Most traditional retention programs, run by student services professionals, are campus-based activities designed to address the research on the correlation between student persistence and involvement (Astin, 1983), and therefore, typically do not serve students who have selected an online education course because they cannot be bound to campus-based activities (Barefoot, 2004). Colleges have considered a number of methods to improve retention in online education, including both face-to-face and online orientation programs, the creation of virtual learning communities, and increased faculty and student interaction (Bocchi, Eastman, & Swift, 2004; Lorenzetti, 2008; Putre, 2008; Santovec, 2004). An additional approach to improving retention is to increase the academic entry requirements of incoming students. Past educational attainments are one of the most widely used predictors of further academic success. High school performance has been positively correlated with college persistence (Cejda & Rewey, 1998; Graham & Hughes, 1994; Hoffman & Lowitzki, 2005; Noble & Sawyer, 2002; Pigge & Marso, 1992). Morris, Wu, & Finnegan (2005) found that high school GPA and SAT mathematics scores were related to retention in online university courses. However, since many online learning students are mature students who are several years removed from the high school experience (Kember, 1995), high school performance may not be a good indicator of online learning success. College grade point average, which is also positively correlated with persistence (Metzner & Bean, 1987; Metzner, Lauer, & Rajecki, 2003; Sorey & Duggan, 2008), may be a more relevant measure for future retention and success of the online learning student. The reasons for attrition in online courses are numerous and complex (Tyler- Smith, 2006), but there is support for a relationship between college grade point average and persistence. In particular, Nora, Barlow, and Crisp (2005) note that numerous studies point to a lingering effect of poor first-year college performance, with a very high risk of dropout if students have a first-year GPA below 2.0. Moreover, several researchers (see Rovai, 2003 for a review) have noted a significant relationship between previous academic performance (as

7 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 15 denoted by GPA) and completion of online learning courses. This is supported by research by Diaz (2002), who found that successful students in an online course had a higher average GPA prior to enrollment (avg. GPA = 3.02) than unsuccessful students (avg. GPA = 2.25) and Muse (2003), who cites GPA as one of several significant factors affecting online retention at the community college level. Thus, previous GPA has been found to be one predictor of persistence and retention in online learning. Open Access vs. Restrictive Enrollment Based on GPA Instituting a minimum grade point average requirement in an open enrollment environment is seen by many as placing additional roadblocks in a student s educational path. Both the NAACP and the ACLU have expressed concern that tougher admissions standards result in drops in minority enrollment (Groups Worry About Nevada s Tougher Admission Standards, 2008). One of the most visible struggles over the balancing act between open admissions and maintaining academic quality occurred at the City University of New York (CUNY), where open admissions policies were accused of reducing the quality of a CUNY degree, in effect creating a diploma mill (Lavin, 1997). CUNY attempted to resolve concerns about quality while maintaining its mission to provide access by increasing academic requirements for admission to the senior colleges and directing students with remedial needs to the community colleges. This solution was termed a battle on two fronts, one for CUNY s democratic mission and the other for the community colleges professional integrity (Reitano, 1999). But questions of balancing access and academic standards are not limited solely to college admission; there are also differing standards among programs within open enrollment institutions. For example, selective policies for admission to nursing programs in community colleges as a result of scarce resources (too few seats, too many applicants) have been cited as an ethical dilemma because they limit educational mobility (Bissett, 1995). In the City Colleges of Chicago system, students must maintain a C average in their core courses in order to graduate, a move designed to burnish the college s reputation for quality and win points with legislators approving funding to the colleges; similar requirements exist at community colleges in South Carolina and Colorado (Haworth, 1999). Since community colleges are now the entry point to higher education for the majority of college freshmen, limiting access to programs and/or courses within the community college could further hinder socioeconomic opportunity for far too many students. Open access, however, needs to be balanced against the reality of a structure in which community colleges are measured by indicators such as graduation and attrition rates. Within this tension between access and quality (as indicated by low attrition rates), the community college that is the subject of this study decided to take steps to limit access to online classes to students whom they believed, based on past academic success, might have a better chance of persisting.

8 16 / HACHEY, WLADIS AND CONWAY CONTEXT OF THE STUDY The urban community college in this study enrolls approximately 22,500 students in degree-programs and 10,000+ more in continuing education programs. The College s diverse student body comes from over 100 countries around the world. Over 80% of the College s student population belongs to groups historically underrepresented in higher education. Thirty-seven percent of the student body is African-American, 33% is Hispanic, 14% is Asian, and 16% is Caucasian. Additionally, about two-thirds of its student body is female. The College s mission statement reads The College is committed to offering quality education in a pluralistic urban environment, to fostering excellence in teaching, to facilitate the enhancement of learning, and to sustaining full access to higher education for those who seek fulfillment of personal, career or socioeconomic goals (college website). In the spring of 2001, a faculty-driven online learning program emerged with the aim of addressing the College mission of open access amid the technology and lifestyle needs of students. By the fall of 2009, the College had 78 online courses available and offered an online Associate s Degree in Liberal Arts. Eighty-two percent of the College s online offerings are in Liberal Arts (with half of those in the Social Sciences). The other 18% of the offerings are in career-oriented majors (Business, Office Administration, Nursing and Teacher Education). Early in the College s online program inception, a comparison of grade performance in online courses and cumulative grade point average indicated that in an online environment, students with strong general academic skills do better than those with poorer academic skills. In the spring of 2004, 754 students enrolled in online classes at the College, with 65.0% of those students in good academic standing (GPA > = 2.0) successfully completing their online courses with a grade of C or above. Analysis of the numbers indicated a significant disparity, with students in poor academic standing (GPA < 2.0) earning only about half as many grades of A as those in good academic standing, earning three times as many F grades, and dropping out at a rate that is 27% higher ((28-22)/22 = 27% is used to calculated the percentage increase) (see Table 1). Based on this data, the College implemented a restrictive registration policy in the fall of The first regulation was that students were limited to two online courses per semester. This regulation addressed possible issues of cognitive overload, noted by Tyler-Smith (2002) as a potential contributor to student attrition. The second regulation was aimed at filtering out students deemed as high risk those with poor academic standing. To enroll in an online course at the College, a student must have a cumulative GPA > = 2.0. In addition, first-semester freshmen were not allowed to enroll in online courses as they would not yet have a GPA score.

9 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 17 Table 1. Spring 2004 Online Grade Distribution by GPA GPA < 2.0 GPA > = 2.0 A/A B+/B/B C+/C/C D+/D/D F ABS/INC W/WU 20% 19% 10% 3% 12% 8% 28% 35% 20% 9% 2% 4% 7% 22% Institutional data indicates that the College s attrition rate for online courses has consistently been low compared to the national standard of 30-40% (with the College ranging from a low of 20% in to a slight a high of 26.9% in , and averaging 23.5% over the past five years). It is believed that the rigorous screening policy currently in place at the College, designed to ensure that academically unprepared students do not enroll in online courses, accounts for higher rate of student retention in online courses. However, the current enrollment policy strikes a tenuous balance between allowing those with a better chance for success to enroll and alignment with the College s mission of sustaining full access for all students. PURPOSE The purpose of this study is to assess the claim that the restrictive GPA policy in place has affected retention rates in online courses by: 1. comparing rates of attrition between online courses before and after the enrollment restrictions; and 2. examining persistence at a course level to determine if there is any difference from the program-level analysis when steps are taken to control for variation in instructors and course offerings online. Then, in order to analyze these results, we also consider: 1. which GPA restrictions would be necessary to create a statistically significantly drop in online attrition rates; 2. how the GPA distribution of online students compares to face-to-face students; and 3. how online attrition rates for each GPA category correspond to face-to-face attrition rates for students with the same GPA.

10 18 / HACHEY, WLADIS AND CONWAY METHODOLOGY Data for this research was provided by the College s Office of Institutional Research and Assessment. We have chosen to compare data taken from both online and traditional face-to-face classes in spring 2004, the semester before the restrictive GPA enrollment policy was first implemented, and comparable data for the semesters following the restrictions, with the aim of analyzing whether or not the new restriction had a significant effect on student success or retention in online courses. Attrition rates are tracked over a five-year period, and variations in attrition rates are assessed by computing the z-values and their corresponding P-values for the differences between the attrition rate before the implementation of enrollment restrictions and the attrition rate after the enrollment restrictions were in place. These z-values were calculated using the standard difference test formula for comparing two proportions. In addition, attrition rates between the same courses taught by the same instructors both before and after the implementation of enrollment restrictions are compared. RESULTS To assess the results of the current restrictive enrollment policy for online courses at the College, an analysis was conducted comparing spring 2004 grades and attrition rates to those of later semesters, after the G.P.A. restriction was implemented. The data do not support that the present policy is having a significant effect on attrition rates, either in aggregate or on an individual course level. Attrition 1 Before and After Implementation of GPA Restriction The attrition rates for spring 2004 through spring 2008 were analyzed: the attrition rate for online courses increased from spring 2004 to fall 2004, when the GPA restriction was first implemented, from 22.7% to 23.7% respectively, while the attrition rate for face-to-face courses dropped slightly from 14.2% to 13.9%. So the gap between online and face-to-face attrition actually rose, from 8.5 to 10 percentage points, from spring 2004 to fall 2004, when the enrollment restriction was first implemented (see Table 2). 1 Attrition rates in this article include only those students who earned a W or WU designation for the course. A WU designation at the College is given to students who stop attending classes before the college s official withdrawal deadline but fail to formally withdraw from the class; this deadline occurs the last day of the ninth week of classes each semester. Students who stop attending class after this date receive an F grade instead of a WU designation. Students who officially withdraw from a course after the third week of classes receive a W designation. ABS and INC designations both indicate an incomplete for the course, and are not included in attrition rates here.

11 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 19 Table 2. Attrition Rates in Online vs. Face-to-Face Classes Spring 2004 Fall 2004 Spring 2005 Fall 2005 Spring 2006 Fall 2006 Spring 2007 Fall 2007 Spring 2008 Online 22.7% 23.9% 20.1% 21.6% 19.9% 23.4% 23.2% 26.3% 26.9% Faceto-face 14.2% 13.9% 14.2% 13.0% 15.0% 16.4% 18.6% 17.9% 16.9% Gap a a Between online and face-to-face attrition (in percentage points). For the remainder of our analysis, we consider only spring semesters so that we can compare comparable semesters to one another we have chosen the spring because that includes the last semester before the change in policy and also the most recent semester for which data were available, giving us the largest set of data; repeating this analysis for fall semesters produces similar results. Looking at these trends in online attrition over a period of five years, the attrition rate at the College for online courses seems to have remained relatively stable over time, decreasing slightly in 2005 and 2006, but then increasing above 2004 levels in 2007 and 2008, even with the more restrictive online course admission policies still in place (see Figure 1). These data suggest that the new GPA policy likely did not have a significant impact on attrition. It is possible that these variations in attrition during the period are due to other factors, or are due to random variation in student enrollment in online courses over this period of time. It is also possible that the new policies produced a small temporary drop in attrition that was later reversed by other factors in later years; to assess whether this might be the case, it is necessary to determine whether these variations in attrition rate are statistically significant. To determine whether the differences in attrition rates from spring 2004 to spring 2005 and spring 2006 are statistically significant ( = 0.05), we computed z-values for each of the drops in attrition rates (with respect to spring 2004 levels), see Table 3 for details. A comparison between attrition rates in spring 2004 to spring 2005 and spring 2006, respectively, yields z-values of 1.23 and 1.37, respectively; these correspond to P-values of and , respectively, indicating that the drops in attrition rate from the spring 2004 value to those computed for spring 2005 and spring 2006 are therefore not statistically significant ( = 0.05). Even more convincing, the attrition rates for spring 2007 and spring 2008 were actually higher than in spring 2004, resulting in P-values of and respectively, and indicating that these higher values were likely not due simply to random variation from semester to semester.

12 20 / HACHEY, WLADIS AND CONWAY Figure 1. Attrition rates in online courses vs. face-to-face. Table 3. Attrition Rates in Online Classes After GPA Restriction Testing the hypothesis that attrition dropped after fall 2004, or that the attrition rate for subsequent semesters is < 22.7%. Spring 2004 Spring 2005 Spring 2006 Spring 2007 Spring 2008 Online 22.7% 20.1% 19.9% 23.2% 26.9% N z-statistic P-value Controlling for Varying Distribution of Courses and for Instructor To be certain that any changes in grade distribution and attrition rates in online courses from spring 2004 to spring are not due to shifts in the composition of online courses and instructors teaching online, we also did a comparison limited specifically to those online courses which were taught by the same instructor in both spring 2004 and in a later spring semester. For each spring semester after 2004, we used this restricted list of courses to calculate a reweighted attrition rate, by multiplying the later semester s attrition rate for

13 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 21 each course by the proportion of online students in that course in spring 2004, and summing these amounts over all courses offered in both semesters (see Table 4). This new evidence allowed us to remove any variation in attrition that might have been due to the proportion of students in different courses changing from semester to semester, since courses in different subject areas, at different levels of difficulty, and as a part of different programs, often have very different attrition rates. Limiting our analysis only to those courses that were taught in both spring 2004 (before the restrictive GPA policy) and spring 2005 (after the restrictive GPA policy was enacted) by the same instructor resulted in a total of 24 courses in disciplines ranging from applied fields such as business and nursing, to liberal arts subjects in areas such as social science and mathematics. A total of 670 students were enrolled in these specific course sections in spring 2004, and 614 students were enrolled in these same course sections in spring For these courses, attrition fell slightly from spring 2004 to spring 2005, from 22.8% to 20.7%. However, the proportion of students in each course varied from spring 2004 to spring 2005, so any differences in attrition could simply be due to differences in distribution of online students in particular classes in 2005 and To control for these differences in class size and distribution, we took the passing rate of each course in spring 2005, and we rebalanced the 2005 attrition rate so that the proportion of total students in each course reflected the composition of students in online courses in spring 2004; for example, a course which had 30 students enrolled in spring 2004 but 20 students enrolled in spring 2005, 4 of whom received a W or WU designation, would have been Table 4. Attrition in Identical Courses Taught by the Same Instructor Attrition Reweighted a attrition Spring 2004 attrition b N c N 2004 z-statistic p-value Spring 2005 Spring 2006 Spring 2007 Spring % 19.1% 23.2% 26.5% 22.0% 19.5% 25.1% 26.3% 22.8% 21.7% 21.7% 20.3% a Each course in the corresponding semester given in the far left column, with its attrition rate reweighted in the total semester s attrition rate, based on the proportion of students in that course in spring (For example, if 5% of online students were in MAT 100 in spring 2004, but 3% of online students were in MAT 100 in spring 2005, then the overall reweighted attrition rate for spring 2005 would be computed by multiplying the spring 2005 attrition rate for MAT 100 by 5% and adding it to the respective reweighted spring 2005 attrition rates for all other courses offered in both semesters.) b This is different in each row, because it includes only those courses in spring 2004 which were taught by the same instructor in the corresponding semester given in the far left column. c For the corresponding sections in the semester given in the far left column.

14 22 / HACHEY, WLADIS AND CONWAY reweighted as a total of 30 students, 6 of whom received a W or WU designation ( 4 6 = 20% attrition rate for spring 2005). The reweighted attrition rate for spring 2005 is then 22.0%, which is only slightly lower than the spring 2004 attrition rate of 22.8%. Computing the z-value for the comparison of these two attrition rates yields 0.37, which corresponds to a P-value of , clearly indicating that the drop in attrition rates for these specific sections from spring 2004 to spring 2005 is not statistically significant ( = 0.05). A repetition of this analysis in a comparison of spring 2004 and spring 2006 yields a total of 22 different courses taught by the same instructor in both semesters. This results in an actual attrition of 19.1% and a reweighted attrition of 19.5% for spring 2006 in comparison to the attrition rate of 21.7% for this same selection of courses in spring Computing the z-statistic comparing the spring 2004 attrition rate to the reweighted spring 2006 attrition rate yields 0.98, which corresponds to a P-value of , which is not statistically significant ( = 0.05). Looking at this analysis for spring 2007 and spring 2008, each compared separately to spring 2004, yields even higher rates of attrition for spring 2007 and spring 2008 than in spring 2004, for both the original and the reweighted attrition rates. This resulted in a total of 16 courses which were taught in both spring 2004 and spring 2007 by the same instructor, and 18 courses which were taught by the same instructor in both spring 2004 and spring The corresponding spring 2007 attrition rates for these 16 courses was initially 23.2% and 25.1% when reweighted, whereas the attrition rate for these same 16 courses in spring 2004 was 21.7%. The corresponding spring 2008 attrition rates for the 18 selected courses was initially 26.5% and 26.3% when reweighted, whereas the attrition rate for these same 18 courses in spring 2004 was 20.3%. These clear increases in attrition rates in spring 2007 and spring 2008 relative to the respective equivalent set of courses also offered in spring 2004 yielded P-values of and respectively, clearly indicating that these higher attrition rates were not likely due simply to random variation. DISCUSSION 2 In this section, we consider three major categories of questions: 1. What GPA restrictions would be necessary to create a drop in online attrition rate that is statistically significantly or which results in an online attrition rate that is statistically comparable to the face-to-face attrition rate? 2 The data in this section comes from the CUNY IR database which keeps data frozen as of a certain date; the data from the previous sections came from CUNYVM, where data may change over time (e.g., when an INC grade is replaced with a letter grade). This results in a few minor differences in some of the percentages quoted here versus those quoted in earlier sections; however, these small variations are small enough to be statistically insignificant.

15 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / Why didn t the 2.0 GPA restriction create a statistically significant drop in attrition as expected (especially since we know that attrition rates rise as GPA decreases)? 3. How should we refocus our analysis of the differences between online and face-to-face attrition to address the fundamental causes of the attrition rate gap between these two instructional delivery methods? Since GPA restrictions excluding students with a GPA below 2.0 did not produce a statistically significant drop in attrition rates, one natural question would be to calculate, by using the proportion of students in different GPA categories (e.g., < 2.0, , , , ) and attrition rates for each of these GPA categories, how high a GPA restriction would need to be in order to either: 1. obtain a statistically significant ( = 0.05) decrease in online attrition rates; or 2. obtain an online attrition rate that is comparable to face-to-face classes. What GPA Restriction would be necessary for online courses in order to create a statistically significant ( = 0.05) drop in attrition rate? Using spring 2004 data, we have computed the attrition rate for students in each GPA category, and used this to calculate the attrition rate for students with a GPA above each category cut-off value. We have already seen that spring 2004 online students with a GPA at or above 2.0 had an attrition rate of 21.9%; for a GPA at or above 2.5, this jumps to 19.4% (see Table 5 for details). We can see that when we consider only the attrition of spring 2004 students with a GPA at or above 3.0, we would likely obtain a drop in online attrition that is large enough to be considered statistically significant ( = 0.05), were it to occur from one semester to the next. This is evident because the group of online students in spring 2004 with a GPA at or above 3.0 had an attrition rate Table 5. Attrition in Online Courses in Spring 2004 by GPA Compared to Total Online Attrition of 22.7% Attrition N for GPA category N for online courses z-statistic p-value 2.0 or above 2.5 or above 3.0 or above 3.5 or above 21.9% 19.4% 15.7% 8.9%

16 24 / HACHEY, WLADIS AND CONWAY of 15.7%, which produces a z-statistic of 2.57 and a corresponding P-value of when compared to the actual spring 2004 online attrition rate, and would therefore be considered to be a statistically significant ( = 0.05) drop in attrition if it were to occur from one semester to the next. In contrast, the P-value for the groups with GPA s at or above 2.5 and 2.0 are and , neither of which is statistically significant. So by looking at Table 6, we can see that the only GPA categories in which the difference between the attrition rate for the GPA category and overall online attrition is statistically significant ( = 0.05) is for the groups of students with a GPA at or above 3.0 or 3.5. This implies that in order to obtain a drop in attrition in online classes when compared to the spring 2004 level, a GPA cut-off would need to be as high as 3.0, a change which would exclude about 58% of those students who were registered for online courses in spring We obtain similar results if we consider which attrition rates would be necessary in order for the rate to be comparable to the face-to-face attrition rate. In Table 6 we consider when the attrition rate of a particular GPA group would represent an attrition rate that is statistically equivalent or less than the attrition rate for spring 2004 face-to-face courses. If we were to restrict online enrollment to students with a GPA at or above 3.0, then we would likely obtain an online attrition rate which is comparable to or better than the attrition rate for face-to-face courses. We can see this by observing that the group of online students in spring 2004 with a GPA at or above 3.0 had an attrition rate of 15.7%, which produces a z-statistic of 0.33 and a corresponding P-value of when compared to the actual spring 2004 face-to-face attrition rate, and therefore the difference between this attrition rate and the spring 2004 face-to-face attrition rate would not be considered to be statistically significant ( = 0.05) if it were to occur from one semester to the next. Our calculations correspond to findings in Diaz (2002), who cites that high GPA s were positively correlated to student success in online learning environments. However, these results are problematic, because they imply that we would have to bar 57.8% of potential online students from taking online courses each semester in order to obtain a statistically significant drop in the online attrition rate, or to obtain online attrition rates which are comparable to face-to-face attrition rates. This goes against the major purpose of community colleges, which is open access. In addition, our analysis raises further questions since students in online classes tend to have higher GPA s than those in face-to-face classes, it seems that GPA is not a good explanation for why online attrition is higher than 3 To be more precise, if we considered GPA s as a continuum rather than as discrete categories as give in the charts in our analysis, the GPA cutoff which would yield statistical significance is likely somewhere between 2.5 and 3.0, and therefore the percentage of students who would be excluded from classes with such a GPA restriction would be somewhere between 30% and 58%.

17 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 25 Table 6. Attrition in Online Courses in Spring 2004 by GPA Compared to Total Face-to-Face Attrition of 15.1% Attrition N for GPA category N for face-to-face courses z-statistic p-value 2.0 or above 2.5 or above 3.0 or above 3.5 or above 21.9% 19.4% 15.7% 8.9% face-to-face attrition: rather, other factors need to be considered, so we continue our analysis by looking at the data in more detail to see if we can better understand the attrition rate gap across all GPA categories. Why Didn t Our Analysis Detect a Difference in Attrition? The results obtained in this article seem surprising, because they seem to imply that the restrictive GPA policy made no difference in attrition rates, and this seems counterintuitive given what we know about dropout rates in online courses for students with GPA s below 2.0. For example, looking at spring 2004 data for online courses (see Table 1), we can see that students with a GPA below 2.0 had an attrition rate which was about 27% higher than the attrition rate of those with a GPA at or above 2.0 (28% vs. 22% respectively, with (28-22)/22 = 27% used to calculated the percentage increase). So preventing students with a GPA below 2.0 should certainly lower the attrition rate somewhat (unless the attrition rate for students with a GPA below 2.0 changes significantly) so does our data analysis indicate otherwise? To answer this question, we need to consider the actual number of online students who have a GPA below 2.0: in spring 2004, only 12.3% of online students fit this criterion in comparison to 33.7% of all face-to-face students (see Table 7). So while the new GPA restrictions likely did produce a slightly lower attrition rate in fall 2004 onward than would have existed had all students been permitted to register for online classes, this difference is clearly so small as to be outmatched by the size of natural variation in attrition rates from semester to semester. One way to analyze this more formally is to compute what the attrition rate in spring 2004 would have been had students with a GPA below 2.0 not registered for online courses that semester. Since 12.3% of all spring 2004 online students had GPA s below 2.0, and 28% of these earned a W/WU designation, this means that only about 3.4% of spring 2004 online students had both a GPA below 2.0 and withdrew from their online course, whereas 8.9% of all

18 26 / HACHEY, WLADIS AND CONWAY Table 7. Students Academic Standing, Spring 2004 GPA < 2.0 GPA > = 2.0 Online Face-to-face 12.3% 33.7% 87.7% 66.3% online students had a GPA below 2.0 and did not receive a W/WU designation for their online course. The actual spring 2004 attrition rate for online students with a GPA at or above 2.0 was 21.9%, in contrast to the actual attrition rate for all online students of 22.7%. This difference of only 0.8 percentage points is actually smaller than the variation in attrition that we see from semester to semester in both online and face-to-face courses; by calculating the z-statistic and corresponding P-value for this difference between the two proportions, we can see that had this exact drop in attrition occurred from one semester to the next, the change would not have been considered to be statistically significant ( = 0.05) (see Table 8). Analyzing Online vs. Face-to-Face Attrition for All GPA Classes The current GPA restrictions that prohibit students with a GPA below 2.0 from registering for online courses is particularly problematic because it assumes that the reason for the increased attrition rates in online courses is that either: 1. a high number of students with low GPA s are taking online courses; or that 2. students with low GPA s have a proportionally larger increase in attrition online than face-to-face. However, our data indicate otherwise: online students have a higher GPA on average than face-to-face students, and online students show higher attrition rates online than face-to-face across all GPA categories. In fact, students with a GPA below 2.0 actually have an attrition rate that is closer to their face-to-face attrition rate than for any other GPA group. For example, looking at the data given in Table 9, we can see that the ratio of attrition rates in online courses to the attrition rates in face-to-face courses for students in the same GPA category is about 2.0 for students with a GPA of Interestingly, the ratio of online attrition rates to face-to-face attrition rates for students with a GPA less than 2.0 is actually smaller than for every other GPA group, at a ratio of 1.3; this is similar to the ratio for students with GPA s above 3.49, who have a ratio of 1.5. This indicates that the largest gaps in attrition actually seem to be concentrated among students with GPA s between 2.0 and 3.5; these students withdraw from online courses at double the rate that they withdraw from face-to-face courses.

19 RESTRICTING ENROLLMENT...IS IT WORTH THE COST? / 27 Table 8. Expected Change in Attrition After GPA Restrictions were Imposed Spring 2004 attrition rate for online students with GPA below 2.0 Percentage of spring 2004 online students with GPA < 2.0 % of spring 2004 online students with GPA < 2.0 AND a W/WU grade % of spring 2004 online students who had a GPA < 2.0 AND who did not receive a W/WU Spring 2004 attrition rate for students who had a GPA >=2.0 Actual spring 2004 attrition rate for online courses N for spring 2004 online students N for spring 2004 online students with GPA >=2.0 z-statistic p-value 28% 12.3% 3.4% 8.9% 21.9% 22.7% Table 9. Attrition Rate by GPA for Online and Face-to-Face Courses, Spring 2004 Online Face-to-face Ratio of online vs. face-to-face attrition rates < % 21.1% % 16.9% % 12.2% % 9.7% % 5.8% 1.5 TOTAL 22.7% 15.1% 1.5 Simple Online and Face-to-Face Attrition Rates are Not Comparable: What can be done to avoid comparing apples to oranges when comparing online and face-to-face attrition rates? Simply comparing a simple percentage for attrition for all students in all online classes and comparing it to a simple percentage for attrition for all students in all face-to-face classes does not actually give us comparable data with which to compare the real differences in attrition between these two instructional delivery methods. For one thing, online students are self-selecting, and in an ideal

20 28 / HACHEY, WLADIS AND CONWAY world, the best assessment of online attrition would be to select a reasonably large sample of students and to randomly assign them to either an online or a face-to-face section of the same course taught by the same instructor, and to be certain that both groups matched with regard to GPA and other relevant characteristics. When such randomization is not possible, then the least we can do is to adjust our calculation of attrition rates so that differences between the online and face-to-face population are accounted for. In particular, in order for attrition rates online to be reasonably comparable to attrition rates face-to-face, we must compare students among comparable GPA categories. The composition of the students in particular GPA categories varies from face-to-face courses to online courses. For example, in spring 2004, the percentage of online students with a GPA below 2.0 was 12.3%, in comparison to 36.9% for face-to-face courses, which is three times higher; so even through online courses had a higher attrition rate in spring 2004, it actually had a significantly lower proportion of students with GPA s lower than 2.0. A similar trend can be seen in other GPA categories in Table 10: online courses at the college tend to have fewer students at the low end of the GPA spectrum, and more students at the high end, when compared to face-to-face courses at the college. In fact, if the attrition rate for online courses in spring 2004 is reweighted by taking the proportions of students in each GPA category in face-to-face courses that semester, and supposing that online courses that semester had had the same GPA distribution (and assuming the given attrition rates for each GPA category in online courses in spring 2004), we obtain a reweighted attrition rate of 24.5%, almost two percentage points higher than the actual attrition rate for online courses in spring The reason for this is obvious: the actual online attrition rate is lower than it would be had the courses the same GPA distribution as face-to-face courses, because a smaller proportion of students with low GPA s and a higher proportion of students with high GPA s seem to take courses online at the college in this study (see Table 10). Table 10. Proportion of Students in Each GPA Category, Spring 2004 < Online 12.3% 17.9% 27.6% 24.3% 17.9% Face-to-face 36.9% 16.7% 17.2% 16.4% 12.8%