A + dvancer College Readiness Online Remedial Math Efficacy: A Body of Evidence

Transcription

1 Running Head: A+dvancer COLLEGE READINESS ONLINE 1 A + dvancer College Readiness Online Remedial Math Efficacy: A Body of Evidence John Vassiliou Miami Dade College, FL Deborah Anderson Front Range Community College, CO Laura Reed Windham Regional Career Center, VT Nadine Cornett Miami Valley Tech Prep Consortium, OH Bill Cornett Miami Valley Tech Prep Consortium, OH Nicholas B. McDonald The American Education Corporation, OK 2010

2 A + dvancer COLLEGE READINESS ONLINE 2 A + dvancer College Readiness Online Software and Remedial Education for Math The value of education becomes ever more apparent and desirable as society modernizes, and this is reflected in the changing demographics of who enrolls in college. In the United States, individuals are making this value choice clear by the increasing number of enrollees at two- and four-year degree-granting institutions. While the number of college freshmen increased by 14% during the ten years between 1987 and 1997, that percentage increased to 26% for the years 1997 to While the percentage of students under 25 years old is increasing (up to 33% from 13%), the percentage of older students returning to college is expected to increase over the next few years. Minority populations are also increasingly enrolled in college courses, expanding the minority population of college students from 15% in 1976 to almost a third (32%) in Due to the current recession, this accelerating trend will probably continue to increase college enrollment, both as traditional and non-traditional first-time enrollees and returning students. Remedial education, academic work that is considered less rigorous than college-level work, is a controversial subject for higher education. Conceptually, the idea of remedial education is simple. Identify via testing students who lack certain critical knowledge and then correct that deficiency with supplemental efforts. However, some research suggests that students who take remedial courses are more likely to succeed (e.g., McCabe, 2000) while other research suggests that too much remedial coursework is associated with increased dropout rates (Deli- Amen & Rosenbaum, 2002; National Center for Education Statistics, 2005). These efforts, mostly in the form of remedial coursework, consume a large amount of resources, both in personnel and money. Estimates vary, but one recent estimate (Strong American Schools, 2008) puts the amount at about two billion dollars annually. In 2000, the number of institutions offering

3 A + dvancer COLLEGE READINESS ONLINE 3 at least one remedial course in reading, writing, or mathematics was about 76%. Different types of institutions had different rates of offering such courses; for example, about 98% of public twoyear colleges offered at least some form of remedial education as compared to rates ranging from 59 to 80 percent for other colleges. Among community colleges (Attewell, Lavin, Domina, & Levey, 2006), up to 58% of the students are enrolled in remedial courses with 44% taking more than one course. Mathematics is the most common remedial subject. About one in four college students enrolls in remedial courses with about twice as many enrolling in mathematical courses rather than reading or writing courses. Most students spent less than one academic year in remedial courses, although up to 5% did exceed the one-year mark. While the percentage of students enrolling in one or more remedial courses did not change significantly between 1995 and 2000, the average length of time that the students spent in those courses increased. This may be due to increased non-academic demands on student time forcing an increase in time to complete the coursework. According to an Office of Program Policy Analysis & Government Accountability (OPAGA) report (Report No , April 2006), in the academic year, 78% of students entering two-year higher education institutions and 10% of entering freshmen at four-year higher education institutions in Florida required remediation in one or more content areas (English, Reading, and/or Mathematics). Overall, the report found that almost two-thirds (62%) of students needed remediation in multiple content areas. According to statistical data produced by the Miami Dade College Institutional Research Department, 63,165 college students (75.2% of the student population) pursuing an Associates in Arts, Science, and Applied Science degrees and Vocational Credit Certificates required remediation in one or more subject areas.

4 A + dvancer COLLEGE READINESS ONLINE 4 The increased need for remediation for college students has increased time for degree completion and increased cost to complete those degrees. For example, in the state of Ohio between , only 15% of remedial students earned a bachelor s degree within 6 years as compared to 47% of the non-remedial students. However, non-remedial students engage in behaviors that are more costly than remedial (attend 4-year colleges more, choose more costly majors, etc.) thus making direct comparisons difficult. For example, actual costs per credit hour are less for remedial students, but this is because they tend to take lighter course loads. Drains on state budgets are significant. For example in , Colorado spent 11.4 million dollars in public money for remedial instruction which does not count the additional costs to students in terms of tuition, increased time to degree completion, additional housing costs, etc. A similar state expense is found in Florida where the total cost of remediation at Florida's 28 public community colleges for was about 32,250,000 dollars. This illustrates a pressing need to reduce costs of remedial education from both the student and college perspective. One way to do this is via Computer Assisted Instruction (CAI). CAI has certain clear advantages over traditional instruction including the ability to reach larger numbers of students, greater accessibility, more flexibility, individualized lesson plans, ability to customize schedules, and ease of updating instructional information. In this paper four independent data sets of remedial students from multiple states who utilized one type of CAI, the A + dvancer College Readiness Online software, are analyzed. The general efficacy of the software is examined with regard to overall improvement with The College Board's ACCUPLACER, and ACT's COMPASS course placement tests, and A+dvancer assessments for two mathematical areas: Arithmetic and Elementary Algebra.

5 A + dvancer COLLEGE READINESS ONLINE 5 Method Data Sets Florida. Over three semesters (summer 2007, spring 2008, and fall 2008), a total of 88 Miami Dade College first-time-in-college (FTIC) students participated in this research. The semester breakdown was 27 students in summer 2007, 37 in spring 2008, and 24 in the fall semester of The majority (69 out of 88) of these students had Hispanic backgrounds with 18 being non-hispanic and one with no response. The majority (69 out of 88) of students were U.S. citizens with an average age of about 24 years (ages varied from 18 to 53) and a gender distribution of 45 females and 43 males. The average amount of time spent on A+dvancer Arithmetic lessons for 43 students was about 10 hours and 55 minutes (with a standard deviation of about 15 hours and 16 minutes). The average amount of time spent on A+dvancer Algebra lessons for 10 students was about two hours and three minutes (with a standard deviation of about one hour and 59 minutes). Ohio. As part of the initial assessments of a longitudinal study, 114 Ohio students utilized A+dvancer College Readiness Online in College Tech Prep coursework. These students spent an average of 3 hours and 55 minutes working with the A+dvancer program over two months. These students were drawn from a rural school with 52 (45.6%) being female and 62 (54.4%) male. The students were predominantly Caucasian (109 or 95.6%) with the remainder being either Asian (2 or 1.8%) or Hispanic (3 or 2.6%). None of these students were classified as having English as a second language nor were any classified as special needs students.

6 A + dvancer COLLEGE READINESS ONLINE 6 Colorado. These 22 remedial Front Range Community College students studied remedial Arithmetic. While the gender, ethnic background, and age of these students are unknown, none were classified as having English as a second language or as special needs. General population characteristics for this college in 2009 have the enrolled student body being about 76% non-minority, 57% female, 67% non-campus resident, and a median age of 23 years. The amount of time students spent working with A+dvancer software was also not available. Vermont. Twenty-two entry-level Windham Regional Career Center students in need of remedial mathematical instruction participated in this 2007 educational intervention; however, only 17 completed both the ACCUPLACER Arithmetic pre- and post-assessments, while only 20 completed the ACCUPLACER Elementary Algebra pre- and post-assessments. The gender, ethnic background, and age of these students were not recorded. At the end of 2007 each student spent an average of 10 hours and 10 minutes spread out over about 70 days on an average of 16 prescribed lessons before being re-assessed with an ACCUPLACER exam. Materials The A+dvancer software. A+dvancer is an online instructional system that is designed to provide individualized learning prescriptions. The prescriptions are generally developed because of brief criterion-referenced assessments within the A+dvancer system that are associated with specific learning skills identified by ACCUPLACER and COMPASS. A+dvancer then provides a skill deficiency report and individualized instruction to address the specific learning needs of the student. ACCUPLACER Assessment. ACCUPLACER is a web-based set of six multiplechoice computerized placement tests in a range of English, reading, and math subjects designed

7 A + dvancer COLLEGE READINESS ONLINE 7 to determine whether or not a student has the skills to be successful in college-level courses. This assessment has established predictive validity and reliability (Mattern & Packman, 2009). The tests were developed with the help of faculty committees and are produced by the College Entrance Examination Board. COMPASS Assessment The COMPASS computer-adaptive college placement test is used by educators to quickly evaluate incoming students' skill levels in language arts and mathematical skills. The results are widely used to place students in appropriate courses and match school resources to the appropriate students to achieve academic success (COMPASS, 2010). The Math Placement Test is a multiple-choice test that evaluates students' ability in terms of basic skills. These include performing a sequence of basic operations, application skills such as applying those sequences of basic operations to unique problems, and analysis skills such as demonstrating conceptual understanding of principles and relationships for mathematical operations. The Math Placement Test offers five subject areas: Pre-Algebra (Arithmetic), Algebra, College Algebra, Geometry, and Trigonometry. Results Mathematics: Arithmetic and Elementary Algebra Arithmetic Florida Data. Significant within-group results were found for the increase in participant mean score performances for all four content areas. Arithmetic scores increased significantly over time by points: t(46) = 4.321, p <.001. Please see Figure 1 for a graph of these results and Table 1 for the pre- and post-test means.

8 A + dvancer COLLEGE READINESS ONLINE 8 Figure 1. Change in ACCUPLACER score over time for Arithmetic test. Table 1. Mean ACCUPLACER scores for all Pre- and Post-tests. ACCUPLACER Assessment Mean Score Number of Participants Standard Deviation Arithmetic Pre-Test Arithmetic Post-Test Ohio Data. Significant improvements in students arithmetic scores are found by comparing the starting and ending COMPASS assessments for 130 students. Significant differences (p <.001) of almost 38 points are found between the starting score (100.48) and the ending score (138.46). See Figure 2 for a graph of the results and Table 2 for the pre- and posttest means.

9 A + dvancer COLLEGE READINESS ONLINE 9 Figure 2. Starting and ending COMPASS Arithmetic mean scores. Table 2. COMPASS Mean scores for Arithmetic Pre- and Post-tests. COMPASS Assessment Mean Score Number of Participants Standard Deviation Arithmetic Pre-Test Arithmetic Post-Test Vermont data. Significant within-group results were found for the increase in participant mean score performance. Arithmetic scores increased significantly over time by 28 points: t(16) = 6.019, p <.001. Please see Figure 3 for a graph of these results and Table 3 for the pre- and post-test means.

10 Mean Score on Arithmetic Test A + dvancer COLLEGE READINESS ONLINE 10 Figure 3. Change in ACCUPLACER score over time for Arithmetic test Pre-Test Post-Test Table 3. Mean scores for Arithmetic Pre- and Post-tests. ACCUPLACER Assessment Mean Score Number of Participants Standard Deviation Arithmetic Pre-Test Arithmetic Post-Test Colorado data. Significant within-group results were found for the increase in participant mean score performance. A+dvancer arithmetic scores increased significantly over time by points: t(21) = 7.646, p <.001. Please see Figure 4 for a graph of these results and Table 4 for pre- and post-test means.

11 A + dvancer COLLEGE READINESS ONLINE 11 Figure 4. A+dvancer means for Arithmetic Pre- and Post-tests. Table 4. A+dvancer mean scores for Arithmetic Pre- and Post-tests. A+dvancer Assessment Mean Score Number of Participants Standard Deviation Arithmetic Pre-Test Arithmetic Post-Test Elementary Algebra Florida Data. Algebra scores also increased significantly over time by points: t(53) = 5.345, p <.001. Please see Figure 5 for a graph of these results and Table 5 for the pre- and post-test means.

12 A + dvancer COLLEGE READINESS ONLINE 12 Figure 5: Change in ACCUPLACER score over time for Algebra test. Table 5: Mean ACCUPLACER scores for Algebra Pre- and Post-tests. ACCUPLACER Assessment Mean Score Number of Participants Standard Deviation Algebra Pre-Test Algebra Post-Test Vermont data. Algebra scores also increased significantly over time by points: t(19) = 3.075, p =.002. Please see Figure 6 for a graph of these results and Table 6 for the pre- and post-test means.

13 Mean Score on Algebra Test A + dvancer COLLEGE READINESS ONLINE 13 Figure 6: Change in ACCUPLACER score over time for Algebra test Pre-Test Post-Test Table 6: Mean scores for Elementary Algebra Pre- and Post-tests. ACCUPLACER Assessment Mean Score Number of Participants Standard Deviation Elementary Algebra Pre-Test Elementary Algebra Post-Test Discussion The results from the four independent data sets of remedial students strongly suggest that CAI in the form of the A+dvancer College Readiness Online software produces a significant increase in the performance of students in need of remedial mathematical instruction in both the areas of arithmetic and algebra. Students who took the algebra coursework saw improvements from pre- to post-test of 30% to 56%, while students working with the arithmetic software had similar improvements ranging from 34% to 58%. It is not trivial to note that this general

14 A + dvancer COLLEGE READINESS ONLINE 14 improvement is replicated across two topic areas with four samples drawn from different geographical locations and utilizing different objective performance assessments. This replication and similarity of results further strengthens the argument that CAI does achieve the goal of improving student mathematical skills to the point where they are ready for more advanced college-level coursework. These improvements in remedial skills were made following a relatively brief intervention, ranging from about 4 to 10 hours per student over a period of 2 to 3 months. The fact that these significant advancements were made by remedial students over a relatively short time is especially encouraging. Shortening the expense in both time and money for students to master the basic skills needed to succeed in college-level mathematical coursework is becoming an increasingly important service for academic institutions to provide due to the increasing numbers of students attending institutions of higher learning. There are limitations on the interpretations and conclusions that can be drawn from the data sets due to the method of data collection for this examination. These restrictions to interpretation include the lack of random assignment and non-cai comparison groups. This lack of random assignment does limit the ability to state conclusively that use of the CAI was responsible for the increases in observed performance. Unfortunately, the potential lack of efficacy for individual students of one randomly assigned instructional method versus another makes many universities and colleges reluctant to endorse such true experimental designs. This limits the collected data to a more observational than experimental classification. Additional limitations to this study are the lack of follow-up data on the students who did significantly improve their scores. What sort of success did those students have in their future college classes? This deficit of supplementary longitudinal data, like graduation rates or higher-

15 A + dvancer COLLEGE READINESS ONLINE 15 level courses grades, prevents longer-term conclusions about the efficacy of this CAI. The lack of demographic analysis along gender, racial, and age-related dimensions further restricts the generalization of the results to specific demographic groups that individual institutions might be serving. Improving the methodology of future research into this specific CAI should attempt to overcome the obstacles of random assignment of students to differing methods of remedial instruction. This goal may prove elusive, so statistical methods of control differences between non-randomly assigned instructional groups (such as covariance or matching) may have to be substituted. Beyond methodological considerations, additional information as to the long-term success of CAI versus more traditional remedial instruction is needed. The success of those CAI remedial students compared to the success of more traditionally educated remedial students on such measures as college-level coursework, drop-out rates, time to degree completion, and actual graduation rates would provide further insight into any lasting differences between CAI and traditional instructional methods. Interactions between demographic variables such as English as a second language or student age and CAI is of interest along with the patterns of the software use such as software use time, time spent in practice exams, use of auxiliary materials, would further clarify the multitude of variables impacting student performance. Finally, CAI efficacy with additional remedial topics such as language and science need to be investigated. However, these significant results of 30 to 50% improvements after using the A+dvancer College Readiness Online software for a relatively short period of time produce a significant increase in the performance of students in need of remedial mathematical instruction.

16 A + dvancer COLLEGE READINESS ONLINE 16 References American Education Corporation (2010). A+dvancer College Readiness Online software. Oklahoma City, OK: The American Education Corporation. [On-line]. Attewell, P., Lavin, D., Domina, T., & Levey, T. (2006). New evidence on college remediation. Journal of Higher Education, 77(5), COMPASS (2009). COMPASS Guide to Effective Student Placement and Retention in Mathematics. Iowa City, Iowa: ACT. [On-line]. Deil-Amen, R., and Rosenbaum, J. E. (2002). The unintended consequences of stigma-free remediation. Sociology of Education, 75, Mattern, Krista, D. & Packman, Sheryl (2009). Research Report No : Predictive Validity of ACCUPLACER Scores for Course Placement: A Meta-Analysis. The College Board, New York, [On-Line] pdf McCabe, Robert H. (2000) No One to Waste: A Report to Public Decision-Makers and Community College Leaders. Washington, DC: Community College Press. National Center for Education Statistics. (2005). Condition of education: Student effort and educational progress: Postsecondary persistence and progress. [On-line]. Available: National Center for Education Statistics. (2000). Digest of education statistics. [On-line]. Available:

17 A + dvancer COLLEGE READINESS ONLINE 17 Office of Program Policy Analysis & Government Accountability (OPAGA) report (Report No , April 2006) Office of Program Policy Analysis & Government Accountability, Florida (2006, April). Steps Can Be Taken to Reduce Remediation Rates; 78% of Community College Students, 10% of University Students Need Remediation. (Report No ). Tallahassee, FL. [On-Line]. Strong American Schools. (2008). Diploma to nowhere. Washington DC. [On-line]. Available: