Achieving the Dream Community Colleges Count WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE BY PAMELA BURDMAN AUGUST 2012
Achieving the Dream Community Colleges Count Achieving the Dream, Inc. is a national nonprofit that is dedicated to helping more community college students, particularly low-income students and students of color, stay in school and earn a college certificate or degree. Evidence-based, student-centered, and built on the values of equity and excellence, Achieving the Dream is closing achievement gaps and accelerating student success nationwide by: 1) guiding evidence-based institutional change; 2) influencing public policy; 3) generating knowledge; and 4) engaging the public. Conceived as an initiative in 2004 by Lumina Foundation and seven founding partner organizations, today Achieving the Dream leads the nation s most comprehensive non-governmental reform network for student success in higher education history. With nearly 200 colleges, 100 coaches and advisors, and 15 state policy teams working throughout 32 states and the District of Columbia Achieving the Dream helps 3.75 million community college students have a better chance of realizing greater economic opportunity and achieving their dreams. WWW.ACHIEVINGTHEDREAM.ORG Jobs for the Future aligns education with today s high-demand careers. With its partners, JFF develops policy solutions and new pathways leading from college readiness to career advancement for struggling and low-income populations in America. WWW.JFF.ORG MDC s mission is to help organizations and communities close the gaps that separate people from opportunity. It has been publishing research and developing programs in education, government policy, workforce development, and asset building for more than 40 years. MDC was the managing partner of Achieving the Dream: Community Colleges Count for six years and was responsible for its incubation as a national nonprofit. It is also the managing partner of the Developmental Education Initiative. WWW.MDCINC.ORG PHOTOGRAPHY: 2010 Mary Beth Meehan, 2008 Community College of Baltimore County, 2004 Community College of Denver
The Developmental Education Initiative consists of 15 Achieving the Dream community colleges that are building on demonstrated results to scale up developmental education innovations at their institutions. Six states are committed to further advancement of their Achieving the Dream state policy work in the developmental education realm. Managed by MDC with funding from the Bill & Melinda Gates Foundation and Lumina Foundation, the initiative aims to expand groundbreaking remedial education programs that experts say are key to dramatically boosting the college completion rates of low-income students and students of color. The innovations developed by the colleges and states participating in the Developmental Education Initiative will help community colleges understand what programs are effective in helping students needing developmental education succeed and how to deliver these results to even more students. WWW.DEIONLINE.ORG ABOUT THE AUTHOR Pamela Burdman is a Chicago-based consultant working with foundations and nonprofits on efforts to improve college readiness and success. She has worked as a journalist, foundation officer, and policy analyst, with an emphasis on higher education policy in California and nationally. As a program officer for four years with The William and Flora Hewlett Foundation, she developed and implemented the foundation s grantmaking strategy to improve student success in California s community colleges. She began her career as a staff writer at the San Francisco Chronicle. Her articles have appeared in The New York Times, San Jose Mercury News, San Francisco Chronicle, Sacramento Bee, Salon, Lingua Franca, Change, National Crosstalk, and the Far Eastern Economic Review, as well as publications by the Institute for College Access and Success and the Spencer Foundation. She is the author of the JFF report Testing Ground: How Florida Schools and Colleges Are Using a New Assessment to Increase College Readiness. ACKNOWLEDGMENTS In addition to participating in JFF s May 2012 convening on assessment, numerous community college leaders and state officials made themselves available for follow-up inquiries to help ensure this policy brief would be accurate and up to date. In particular, the author would like to thank Brad Bostian of Central Piedmont Community College for his admirable patience in responding to inquiries about North Carolina s practices and Shanna Smith Jaggars of the Community College Research Center for addressing questions about research findings. I also thank Lara Couturier, Richard Kazis, Gretchen Schmidt, and Katrina Reichert of Jobs for the Future for their dedication and collaboration.
WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE EXECUTIVE SUMMARY vii SHIFTING GROUND: RESEARCH REVELATIONS SHAKE UP REMEDIAL REFORM 1 SOUL SEARCHING: TESTS BECOME A FOCUS OF INNOVATION 3 RESEARCH CHALLENGES ASSUMPTIONS 5 Placement Exams Are High-stakes Tests 6 The Effectiveness of Traditional Developmental Education Is Unclear 6 Acceleration Strategies Look Promising 7 Placement Exams Are Weak Predictors 8 Assessments Provide a Myopic Picture of College Readiness 8 NEW DIRECTIONS FOR PLACEMENT EXAMS 10 Downplaying the Tests: Systems Reduce Reliance on Placement Scores 11 Multiple Measures 11 Test Waivers 12 Mainstreaming with Support 13 Program-specific Placement 13 Informed Self-placement 14 Changing the Tests: Systems Seek Better Assessments 15 Customized or Aligned Assessments 15 Diagnostic Assessments 15 Assessments of Key Cognitive Strategies 16 Non-cognitive Assessments 16 Supporting Students Around Tests: Systems Change Practices and Conditions 17 College-readiness Tests in High School 17 Senior-year Transitional Courses 18 Test Preparation Assistance 18 Counseling and Advising 19 THINKING ABOUT INNOVATIONS IN PLACEMENT AND ASSESSMENT 20 Cost 20 Time 20 Validity 21 Tensions and Tradeoffs 22 THE NEXT ROUND OF RESEARCH 24 A CHANGE OF HEART 25 ENDNOTES 26 REFERENCES 27 JOBS FOR THE FUTURE v
vi WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE
EXECUTIVE SUMMARY For years, colleges have used placement exams to determine whether to deem incoming students college ready or assign them to developmental education. But emerging information reveals the tests have little correlation to students future success, casting doubt on their use even as the high stakes for students of taking remedial courses become clear. Educators are rethinking whether the tests are fair and wondering if their traditional use constitutes a barrier to college completion. For many states, efforts to strengthen the college readiness of high school graduates and improve college completion rates have created pressure to develop a coherent statewide policy framework for placement assessment. Systems are exploring reforms that range from adopting new tests and deemphasizing test scores to creating new policies for test preparation and administration. As they do, they seek guidance and models. In spring 2012, Jobs for the Future convened state officials and community college leaders including representatives from the Achieving the Dream, Developmental Education Initiative, and Completion by Design state policy networks to discuss research and innovations in placement and assessment and how they might help improve outcomes for students traditionally served by developmental education. Supplementing those discussions with interviews and analysis, Where to Begin? explores the various ways states and systems are grappling with new, sometimes confusing, information about placement exams as part of broader student success initiatives. RESEARCH CHALLENGES ASSUMPTIONS While changes in placement testing policies have come about through a variety of motives and mechanisms, many have been informed by new research. A plethora of studies on a range of topics related to student success has chipped away at many assumptions long shaping thinking about college readiness, creating a new narrative about how best to serve students. While the traditional narrative casts placement exams as low-stakes tests that help students by ensuring they take the appropriate classes, the new narrative emphasizes the role of colleges in facilitating student progress toward graduation. The reform narrative, which underscores the high-stakes nature of consigning students to noncredit remedial sequences with unclear effectiveness, is comprised of five key elements: Placement exams are high-stakes tests. Research challenges the traditional notion that placement exams are lowstakes tests, affecting at most a few courses a student takes. In fact, placement into a developmental course can affect a student s entire educational trajectory, putting additional barriers in the way of a college education. The effectiveness of traditional developmental education is unclear. Students placed into developmental classes are unlikely to complete college. That has been considered the result of poor preparation, but there is increasing scrutiny about whether the developmental experience improves student outcomes. Accelerating some students through or out of developmental courses seems promising. According to early evaluations, some experiments with condensing students time in developmental courses have led to better outcomes for some students. Placement exams are weak predictors of success in gateway courses. In fact, high school grades do a better job. Of particular concern are findings that many students required to take remedial classes could have succeeded in collegelevel coursework. Math and English assessments provide at best a narrow picture of students readiness for college. Placement tests do not measure many of the skills needed for college success including persistence, motivation, and critical thinking. And only some students need most of the assessed math skills. The system of placing and remediating students appears to have veered away from its intended goals. Still, it is one thing to realize this and another to determine how changes in placement policies and exams figure into broader JOBS FOR THE FUTURE vii
developmental reforms. Three broad categories of innovations are being explored: downplaying the tests; changing the tests; and supporting students around the tests. DOWNPLAYING THE TESTS While no state has stopped using placement tests altogether, many are making them less prescriptive or becoming less stringent about requiring students to be assessed. This downgrading takes several forms: Multiple measures: In most states, test scores constitute the only basis for assigning students to remedial classes. Systems are now implementing or considering adding high school grades and other measures. Test waivers: While waiving placement tests based on high SAT or ACT scores has been common, systems are considering high school performance and other additional grounds for placing students directly into college-level courses. Also being explored are practices such as mainstreaming students into college-level courses with extra support, basing placement on students academic goals, and allowing them to make their own placement decisions. CHANGING THE TESTS Several states are considering new assessment instruments to bolster efforts to improve students preparation in high school as well as their outcomes in college. Systems are seeking several key features in these assessments and looking past off-the-shelf assessments to those that are customized for each state. Customized assessments: Several states are adopting assessments aligned with their curricula. Diagnostic assessments: These could offer more information on students strengths and weaknesses than traditional cutoff scores. However, states vary in how they define diagnostic. Colleges and systems are also exploring whether they can better help students by assessing cognitive strategies (e.g., critical thinking, problem solving) as well as non-cognitive factors (e.g., persistence, motivation). SUPPORTING STUDENTS AROUND TESTS Concerns about high developmental education enrollments also drive changes in policies related to test preparation and testing conditions. Systems and colleges are exploring strategies to ensure that rusty skills or a bad day do not relegate students to remedial courses they may not need. College-readiness tests and courses in high school: Some systems have adopted programs or policies for eleventh graders to take college placement tests, based on the theory that the tests send a signal to high schools about the preparation students need. Other test preparation assistance: Many students do not realize the high stakes of placement tests; also, severe placement errors are common. These facts suggest that some students could bypass developmental education if they brushed up their skills in math or English. Colleges experimenting with this approach report early success. THE NEXT ROUND OF RESEARCH As colleges and systems make choices on their own or as a result of legislative mandates the field will need greater understanding about how the new policies are implemented as well as about their results. Key questions for researchers include: > Are the new customized assessments more predictive of student performance than the off-the-shelf tests of the past? > Do efforts to better prepare students and increase awareness of the high-stakes nature of placement tests lead to higher scores and better predictive value? > What strategies best serve underprepared students? viii WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE
SHIFTING GROUND: RESEARCH REVELATIONS SHAKE UP REMEDIAL REFORM REPEATEDLY, POLICYMAKERS HAVE QUESTIONED THE VALUE OF COLLEGE-LEVEL DEVELOPMENTAL EDUCATION. RE-TEACHING HIGH SCHOOL MATERIAL TO COLLEGE STUDENTS SOUNDS LIKE A POOR USE OF TAXPAYER DOLLARS AND STUDENT TIME, WHATEVER THE REALITY. THE NEGATIVE CONNOTATION OF ITS MORE COMMON NAME REMEDIAL EDUCATION DOESN T HELP. YET, DESPITE ITS LOW ESTEEM AND ATTEMPTS TO ABOLISH IT, REMEDIAL EDUCATION CONTINUES TO BE A FIXTURE OF PUBLIC HIGHER EDUCATION, ITS UNDERLYING STRUCTURE REMARKABLY UNCHANGED. IN MOST STATES, STUDENTS ENROLLING IN PUBLIC HIGHER EDUCATION TAKE PLACEMENT EXAMS, AFTER WHICH LARGE NUMBERS OF THEM ARE ASSIGNED TO REMEDIAL COURSES. Consider the City University of New York s high-profile attempt to phase out remedial instruction, beginning in the mid-1990s. Since then, the overall proportion of entering students requiring developmental education has only risen. 1 The difference is that today s students must take developmental courses at two-year, not four-year, campuses. At CUNY and elsewhere, remedial education policies may have changed, but the underlying reality has stayed the same: Large and growing proportions of incoming college students require developmental courses, and increasingly, community colleges bear responsibility for serving them. At two-year colleges nationally, some 60 percent of recent high school graduates and 42 percent of all students take remedial or developmental courses (Bailey 2009; Parsad & Lewis 2003). More recently, these grim statistics have driven new initiatives aimed at strengthening student success: On the one hand, adoption of college-readiness standards in K-12 by many states is intended to improve the preparation of students before they reach college, reducing the need for remediation. On the other hand, higher education institutions, policymakers, and foundations intent on improving college graduation rates are investing in redesigning the very developmental education sequences that present a barrier for so many students. However, just as these efforts to tackle developmental education from both ends appear to be taking root, the ground is shifting in altogether new ways, creating fissures in some of the bedrock assumptions underlying developmental education. With education reformers keenly focused on remedial education, new research using longitudinal data systems questions the efficacy and fairness of the very tests on which the system of remedial education relies. JOBS FOR THE FUTURE 1
Rather than a support to ensure that students take the courses they need, testing policies could pose an unnecessary obstacle to student progression. How colleges and systems react to this new knowledge could redefine the role of developmental education for years to come or speed up the demise some have long advocated. Those responses are playing out as higher education officials strain to keep pace with the winds of reform. In most states, the conversations center on community colleges, though some also involve four-year institutions. With most states adopting the new Common Core State Standards as well as new goals for college attainment, addressing developmental education is becoming a statewide imperative with little time to waste. 2 Traditionally, placement policies in most states were determined at the institution level, and often practices varied widely. More recently, pressure to develop a coherent placement assessment policy framework has made placement policy a state-level issue in a growing number of states, with college systems involved in Achieving the Dream among the leaders of this trend (Collins 2008). The reforms that systems are exploring range from adopting new tests and deemphasizing test scores to creating new policies for test preparation and administration. And system leaders are eagerly looking for guidance and models from other states. This interest was on display in spring 2012 when Jobs for the Future convened state officials and community college officials in Boston. Representatives from the Achieving the Dream, Developmental Education Initiative, and Completion by Design state policy networks discussed the latest research and innovations in placement and assessment and their role in improving outcomes for students traditionally served by developmental education. (See the inside front cover for information on these initiatives.) THE REFORMS THAT SYSTEMS ARE EXPLORING RANGE FROM ADOPTING NEW TESTS AND DEEMPHASIZING TEST SCORES TO CREATING NEW POLICIES FOR TEST PREPARATION AND ADMINISTRATION. AND SYSTEM LEADERS ARE EAGERLY LOOKING FOR GUIDANCE AND MODELS FROM OTHER STATES. Where to Begin? supplements JFF s meeting with additional interviews and analysis to explore the various ways states and especially community college systems are grappling with new and sometimes confusing information about placement exams. They are experimenting, innovating, and changing direction, even as economic challenges loom large. The report also highlights some of the choices and tradeoffs that education policy leaders seeking a rational college placement system ultimately will need to face. 2 WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE
SOUL SEARCHING: TESTS BECOME A FOCUS OF INNOVATION For years, public colleges and universities have used placement exams to determine whether to deem incoming students college ready or assign them to remedial courses. But emerging information reveals that the tests have little correlation to students future success, casting doubt on their use even as the high stakes for students of taking remedial courses are becoming clear. The idea that tests with modest predictive validity could be inhibiting students progress is giving pause to many educators who have long put their faith in them. They are rethinking whether the tests are fair to students and wondering if their traditional use actually constitutes a barrier to improving college completion. Among the North Carolina Community Colleges, soul-searching about the supremacy of test scores is well under way. Like individual colleges and college systems in many states, North Carolina has developed goals for improving college completion. As part of the SuccessNC initiative, the state s community colleges set a target of increasing the percentage of students who transfer, complete credentials, or remain continuously enrolled after six years. As an early Achieving the Dream state that went on to join the Developmental Education Initiative, North Carolina already had a policy team working toward the ultimate goal of improving completion rates of students requiring developmental education. As part of that effort, the system commissioned a study of the efficacy of test scores and high school grades for placing students into developmental education. We had known for a while that the placement test scores probably weren t the best measure, said Van Wilson, the system s associate vice president for student learning and success. Like most college systems in the country, though, North Carolina was basing most placement decisions on those very scores. Despite the cognizance that, like admissions, placement should not be based on test scores and high school grades alone, college leaders were unprepared for what the study, conducted by the Community College Research Center (CCRC), found: It showed that high school grades were a much better predictor of student success in college than placement test scores. In addition, up to one-third of students were found to be severely mis-assigned using placement test results, and that error rate could be cut in half by using high school grades instead of test scores (Belfield & Crosta 2012). 3 When the results of that study were delivered to the presidents, chief academic officers, system office administrators, and faculty on the [Developmental Education Initiative] state policy team, their jaws just dropped, said Wilson. They weren t sure whether to keep the test, change it, or replace it with high school grades. That news came toward the end of 2011, amid a flurry of activity at North Carolina colleges related to college readiness and success. Faculty teams around the state were redesigning developmental math courses as part of the system s involvement in the Developmental Education Initiative. Leaders at five colleges were developing plans for new pathways to maximize students chances of completing credentials as part of the new Bill & Melinda Gates Foundation-funded Completion by Design initiative. And conversations were ongoing with both K-12 and the University of North Carolina system about creating seamless pathways that spanned the entire education system. Changing the college system s placement exam could unsettle all of these projects. But keeping it meant that those efforts would be based on a foundation that seemed to college leaders less and less sound. Around the same time, officials in other states were also in various stages of weighing or implementing changes to their placement exams: > In Connecticut, a Democratic state senator was contemplating introducing legislation for the 2012 session that would strike an even stronger blow to placement exams. As part of an effort to increase the number of Connecticut residents with a college degree, Senator Beth Bye was working on open access legislation that would bar colleges and universities from offering any remedial courses and potentially eliminate placement testing altogether. > Both Florida and Virginia s community college systems were implementing new, customized placement assessments developed with faculty input and aligned with each state s curriculum. In each case, the new test had become a central element in the system s developmental education reform efforts (Burdman 2011; Asera 2011). JOBS FOR THE FUTURE 3
> The Texas Higher Education Coordinating Board was responding to a legislative mandate to adopt a single readiness standard for college-level work in the state s community colleges by soliciting proposals from vendors to develop a new placement test. > In California, following passage of a law to create a common statewide community college placement tool, a legislatively mandated student success task force was recommending that the common instrument provide the ability to diagnose the academic needs of students. (However, neither the legislature nor the task force had identified funding for a new test.) MOTIVES AND MECHANISMS Changes to community college system placement exams or placement policies have been motivated by a variety of priorities and initiated via a range of mechanisms. Here are some examples: Motives for changing placement policies generally relate to improving success for underprepared students (or better preparing students for college). They include an interest in: > Better alignment with a system s curriculum (e.g., after curriculum redesign); > Better alignment with K-12 (and the ability to send clear signals); > A common standard across colleges; > Diagnostic information to assign students to developmental education modules; > Diagnostic information to improve instruction in developmental education generally; and > Interest in more accurate placement (i.e., better predictive validity). Mechanisms or strategies for bringing about change often begin with system or legislative interest in student success. Change may come via a policy mandate or through the establishment of a commission or task force that involves college leaders and instructors in recommending new policies. Examples include: > A state-level commission (Florida); > A system-level initiative or committee (Virginia, North Carolina); > State legislation (Texas, Connecticut); and > A legislatively mandated, system-level task force (California). 4 WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE
RESEARCH CHALLENGES ASSUMPTIONS While changes in colleges placement exam policies have come about through a variety of motives and mechanisms, new understanding about the exams themselves and the placement process has informed these decisions. Since 2004, when foundations began investing in Achieving the Dream along with other community college reform initiatives, two-year colleges have been the focus of a plethora of research studies on a range of topics related to student success. In that time, evidence has chipped away at many of the assumptions that have long shaped colleges thinking about college readiness. As these findings seep into the consciousness of faculty and administrators, a new narrative about how to serve students with weak preparation is emerging. The new narrative sees the role of colleges as facilitating student progress toward graduation. This presents a direct challenge to the traditional notion of colleges enforcing standards to prevent unworthy students from enrolling or earning degrees. The traditional narrative casts placement exams as low-stakes tests that are by and large helpful to students by ensuring that they take the appropriate level classes. The reform narrative underscores the high-stakes nature of consigning students to noncredit remedial sequences with unknown effectiveness, especially in light of new evidence that many of those students might do just as well or better without remediation. Currently, the traditional and reform narratives both have adherents within higher education. 4 However, with attention to graduation and attainment rates rising, the reform line clearly is attracting attention. It is comprised of five key elements that derive from the research literature: > Placement exams are high-stakes tests. THE NEW NARRATIVE SEES THE ROLE OF COLLEGES AS FACILITATING STUDENT PROGRESS TOWARD GRADUATION, PRESENTING A DIRECT CHALLENGE TO THE TRADITIONAL NOTION OF COLLEGES ENFORCING STANDARDS TO PREVENT UNWORTHY STUDENTS FROM ENROLLING OR EARNING DEGREES. > The effectiveness of traditional developmental education is unclear. > Accelerating some students through or out of developmental courses seems promising. > Placement exams are weak predictors of gateway course success; high school grades do a better job. > Math and English assessments provide at best a narrow picture of students college readiness and some assessed skills may not be needed by many college students. JOBS FOR THE FUTURE 5
RESEARCH WRITES A NEW NARRATIVE ABOUT PLACEMENT TESTS TRADITIONAL NARRATIVE REFORM NARRATIVE Placement exams are low-stakes tests. Developmental education helps underprepared students succeed in college. Students cannot successfully skip recommended developmental courses. Placement exams predict whether students can succeed in college-level classes. The math and English skills assessed by placement exams (and taught in remedial courses) are critical to college success. Placement exams are high-stakes tests. The effectiveness of traditional developmental education is unclear. Accelerating some students through or out of developmental courses seems promising. Placement exams are weak predictors of gateway course success; high school grades do a better job. Math and English assessments provide at best a narrow picture of students college readiness and some assessed skills may not be needed by all college students. PLACEMENT EXAMS ARE HIGH-STAKES TESTS The notion that a single test score should not be used to make high-stakes decisions has been a standard of the testing industry for years. 5 In making admissions decisions, for example, selective colleges at a minimum tend to look at students test scores and high school grades. Research has long shown that high school grades do a better job of predicting how students will fare in college than do admissions tests (Geiser & Studley 2003; Geiser & Santelices 2007). When it comes to course placement at less selective schools (many of which do not use admissions tests at all), tests have dominated. Until recently, most colleges have used cutoff scores in English and math to determine whether students are college ready or to place them into multiple levels of developmental education. For some systems, those cutoff scores are set by state policy, reinforcing their importance for colleges. With little research on the topic, it has been easy to view college placement as a low-stakes issue. Whether a student has to take an extra course or even a few never seemed as important an issue as, say, which colleges he or she could attend. Recent research challenges that assumption. A study by CCRC found that nearly one-third of students assigned to developmental education did not complete their developmental sequence because they never enrolled in a developmental course not because they could not handle the coursework. Roughly another 10 percent students who were required to take more than one developmental course passed at least one course but did not take the next course in the sequence (Bailey, Jeong, & Cho 2010). This suggests that placement into a remedial course (and in particular a sequence of courses) may have a significant impact on students trajectories by placing additional hurdles in their way. Additional research by WestEd reveals that many students have no idea that the exams could lengthen their pathways toward a degree. Students in California s community colleges generally experience assessment and placement not as a process for which they begin preparing in high school, but as a single event a one-shot deal, with pivotal consequences, for which many feel uninformed and underprepared, wrote the authors (Venezia, Bracco, & Nodine 2010). Like colleges, students have assumed the stakes are low, the test merely a formality. In effect, whether for students or for colleges, placement exams until recently have been high-stakes tests masquerading as low-stakes tests. THE EFFECTIVENESS OF TRADITIONAL DEVELOPMENTAL EDUCATION IS UNCLEAR Longitudinal studies are showing that students who are placed into developmental classes have a very low likelihood of ever completing college. While that has been considered the result of poor preparation (and possibly a discouraging or cooling out effect of low placement), there is increasing scrutiny about whether the developmental experience improves student outcomes. Several studies have found that students taking developmental courses fare no better in terms of transfer or degree outcomes than similar students who do not take the courses. And in one case, reading in 6 WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE
Florida, assignment to remediation actually had a negative effect on those outcomes (Bailey 2009). These studies do not provide evidence that remedial education does not work. Another explanation could be that colleges are not placing the right students into the courses. However, the research does raise serious questions. As a review of the research noted, Developmental education costs students, the colleges, and the public sector real resources, and in any case it exists to strengthen the outcomes for students concluding that developmental students do as well as similar students who go directly into college courses is not good enough and suggests that remediation wastes money and time (Bailey 2009). Though there have been few studies on teaching in remedial courses, one recent study suggests a need to improve instruction. According to research in California by W. Norton Grubb and colleagues, the courses typically involve drill and practice... on small sub-skills... that most students have been taught many times before, in decontextualized ways that fail to clarify to students the reasons for or the importance of these sub-skills (Grubb, et al. 2011b). Grubb notes that there is some evidence supporting more studentcentered pedagogical approaches such as learning communities, contextualized instruction, and Reading Apprenticeship, and that many other innovations being pursued have not been fully evaluated (Grubb, et al. 2011c). ACCELERATION STRATEGIES LOOK PROMISING Concerned about increasing completion rates, colleges around the country are experimenting with various ways of accelerating students progress by condensing their time in remedial courses. And research is beginning to show that these programs have promise. Students who are placed into shorter remedial sequences (versus longer sequences covering the same material) are more likely to take and pass math and English gatekeeper courses, according to research at the City University of New York, Chabot College, and Community College of Denver. Among the explanations for the better results are: accelerated sequences have fewer exit points and, therefore, fewer chances for diversion or discouragement; exposure to more rigorous coursework may help motivate students; and the students may have been under-placed by test scores, meaning they are required by either college- or state-level policy to take remedial classes even though they could have succeeded in college-level coursework (Hodara & Jaggars 2012; Jaggars 2012). COLLEGES AROUND THE COUNTRY ARE EXPERIMENTING WITH VARIOUS WAYS OF ACCELERATING STUDENTS PROGRESS BY CONDENSING THEIR TIME IN REMEDIAL COURSES. In addition, some programs that place students into college-level courses and provide them with structured supports such as the Accelerated Learning Program at the Community College of Baltimore County, appear successful and cost-effective for students who test into the highest level of developmental education (Jenkins, et al. 2010). Yet the research is preliminary. While the empirical basis for acceleration is not as strong as is desirable, existing evidence suggests that there are a variety of models of course redesign and mainstreaming that community colleges can employ to enhance student outcomes, according to a recent review of the research literature (Edgecombe 2010). In addition, the research does not point to strategies for serving severely underprepared students. Still, early experiments appear promising enough that reformers are embarking on a plethora of initiatives that will, in turn, ideally provide new data for researchers. JOBS FOR THE FUTURE 7
PLACEMENT EXAMS ARE WEAK PREDICTORS With all that evidence accumulating, there was already significant questioning about common practices for remedial placement. The new narrative was further strengthened with the release of a literature review showing the weakness of the evidence for placement exams, followed by the publication of two quantitative analyses showing that they are weak predictors of college success (Hughes & Scott-Clayton 2011; Belfield & Crosta 2012; Scott-Clayton 2012). Of particular concern were findings that many students are under-placed. As with the North Carolina study, the newly published analyses found that, compared to placement test scores, high school grades used alone would increase the success rates of those going directly into college-level courses, reduce the severe error rates in terms of students who appear under-placed, and increase rates of immediate college-level success. The findings apply to both math and English but are more pronounced in English, where test scores explained less than 2 percent of the variation in freshman English grades. The most recent study found that combining high school grades with test scores offered the best predictive value, a finding that mirrors what researchers have been saying for years about admissions tests (Geiser & Santelices 2007). Likewise, studies have found that faculty s biggest complaint about assessments is that they provide no diagnostic information to help instructors understand students strengths and weaknesses. We re taking someone who may have had a bad day and making them take three semesters of remediation, one faculty member told researchers (Grubb, et al. 2011a). ASSESSMENTS PROVIDE A MYOPIC PICTURE OF COLLEGE READINESS Historically, placement test scores have been a proxy for students college readiness. But increasingly the field is realizing that college readiness is not a cutoff score (or two). 6 Besides the limitations of tests for measuring students competency in math and English, colleges are beginning to embrace a more robust understanding of college readiness. Typically cited is David Conley s definition, which grew out of a study involving 400 university faculty and staff identifying the skills that students need for college success. Through subsequent research, the definition has been refined into what Conley now calls the Four Keys to College and Career Readiness (Conley 2007, 2012). At best, standardized tests measure two of the four, as shown in the table below. Educators and policymakers are increasingly interested in psychosocial or non-cognitive factors that is, behaviors such as academic persistence and motivation that shape students ability to learn. 7 Under a more expansive definition of college readiness, the portion assessed by placement tests appears narrow indeed. FOUR KEYS TO COLLEGE AND CAREER READINESS KEY DESCRIPTION MEASURABLE BY STANDARDIZED TESTS? Key Content Knowledge Terms, facts, concepts, ideas, etc. Yes. Key Cognitive Strategies Key Learning Skills and Techniques Key Transition Knowledge and Skills SOURCE: Conley 2012. Strategies such as hypothesizing, analyzing, evaluating, organizing, communicating Skills and behaviors including persistence, motivation, goal setting, note taking (similar to non-cognitive skills or social-emotional learning) Postsecondary awareness (e.g., application and financial aid processes; course selection and academic planning) and skills (previously called College Knowledge) Yes but not all placement exams assess these. Not really. Assessments exist to measure some aspects (but tests used to assess math and English skills do not measure). High school GPA is considered a good proxy. No. Tests used to assess math and English skills generally do not measure. 8 WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE
And even within the content areas assessed by the tests, there are growing questions about whether the tests content aligns with the math skills that students need. The greatest concern is in math because of the great obstacle developmental math courses pose to students who are required to take them. The relatively high proportion of students requiring math remediation and the relatively low rates of success in these courses combine to make placement in developmental math one of the biggest barriers to successful college completion. 8 Until recently, colleges viewed this as a problem requiring more math instruction. More recently, educators are starting to challenge the standard approach to math remediation, starting with the very math that students are expected to learn. There is growing thinking though far from a consensus among researchers who study math education that not all students need the science-oriented math curriculum that is still required in many states. As one overview of new developments in the field said: Although little systematic work has been published in this area, some studies suggest that less than a quarter of all majors require rigorous preparation in calculus. Moreover, a recent study of the use of mathematics in the workplace found that only about one fifth of jobs including high-paying white-collar jobs require more than a deep knowledge of middle school mathematics (Cullinane & Treisman 2010). Not only is better instruction required, the argument goes, but also a different curriculum one that emphasizes statistics and quantitative reasoning rather than algebra and calculus. If indeed the science-oriented courses are not necessary for most students to succeed in college and careers, maintaining them as a requirement needlessly holds back large numbers of students. Based on such insights, dozens of colleges around the country are participating in research and development projects with the Carnegie Foundation for the Advancement of Teaching and the Dana Center at the University of Texas. They are redesigning their math curricula into new pathways oriented toward statistics and quantitative literacy, with names such as Statway, Quantway, and Mathways. 9 JOBS FOR THE FUTURE 9
NEW DIRECTIONS FOR PLACEMENT EXAMS The preponderance of evidence suggests that the system of placing and remediating students has veered away from its intended goals. Still, it is one thing to realize this and another to determine how changes in placement exams figure into a broader remedial reform. Developmental education reforms are increasingly being pursued as statewide policy initiatives, not just individual college projects, underscoring the challenges of addressing academic issues such as testing at the state level. Besides the obvious hurdle of developing innovations that administrators and faculty embrace, there is no template for what those innovations should be or how to implement them. Furthermore, placement exams have a very complicated job. Ideally, they should align with both the higher education curriculum as well as the state s high school curriculum. They need to assess students coming from high school as well as those coming from adult education or the workforce. And, in at least a few states, they identify those students whose skills are not strong enough for developmental education and therefore require Adult Basic Education instead. Not surprisingly, the response to the emerging research is far from uniform. For some, the accumulation of evidence calls into question the whole enterprise of developmental education and the assessments upon which it hinges. In Connecticut, while legislators did not eliminate placement exams, a new law passed in May 2012 will drastically reduce remedial offerings starting in 2014. Only severely underprepared students will be eligible to take remedial coursework, and it will be limited to one semester. PLACEMENT EXAMS HAVE A VERY COMPLICATED JOB. IDEALLY, THEY SHOULD ALIGN WITH BOTH THE HIGHER EDUCATION CURRICULUM AS WELL AS THE STATE S HIGH SCHOOL CURRICULUM. For others, it suggests reforming the existing system rather than eliminating it altogether. North Carolina college leaders decided that the success of their ongoing innovations would be in doubt if colleges continued to rely on their existing placement tests. Our presidents were very clear that with the data that they had received from CCRC they did not want to continue the practice of misplacing students, recalled Wilson. That was our starting point to know we had to start doing things differently. We were jeopardizing the success of so many students because the tools we were using were not effective. The Developmental Education Initiative team that reviewed the placement study recently decided to commission a new test that will be better aligned with the state s curriculum as well as ongoing reforms. The system has signed a contract with the College Board to develop a customized diagnostic assessment that goes beyond multiple-choice items to include interactive questions designed to assess performance and proficiency. But at least as important as replacing the test itself, the system is considering changes to how colleges use the test for placement. 10 A multiple measures committee of the Developmental Education Initiative team is looking at high school grades and other measures that the system might ask colleges to use in addition to test scores. The team is also wondering how non-cognitive factors can be weighed. 10 WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE
Around the country, states and colleges are asking similar questions. While the new assessments may have attracted the most attention, systems are also adopting practices to reduce the weight of test scores or change the conditions under which students prepare for or take the tests. Many of these decisions have been influenced by the emerging research. Three broad categories of innovations are being explored, and states may be considering changes in one, two, or all of these categories: > Downplaying the tests; > Changing the tests; and > Supporting students around the tests. DOWNPLAYING THE TESTS: SYSTEMS REDUCE RELIANCE ON PLACEMENT SCORES While no state system has eliminated the use of placement tests, many are making them less prescriptive or becoming less stringent about requiring students to be assessed. This downgrading of test scores takes several forms. MULTIPLE MEASURES In most states, test scores have been the only basis for assigning students to remedial classes (Collins 2008; Hughes & Scott-Clayton 2011). Test companies long-time caution to use other measures in addition to test scores along with the recent research evidence have combined to make the idea of multiple measures one of the most common reforms that systems are adopting or considering. How to do so is most obvious for recent high school graduates, while measures for nontraditional adult learners are less well-developed. The California community college system has long used multiple measures. Under a 1991 settlement of a civil-rights lawsuit, colleges are not supposed to use a test score as the sole factor in requiring a student to take remedial courses. According to a survey, at least 45 of the state s 112 colleges embed questions within their computerized assessment asking about students experience in the subject, self-reported high school grades, and other relevant experience (Venezia, Bracco, & Nodine 2010). While some colleges automatically factor the responses into the test score algorithm, others use them primarily if a student challenges his or her placement into remedial education (Bunch, et al. 2011). It s non-uniform, noted system Vice Chancellor Patrick Perry. The vast majority of students are not run through a multiple measures system. They have to go back to a counselor if they don t like their test score. High school grades are by far the most commonly mentioned supplemental measure. However, one of the greatest barriers to using these and other multiple measures is a technical one. Few states have fully operational K-16 or P-20 data systems, and even those that do often experience time lags that prevent colleges from accessing students high school records in time to influence a placement decision. They also lack automated systems that allow advisors or faculty to view test scores and high school grades side by side. 11 Another barrier is procedural: the time it takes to vet any changes in policies. Lastly, research has not yet clarified what the shelf life of high school grades should be, and so policymakers looking at adopting such policies are reluctant to consider high school grades for students more than a year or two out of high school. 12 Systems are in various stages of implementing or considering a range of multiple measures: Though North Carolina has the technical capacity to access students high school transcripts, there are practical obstacles. A decision about using those and other measures is being weighed by a multiple measures committee made up of faculty and administrators that is reviewing the research and developing recommendations. At the highest level, it would require state board action, noted Wilson. Before it could get to the state board, it would have to go through the presidents and the chief academic officers. One practice is to use additional measures for students who score just above or below a cutoff score. Earlier this year, New Jersey colleges decided to begin using high school grades to refine placement decisions for students whose ACCUPLACER scores are within the decision zone. In Texas, Austin Community College has trained instructors to evaluate student essays to refine placement decisions for scores in the gray zone. JOBS FOR THE FUTURE 11
Connecticut s new legislation limiting developmental education requires colleges and universities to use multiple measures to determine whether students require college-readiness support. 13 There are some pilots we will be conducting to begin getting those data in. We ll probably start with the high school grades piece because it s something that we know works really well as a predictor, said Braden Hosch, director of policy and research for the Connecticut State Colleges and Universities. I have not talked to a single person who objects to using high school information. TEST WAIVERS While waiving placement tests on the basis of, say, high SAT or ACT scores has been a fairly common practice, state systems are now looking at additional grounds for placing students into college-level courses without testing. As with multiple measures, these include high school performance. Two studies in California using data from the California Partnership for Achieving Student Success (Cal-PASS) are making different arguments for waiving testing. 14 The first was based on curricular alignment: Instructors from a San Diego-area high school worked with nearby college instructors to develop a senior-year course to ensure that students were learning the skills the college courses required. Students who completed the high school course were allowed to take freshman English at the community college regardless of their placement score. More than 80 percent passed the course, compared with an average of 68 percent among other students. The second study involved Long Beach City College, where 90 percent of students from a local unified school district (LUSD) were placing into remedial education, and students were required to complete on average 5.6 semesters of remedial courses. As instructors were considering raising cutoff scores, the study found: Students discipline grades and their overall high school GPA were virtually unrelated to how students were placed into courses at LBCC, but were by far the strongest predictors of performance in our courses.... Initial estimates suggest that such a realignment to employ broadly-based, multiple measures to holistically capture the potential of our students to perform college level work could, in the short-term, meaningfully improve success rates, reduce the number of semesters or required development coursework for LUSD students by more than half, and, for example, increase the number of LUSD students eligible for transfer-level English by almost 500% (RP Perspectives 2012). A statewide research group is now seeking to replicate the study at an additional 22 colleges. A similar finding could influence system-wide policies, said vice chancellor Patrick Perry. The sands are starting to shift away from the idea that we need one single statewide test, he noted. The development of Common Core assessments by two multistate consortia is presenting another opportunity for waiving placement tests. Member states of the Smarter Balanced Assessment Consortium (SBAC) and the Partnership for Assessment of Readiness for College and Careers (PARCC) are expected to THE SANDS ARE STARTING TO SHIFT AWAY FROM THE IDEA THAT WE NEED ONE SINGLE STATEWIDE TEST. PATRICK PERRY, VICE CHANCELLOR, CALIFORNIA COMMUNITY COLLEGES 12 WHERE TO BEGIN? THE EVOLVING ROLE OF PLACEMENT EXAMS FOR STUDENTS STARTING COLLEGE