Excellence for All: World Class Instructional Systems for Our Schools

Transcription

1 Excellence for All: World Class Instructional Systems for Our Schools Setting College-Ready Qualification Scores for English and Mathematics Over the past two years the National Center on Education and the Economy s (NCEE) Technical Advisory Committee has been designing and then executing a plan to establish college-ready qualification scores on the lower division English and mathematics end-of-course examinations offered by ACT s QualityCore program and the University of Cambridge s International General Certificate of Secondary Education (IGCSE) program. Each offers an aligned instructional system that joins rigorous syllabi and curricula with customized professional development, instructional materials and professionally scored examinations that are both driven by the curricula and set to capture college-ready student performances. A core idea here is to ensure that when students leave high school they have accomplished more than accumulating course credits. Rather they have met performance criteria that provide assurances that they are in fact ready to take credit-bearing courses without remediation and succeed in open enrollment colleges. The Excellence for All initiative calls for students to demonstrate proficiency not just in language and mathematics, but in the sciences, history and the arts as well. Students must achieve collegeready scores in all five of these disciplines to earn a proficiency-based diploma, which they can do as early as the end of their sophomore year. Each state participating in Excellence for All is free to set the qualification scores for the science, history and arts courses where it wishes, but the initiative has been designed so that all students, across the states, must meet a common standard in English and mathematics. This is because both students and participating openadmissions postsecondary institutions have been promised that a student who has demonstrated proficiency on these examinations has the mathematical and English literacy needed to succeed in the initial credit-bearing courses in these institutions. Thus the setting of these qualification scores must be based solely on the empirical data that speaks directly to the probability of success in these courses. NCEE recruited some of the world s leading experts in education measurement, cognitive science, language and mathematics to the Technical Advisory Committee (TAC) responsible for setting these qualification scores for the lower division courses. Howard Everson of City University of New York and James Pellegrino of the University of Illinois at Chicago co-chair the committee. 1 The ACT and Cambridge programs mentioned above were selected to serve as the core instructional systems for the lower division, typically the first two years, of high school on the basis of an open competition conducted by the Kentucky Department of Education, using criteria designed to capture the qualities exhibited by the world s leading instructional systems. These choices were vetted by a state panel composed of representatives from ten states. The 1 The other members of the TAC are: Lloyd Bond, Carnegie Foundation for the Advancement of Teaching; Philip Daro, America s Choice; Richard Durán, University of California, Santa Barbara; Edward Haertel, Stanford University; Joan Herman, UCLA; Robert Linn, University of Colorado; Catherine Snow, Harvard University; and Dylan Wiliam, University of London. May 30, 2012

2 2 competition also qualified four upper division programs for high school juniors and seniors: The College Board s Advanced Placement International Diploma program, the International Baccalaureate Diploma program, the University of Cambridge s A-level program and ACT s QualityCore, each of which is designed to prepare students for success at selective colleges and universities. The qualification scores for the lower division programs are not only intended to signal that students are ready for community college but for these upper division high school programs as well. As the TAC approached its work it understood that its central task was to address the fact that during the first few years of Excellence for All there would be no data on how the students in the pilot schools performed in college (since none of these students will have entered college), so proxy measures that link student performance on the QualityCore and IGCSE exams with college grades or other predictors of college success, such as the ACT and SAT exams, would have to be identified and analyzed. The TAC set out to do so and found or developed the following resources: ACT Validation Data These are records on more than 20,000 students who have taken either the QualityCore English 10, Algebra I or Geometry exam in the past several years and then subsequently took the PLAN or ACT college readiness tests or both. Cambridge IGCSE/PSAT Study International students who had just completed their sophomore year of high school took the winter version of the fall 2011 PSAT earlier this year. All 342 of these students had previously taken the IGCSE Mathematics, First Language English and English Literature examinations. Community College Study Over 1,000 first year community college students from eight colleges in four states took one of the three QualityCore exams noted above or the IGCSE Mathematics or First Language English exam earlier this year. Their fall 2011 first semester grades and other data were then linked to their exam scores. The TAC then analyzed these data sets and joined them with other relevant analyses to reach decisions on the qualification scores for the three ACT QualityCore and three Cambridge IGCSE exams that will signify which students meet the English and mathematics criteria to qualify for a proficiency-based diploma. The TAC s view of college-success for the purposes of this work was defined as the student performance level that indicated a student had a 67% chance of earning a first semester GPA of B- (2.75) or better in community college. This was influenced by both the College Board and ACT definitions for their college readiness benchmarks. The College Board benchmark is set where students have a 65% chance of earning a first year GPA 2.7 and the ACT benchmark is set where students have a 50% chance of earning a GPA of B or better and a 75% chance of earning a C or better. The decisions the TAC made and the basis for each now follows.

3 3 ACT QualityCore English 10 QualityCore examinations are scored on a scale that runs from The TAC selected 159 as the qualification score for the English 10 exam. It corresponds to a 67% chance of earning a 2.75 GPA or better in community college. Furthermore, the ACT Validation Study indicates that a score of 159 corresponds to a range of on the PLAN exam (typically given in the 10 th grade) where a 15 is the PLAN college benchmark and 18 is the ACT benchmark. So 159 is higher than the indicator that suggests college success after two more years of high school study (i.e., the PLAN score) and right on the mark for students about to graduate from high school (i.e., the ACT score). Thus, it can fairly be viewed as a rigorous standard and this is quite deliberate as the TAC thinks it important to guard against declaring students college-ready when they are in fact not. The nation already has too many students being awarded high school diplomas only to find that when they arrive on campus they have to take remedial courses, an all too frequent path to dropping out of college. Rectifying this problem rather than perpetuating it is a high priority for Excellence for All. ACT QualityCore Algebra I The ACT Validation Study found that the PLAN mathematics benchmark of 19 corresponds to a range of on the Algebra I exam. The TAC decided on a qualification score in the heart of this range, a score of 150, in part because it projected to a 62% probability of a first semester GPA of 2.75 or higher. It also represented the average performance of the community college students who were taking credit-bearing math courses and was above the average score achieved by students taking remedial math courses. Students taking credit-bearing courses were also carrying solid GPAs across all of their first-year courses. ACT QualityCore Geometry With this being the second of two math exams that QualityCore students need to take to qualify for a proficiency-based diploma, the TAC decided that a less stringent qualification score than the Algebra I score was warranted given that on-going NCEE research on community colleges indicates a much greater need of algebra than geometry knowledge in entry level credit-bearing courses. With this as a backdrop, the TAC selected 145 as the qualification score for QualityCore Geometry, which is associated with ACT scores of even as the ACT readiness benchmark is 22. The TAC found that the community college students it studied who scored 145 or higher on this exam had a 65% chance of earning a GPA of 2.75 or better, and a score of 145 also represented the average performance of the community college students taking credit-bearing mathematics courses in their first year and earning quite respectable GPAs. Cambridge IGCSE First Language English This and other IGCSE exams produce two scores for students: (i) a letter grade that ranges from a high of A* to a low of G; and (ii) a percentage uniform mark (PUM) that ranges from 0-99 (where each letter grade is divided into 10 PUM levels, for example A* ranges from PUMs, A ranges from PUMs, B ranges from PUMs, etc. all the way down to G, which ranges from PUMs). It is also quite important to note that Cambridge letter grades

4 4 do not carry the same meanings we commonly associate with letter grades in American high schools. The best example of this may be a C, a grade viewed by most as quite marginal in American high schools that represents even less at the community college level. But a C in the IGCSE grading system is seen as quite a good grade, to the point that students earning Cs are seen in England as ready for their further education colleges or for entry into upper division A- level courses. With two data sets to rely on to set the qualification score for the First Language English exam, the TAC found the results from international IGCSE students more robust than those from U.S. community college students. The international students took their IGCSE exams at the close of the corresponding IGCSE course, a degree of alignment that none of the U.S. community college students enjoyed. The international students performance on the PSAT also mirrored that of overall U.S. students even though this was a new exam for them, an especially encouraging finding. In contrast, U.S. college students taking the IGCSE exams confronted several obstacles. They hadn t experienced the courses these exams are designed to measure, and the constructed response format of these exams was more demanding than most of the multiple-choice exams U.S. students are used to taking. It appeared these factors caused many of the community college students to struggle through the IGCSE exams. Consequently, community college students didn t do as well on the IGCSE exams as the international students did on the PSAT. Given these circumstances, it was not surprising that the analyses of the two populations of students on the First Language English exam pulled in different directions. The international students who earned A and A* scores on this IGCSE exam produced PSAT scores in critical reading and writing suggesting very strong probabilities of college success, while students earning Bs had PSAT scores indicating somewhat more modest probabilities of success. On the other side of the ledger, the community college students who earned a C or D on their IGCSE exams had average community college GPAs above 3.0. As with the ACT exams, the TAC thought it important to guard against false positives that might send students off to college only to see them fail. So caution on this matter led them to decide that the qualification score should be set at a 70 on the PUM scale, the equivalent of the lowest B score in the IGCSE grading system. Cambridge IGCSE English Literature Here there was only evidence from the sample of international students to rely on as the curriculum and exam for this course are built around a list of readings (drama, poetry and prose) that is prescribed in advance each year, so administering this exam to community college students who had not taken the course made no sense. The TAC, in considering where to set the English Literature qualification score, was sensitive to the fact that this exam represented a second literacy hurdle for students. Interested in not imposing an artificially high qualification score and believing that the First Language English curriculum carried more weight in preparing for college than did the English Literature curriculum, it was decided that, all other things being equal, a less demanding qualification score should be set for the English Literature exam.

5 5 The relationship between the English Literature IGCSE grades and PSAT results mirrored the First Language English results in almost every respect (i.e., A and A* marks were associated with very good PSAT critical reading and writing scores with Bs suggesting more modest probabilities of success). At the same time, the fact that an IGCSE grade of C is considered a college ready score was hard to ignore even as it was associated with somewhat lower PSAT performances by the international students. However, the TAC took into consideration that these students had no prior experience with the PSAT exam and its multiple-choice format, which suggested to them that their PSAT scores likely under-represent these students true academic abilities. Taken together, these factors led the TAC to set the qualification score on the IGCSE English Literature exam at a PUM score of 65, the midpoint between the C threshold of 60 and the B threshold of Cambridge IGCSE Mathematics This exam is designed for a two-year course that has been refined to ensure it is compatible with the new Common Core State Standards in Mathematics. Community college students found this exam to be especially challenging. It might have been an unfamiliar extended response format that threw students off, greater cognitive demand than they were used to, or just unfamiliar content. Whatever the cause, it resulted in a distribution of IGCSE scores that the TAC found not to be useful, so they focused on the international students results, which, as noted above, more closely mirrored the performance of U.S. high school students on the PSAT. Here they found that students earning a B or better on the IGCSE Mathematics exam had average PSAT math scores right at the College Board college readiness benchmark level. Not unexpectedly, students earning Cs on the IGCSE Mathematics exam had somewhat lower PSAT math scores and the likelihood of their reaching the college benchmark was lower than desired. This led the TAC to set the qualification score on the IGCSE Mathematics exam at a PUM score of 65, the midpoint between the C threshold of 60 and the B threshold of Going Forward These qualification scores are the result of extensive deliberations, based on the best information available at this time. Over the next four years the TAC will be gathering additional evidence. It will pay particular attention to the way in which students in the pilot schools perform on the ACT QualityCore and the Cambridge IGCSE examinations in future courses, as well as how they do on college admissions exams as they advance from grade to grade and eventually on the grades they receive in college. The TAC will return to these decisions at least every two years and consider whether the new evidence suggests any refinement in the qualification scores. 2 A letter grade of C on the IGCSE scale is awarded for exams scoring anywhere from PUMs, so not every student who is awarded a C has reached the qualification score. A student must score at least a C that is in the top half of the distribution of C grades (i.e PUMs) to reach the qualification score.