American Diploma Project Algebra II End-of-Course Exam

Transcription

1 Standard Setting Meeting Chicago, IL July 22-24, 2009 American Diploma Project Algebra II End-of-Course Exam Standard Setting Briefing Book

2 Copyright 2009 Copyright Pearson 2009 Education, Pearson Inc. Education, or its affiliate(s) Inc. or its and affiliate(s). Achieve.

3 1: BRIEFING BOOK OVERVIEW This section of the Briefing Book includes an overview of 1) the American Diploma Project, 2) the ADP Algebra II End-of-Course Exam, 3) the standard setting process, and 4) the validity studies conducted to inform standard setting. About the American Diploma Project In many states today, students can meet the requirements for high school graduation and still be unprepared for success in college or in the workplace. Simply put, current state standards and assessments have not kept pace with the world students are entering after high school. According to a survey commissioned by Achieve, 39 percent of recent graduates enrolled in college and 46 percent in the workforce say there were significant gaps in their preparation for life after high school. Professors and employers estimate that four out of ten graduates are unprepared for college or employment. States, post secondary institutions, employers and young people spend an estimated $17 billion each year on remedial classes to re-teach material that should have been mastered in high school. The price tag might be acceptable if remediation was a proven fix, but one national study indicates that 7 percent of students who take remedial courses in reading and 63 percent of students who take one or two remedial courses in mathematics fail to earn college degrees. In 2004, Achieve published Ready or Not: Creating a High School Diploma that Counts, the result of two years of research launched by Achieve in partnership with The Education Trust and the Thomas B. Fordham Foundation. The report includes English and mathematics benchmarks that describe the specific content and skills that graduates should have mastered by the time they leave high school if they expect to succeed in postsecondary education or in high-growth jobs. In English, this content is equivalent to 4 years of gradelevel English/Language Arts. In mathematics, this content is included in courses typically found in Algebra I, Geometry, and Algebra II, although other pathways can be used. At the time of the creation of the ADP Benchmarks, only Arkansas and Texas required an Algebra II-level course for graduation, and even those states had an opt-out clause. Subsequent reports have assessed the rigor of state high school exit exams and high school graduation requirements, and examined the use of advanced math knowledge and skills in a range of workplaces. The American Diploma Project (ADP) is an initiative created to ensure that all students graduate from high school prepared to face the challenges of work and college. Governors, state superintendents of education, business executives and college leaders are working to bring value to the high school diploma by raising the rigor of high school standards, assessments and curriculum and aligning expectations with the demands of postsecondary education and the workplace. 1-1

4 The ADP Network now includes 35 states, which collectively educate nearly 85 percent of all U.S. public school students: Alabama, Arizona, Arkansas, California, Colorado, Connecticut, Delaware, Florida, Georgia, Hawaii, Idaho, Illinois, Indiana, Kentucky, Louisiana, Maine, Maryland, Massachusetts, Michigan, Minnesota, Mississippi, Nebraska, New Jersey, New Mexico, North Carolina, Ohio, Oklahoma, Oregon, Pennsylvania, Rhode Island, Tennessee, Texas, Virginia, Washington and Wisconsin. CA WA OR NV ID UT MT WY CO ND SD NE KS MN IA MO WI IL MI OH IN KY VT NH ME MA NY RI PA CT NJ DE WV MD VA AZ NM OK AR TN NC SC TX LA MS AL GA AK FL HI What Work Have the ADP Network States Committed to Undertake? Research shows that the ADP benchmarks, which contain skills students need to be successful in college or careers after graduating high school, are significantly more rigorous than current high school standards, resulting in an expectations gap that explains why many high school graduates aren't prepared to succeed when they arrive at college or the workplace. To close the expectations gap, ADP Network states have committed to the following four actions: 1. Align high school standards and assessments with the knowledge and skills required for success after high school. 2. Require all high school graduates to complete a college- and career-ready curriculum, including requiring students to complete a course that covers advanced Algebra, so that earning a diploma assures a student is prepared for opportunities after high school. 3. Build assessments into the statewide system that measure students readiness for college and careers. 4. Develop an accountability system that promotes college and career readiness. Although all Network states are committed to a common set of key policy priorities, there's no one-size-fitsall approach. Each state has developed its own action plan for carrying out the agenda. Each year since the ADP Network was launched in 2005, Achieve has published a report entitled Closing the Expectations Gap to report on the progress the 50 states and DC have made in implementing the pillars of the ADP policy agenda. To date, ADP Network states have led the nation in implementing college- and careerready policies, and many more are in the process of adopting these policies in their states. In 2008, Achieve released a report that described the emergence of a consistent and common core of knowledge in English and mathematics required of students in states that have adopted college- and careerready standards. Some of these states have formally aligned their standards to the ADP benchmarks, but others worked independently. 1-2

5 About the ADP Algebra II End-of-Course Exam As states began work on the ADP agenda, they soon realized they would need new assessments to match their raised expectations. They were also interested in streamlining the transition from high school to college and wanted a test that could signal to high school students whether they were ready for credit bearing college mathematics courses. To that end, the chief state school officers in a number of ADP states formed the ADP Assessment Consortium to create a rigorous college-ready Algebra II end-of-course exam. In March of 2007, original ADP Assessment Consortium members, with the state of Ohio serving as the procurement lead, awarded Pearson the contract to develop and administer the ADP Algebra II End-of-Course Exam. A subset of Assessment Consortium states have also worked together, with Achieve and Pearson, to create an ADP Algebra I End-of-Course Exam. The consortium represents the largest multi-state collaborative assessment ever undertaken. It is a dramatic departure from past testing practices in which states developed their own exams, based on their own standards and often at considerable individual state expense. Currently the ADP Assessment Consortium includes 15 states Arkansas, Arizona, Florida, Hawaii, Indiana, Kentucky, Maryland, Massachusetts, Minnesota, New Jersey, North Carolina, Ohio, Pennsylvania, Rhode Island, and Washington. Additional ADP Network states may also join. Why Algebra II? Because mastery of skills contained in an Algebra II course, or an integrated course covering the same content, is important for all high school graduates, the states wanted to be sure that the courses they offered titled Algebra II contained the right content and were at an appropriate level of rigor. Why does Algebra II matter?! Algebra II fosters problem solving, abstract reasoning and critical thinking skills that are used long after the course ends.! Algebra II and other higher level mathematics classes improve access to postsecondary education. Algebra II includes the advanced content that faculty at two-and four-year institutions say is critical for success in credit-bearing mathematics college coursework.! Students who study mathematics at least through Algebra II in high school are more than twice as likely as those who do not to earn a four-year degree and the level of mathematics a student reaches in high school is the most accurate predictor of whether that student will earn a Bachelor s degree. In contrast, students who have not mastered Algebra II in high school are more likely to need remediation and, therefore, less likely to complete a college degree. Why an End-of-Course Exam? End-of-course exams are attractive with states because they align directly to curriculum standards and courses students need to take for graduation. End-of-course exams are also more sensitive to instruction than are grade-level survey exams because they are taken right after a student has completed a course and can provide teachers with relevant information about students understanding of the content, enabling teacher to adjust instruction for subsequent classes accordingly. In addition, end-of-course tests serve as a way to ensure consistency and rigor in classrooms within and across states, so that all students are exposed to a rigorous curriculum. 1-3

6 The ADP Algebra II End-of-Course Exam serves as a means to ensure consistency and rigor as the number of students enrolled in the course grows, while simultaneously offering students a signal of readiness that can be valued and utilized by postsecondary institutions. What is the Purpose of the Exam? The ADP Algebra II End-of-Course Exam serves three goals: 1. To improve curriculum and instruction and ensure rigor and consistency across states. The test will help classroom teachers focus on the most important concepts and skills in Algebra II and identify areas where the curriculum needs to be strengthened. 2. To serve as an indicator of college readiness. The test is aligned to the ADP mathematics benchmarks and designed to help colleges determine if students are ready to do credit-bearing work. It measures rigorous content and skills students need to enter and succeed in first-year, credit-bearing mathematics courses. Postsecondary institutions will be able to use the results of the test to tell high school students whether they are ready for college-level work, or if they have content and skill gaps that need to be filled before they enroll in college. This information should help high schools better prepare their students for college, and reduce the need for colleges to provide costly remediation courses. 3. To compare performance and progress among the participating states. Having agreed on the core content expectations of Algebra II, states are interested in tracking student performance over time. Each year, Achieve will issue a report comparing performance and progress among the participating states. This report aims to help state education leaders, educators, and the public assess performance, identify areas for improvement, and evaluate the impact of state strategies for improving secondary math achievement. What Does the Exam Measure? The Algebra II End-of-Course Exam was created to provide an honest assessment of how well students have mastered the advanced knowledge and skills that are necessary for success in credit-bearing college mathematics courses. By design the test is challenging. To develop the test, the participating states engaged high school educators and college faculty in all stages of the development of the exam, from the writing of the content standards to reviewing the exam questions to determining how the student responses should be scored. Common, rigorous standards The standards on which the exam is based were developed collaboratively by the partner states, based largely on the ADP mathematics benchmarks. The ADP Algebra II End-of-Course Exam standards are robust, emphasizing advanced algebra, critical thinking, and problem solving. The mathematics content assessed consists of five key strands. A description of each standard and its associated emphasis on the exam is listed in Table 1. In each strand students are expected to model and solve problems in context and translate among multiple representations. 1-4

7 Table 1: Algebra II Exam Standards Standard Topics Addressed Percentage of Total Points Operations on Numbers and Operations with numbers and algebraic expressions, Expressions involving real and complex numbers. 15% Equations and Inequalities Linear and nonlinear equations and inequalities, and systems of linear equations and inequalities. 20% Polynomial and Rational Quadratic functions and higher-order polynomial and Functions simple rational functions. 30% Exponential Functions Exponential functions and basic logarithms and their relationship to exponents. 20% Function Operations and Inverses Combinations and inverses of functions. 15% The emphasis a particular standard was given on the exam is related directly to the emphasis that topic should be given in an Algebra II classroom. Item Types The operational core test includes 55 test questions:! 46 multiple-choice (1 point each)! 6 short-answer (2 points each)! 3 extended-response (4 points each) In response to state requests, the test was designed so that thirty percent of the score is based on the shortanswer and extended-response items. Additional field-test items are embedded in the operational form but do not count towards the student score. Calculator Use In developing the content of the exam, the mathematics experts felt it was necessary for students to demonstrate fluency in mathematics both with and without the use of technology. As a result, the exam is structured into two sections: one that allows calculator, and one that does not. Although not required, the use of a graphing calculator is highly recommended on the calculator section. Room for Growth The ADP Algebra II End-of-Course Exam content standards identify content for the core exam as well as content for seven optional modules. The modules were developed to further challenge students and to enable growth of the ADP Algebra II End-of-Course Exam beyond the traditional Algebra II curriculum. These include: 1. Data and Statistics 2. Probability 3. Logarithmic Functions 4. Trigonometric Functions 5. Matrices 6. Conic Sections 7. Sequences and Series The modules have been field tested but have not been administered operationally. Only the core exam will be considered for standard setting. 1-5

8 How can Post-Secondary Institutions Use this Exam? The ability of the K-12 system to raise standards and adopt a more rigorous curriculum depends heavily on buy-in and support from postsecondary institutions and systems, in sending signals to students, parents, schools and communities that improving standards and achievement at K-12 is essential to student postsecondary success. Higher education s voice in support of raising standards, often in concert with the business community, has led many states to stronger curriculum for all students; the same voices are equally if not more important in making the case for more rigorous high school assessments. Although the Algebra II End-of-Course Exam is not intended as a replacement for college mathematics placement tests or institutional placement instruments such as ACCUPLACER and COMPASS, colleges can use the Algebra II test as an indication of college mathematics readiness for entry-level credit-bearing mathematics courses such as College Algebra. (For higher-level courses such as calculus or pre-calculus, other placement instruments will be necessary.) For example, for a student who does well on the Algebra II exam, a college/department might exempt the student from taking a separate placement test for the first creditbearing mathematics course or courses, provided s/he continues to take advanced mathematics every year until graduation. This would reduce the number of tests a student would take, decrease the cost to colleges and departments for administering tests, and have an added benefit of signaling to high school students that the exam is valued by postsecondary and thus worth being taken seriously both by students and their high schools. When given early enough in a student s high school career, and once results are provided soon after administration, the assessment can also signal to students and schools where improvement/further instruction and support is needed to help students reach college-readiness in mathematics by the time of high school graduation. How does the Content Compare to Other Standards and Exams? Although states are in the process of adopting college- and career-ready standards, the rigorous nature of the ADP Algebra II End-of-Course Exam s items and the exam s purpose to measure mastery of advanced Algebra content, including the knowledge and skills necessary to be successful in a first-year credit-bearing college mathematics courses makes this exam more rigorous than most statewide high school exams. Many state exams, as well as college admissions and placement tests are survey exams, oftentimes measure a broad range of mathematics content that is less advanced than that found in an Algebra II course. Relatively few states have developed their own Algebra II end-of-course exams, which would be more closely tied to the curriculum and the courses that the states require for graduation. Appendix B includes results from four studies conducted by Achieve that describe the variability of Algebra II standards within the United States, the limited focus on Algebra II content found on existing college admissions and placement tests, the Algebra II content included in a sample of College Algebra and Precalculus college courses, and a comparison of the Algebra II exam standards to international standards. Although Algebra II Standards Vary Within the United States, the ADP Algebra II End-of-Course Exam Standards Addresses Many Topics Found in State Courses (Appendix B1) To determine how the ADP Algebra II End-of-Course (EOC) Exam compares to Algebra II expectations from states, Achieve compared the ADP Algebra II End-of-Course Exam standards with the Algebra II course standards and end-of-course exam standards from 15 states. Ten of the states included in the analyses are members of the ADP Network, and two of those ten states are also members of the Assessment Consortium. The data were examined from two different perspectives. 1-6

9 ! The first analysis was conducted to understand the degree to which state Algebra II course and exam standards are aligned, as a whole, with the ADP EOC Assessment.! The second analysis was conducted to better understand how prominently the core ADP Algebra II Exam benchmarks are currently represented in state course and exam standards First Analysis Of the state documents reviewed, most of the state standards documents (18 of 20) are 34% to 66% aligned to the EOC benchmarks. Only two sets of state standards are less than 33% aligned. No set of state standards is 100% aligned to the EOC standards. Second Analysis In comparing the 41 EOC benchmarks across the state standards, seven of the EOC benchmarks that have counterparts in more than 75% of the state documents analyzed and nine benchmarks that have counterparts in fewer than 25% of the state documents. Of the remaining 25 EOC benchmarks, the majority of them (16) have counterparts in 50%-74% of the state documents reviewed. No single benchmark is included in all of the state documents. Summary The purpose of this study was to determine how the ADP Algebra II Exam compares to Algebra II expectations across the states. The findings indicate that there is great variation in Algebra II courses and although there are some common topics, the benchmarks and skills vary within those topics. Although the ADP Algebra II End-of-Course Exam aligns partially with the majority of state standards analyzed, there is not perfect alignment, even with the topics on this exam. In other words, the ADP Algebra II End-of-Course Exam, although not a perfect match, addresses many of the topics found in the state standards documents analyzed. College Admissions and Placement Tests Do Not Fully Measure Algebra II Content (Appendix B2) As many states raise high school standards, college admissions and placements tests are being used for purposes for which they were not intentionally designed. For example, states are using the ACT or SAT as their official statewide high school graduation exam and incorporating these tests into their state assessment and accountability systems. Achieve conducted the study Aligned Expectations? A Closer Look at the College Admissions and Placements Tests to help inform the decisions that states are making about high school assessments by providing information about the content included on college admissions and placement tests. Achieve analyzed more than 2,000 questions from college admissions and placement exams to determine how these tests compare to one another and how well they measure the college and work readiness benchmarks created by the American Diploma Project. For mathematics, the study found that admissions and placement tests in mathematics emphasize algebra, which is critical for credit-bearing mathematics courses. However, the algebra content assessed tends emphasize pre-algebra and basic algebra over the advanced algebraic concepts and skills essential for college readiness. Although placement tests, often developed by each institution, are narrowly focused on algebra, admissions tests such as the SAT and ACT are broader, measuring a range of other important topics such as data analysis, statistics, and geometry. 1-7

10 The ADP Algebra II End-of-Course Exam Standards Provide a Solid Foundation for College Algebra and Pre-Calculus College Courses (Appendix B3) Achieve analyzed college syllabi from 69 College Algebra and 30 Pre-Calculus courses. Professors representing these courses attended one of three judgment studies described in the Judgment Studies (Tab 4). The purpose of this analysis was to provide information regarding the level of college mathematics course that a student performing at the college-ready level on the ADP Algebra II End-of-Course Exam (EOC) would be prepared to take. Each type of course, college algebra and pre-calculus, was analyzed from two different perspectives.! For the first analysis, Achieve content experts reviewed each ADP Algebra II EOC benchmark individually across each course, college algebra and pre-calculus to determine how well the content in the Algebra II exam standards would prepare a student for the content in each course.! For the second analysis, Achieve content experts reviewed each syllabus, as a whole, against the ADP Algebra II EOC standards to determine the degree of overlap between each syllabus and the Algebra II standards. First Analysis When considering each ADP Algebra II EOC benchmark individually with the content topics in the college algebra syllabi, the overlap at the content standard level ranges from 19% to 48% and the overlap of the content in the pre-calculus syllabi and Algebra II EOC exam at the standard level ranges from 7% to 48%, indicating that the ADP Algebra II EOC Exam standards will provide a strong mathematics foundation for students entering either a college algebra or pre-calculus course. Second Analysis When considering each syllabus, as a whole, in comparison to the ADP Algebra II EOC standards, there is considerable overlap between the Algebra II standards and the college algebra syllabi. On average, the college algebra syllabi overlap in 34% of the 41 benchmarks (14 benchmarks). There is less content in common between the ADP Algebra II EOC standards and the pre-calculus syllabi. The pre-calculus syllabi average 24% of the benchmarks (10 benchmarks) overlapping with the Algebra II standards. Summary Overall, the content covered on the ADP Algebra II End-of-Course Exam would provide a solid foundation for either a college algebra or pre-calculus college-level course. There is a small degree of overlap between the Algebra II Exam standards and the college syllabi that would allow for a review of previous mathematics content at the beginning of a college course, but not so much that a student would be repeating the exact course already taken. In fact, the college courses are natural extensions of the ADP Algebra II Exam standards. The standards, therefore, seem at the appropriate level, on average, to prepare students for the first credit-bearing college mathematics courses. The ADP Algebra II Exam Standards are Not Too Rigorous When Viewed in an International Context (Appendix B4) Achieve content specialists compared the content and rigor of the core ADP Algebra II End-of-Course Exam (EOC) Standards with upper-level secondary mathematics standards in eleven countries. Four analyses were included in this study:! The first analysis compared the distribution of mathematics content across the Number, Algebra, Geometry/Measurement, and Data Analysis/Statistics strands.! The second comparison evaluated the level of algebra content, classified as either pre-algebra, basic algebra or advanced algebra.! The third analysis evaluated the international grade placement of the mathematics content 1-8

11 ! The fourth analysis determined the level of rigor of all of the reviewed standards. Overall, the ADP Algebra II End-of-Course Exam standards are well in line with international expectations; in fact a number of countries have standards that are more rigorous. Although not the highest in any of the four analyses, the ADP Algebra II EOC Exam Standards are not considered the lowest in any of the four analyses either. With this in mind, the ADP Algebra II Exam Standards (which were created for a rigorous exam in the United States, where Algebra II is not a mandatory course or exam for many high school students) are average when compared to international standards that all upper secondary students are required to meet in their respective countries. The standards from other countries demand that all of their students master at least this content, if not higher level mathematics content. Administration of the ADP Algebra II End-of-Course Exam Most of the states in the ADP Assessment Consortium are in the process of developing and adopting policies that will govern participation in the exam and address how the exam results will ultimately be used. For the first operational administration in the spring of 2008, states used a variety of approaches, both in terms of who decided which students would take the exam (school, district, or state decision) and how the cost of the exam would be funded (district or state). Two states (Arkansas and Hawaii) required all students who took an Algebra II course in the school year to take the exam in spring Arkansas was one of 20 states at that time that required all students, starting with the class of 2010, to complete Algebra II to graduate from high school. Hawaii is working with both its postsecondary institutions and employers to create incentives for students to complete a set of core curriculum courses, including Algebra II. Other states chose to pilot the exam with only a subset of their students taking the exam, as they considered how to incorporate it into their broader assessment systems. Since the initial 2008 operational administration, Indiana has increased its participation and like Arkansas and Hawaii plans to require all students completing and Algebra II course to take the exam. Spring 2008 Results Nearly 90,000 students from 12 states participated in the first operational administration of the ADP Algebra II End-of-Course Exam in the spring of The following tables provide results from the initial administration, and reflect that overall, across the states the performance was low. This data is shared to provide context about the difficulty of the exam. The results from the spring 2009 administration will be shared with panelists after round 2 of standard setting, and will be reported to schools, districts, and states on August 24, Participation in the spring 2009 exam increased by 10% from last spring over 100,000 students from 13 states participated in the spring 2009 administration. 1-9

12 Table 2: Average Number of Points and Percent Correct, By Question Type and State Multiple-Choice Items Constructed Response All Items (Total possible (Total possible points = Items (Total possible points = 76) 46) points = 30) State* Average Number of Points Correct Average Percent of Points Correct Average Number of Items Correct Average Percent of Items Correct Average Number of Points Correct Average Percent of Points Correct Total % % % AZ % % 2 6.4% AR % % % HI % % 3 8.2% IN % % 2 7.6% KY % % 2 5.4% MN % % 1 4.2% NJ % % % NC % % % OH % % % PA % % 3 8.7% RI % % 2 7.1% WA % % % Table 3: Average Number of Points and Percent Correct, by Content Standard CONTENT STANDARDS Number of Points Possible Average Number of Points Correct Average Percent of Correct Items Operations on Numbers and Expressions Equations and Inequalities Polynomial and Rational Functions Exponential Functions Function Operations and Inverses % 25.0% 31.8% 27.6% 17.9% 1-10

13 Table 4: Average Number of Points and Percent Correct, By Grade Level Average Number of Number of Grade Level Points Correct Students Tested (Out of Possible 76) Average Percent of Correct Points Grade Eight % Grade Nine 5, % Grade Ten 27, % Grade Eleven 42, % Grade Twelve 10, While Maryland and Massachusetts were part of the Assessment Consortium they did not participate in the spring 2008 administration. Florida did not join the Assessment Consortium until Reliability Table 5 below shows the Cronbach s alpha (Alpha), mean raw score (mean), standard deviation of this raw score (S.D), standard error of the mean (SEM), average points correct (p-value), and average point-biserial correlation (Pt.Bis) for the overall group of test-takers for the spring 2008 operational administration. The alpha of 0.87 is consistent with other large-scale assessments and meets industry standards for reliability. The mean score was out of 76 points (27% of points possible). The average p-value was 0.38 which is further evidence of the rigorousness of this content. Table 5: Spring 2008 Overall Reliability Interpreting the Results While performance was low across the states, the ability to interpret the spring 2008 test results and make comparisons across states was limited for several reasons. First, performance standards or cut scores had not yet been established, so there was no clear basis for interpreting the results. Scores were reported based on the percentage of test questions answered correctly. A second challenge in interpreting and comparing the results was that the number of test takers varied significantly across states. In some states all students enrolled in an Algebra II course took the exam, while in others just a subset participated. It is expected that as the states continue to expand their use of the test, the number of test takers will increase and the results will yield more comparable data. When looking at the data in aggregate, initial findings from the first administration suggest:! Student performance was low across all states and in all content strands! Constructed response items are a particular challenge for students! Students who take Algebra II in earlier grades perform better! Student motivation may have impacted performance! States vary in their policies regarding course requirements and data collection 1-11

14 Next Steps for Policymakers Achieve issued The American Diploma Project Algebra II End-of-Course Exam: 2008 Annual Report in August 2008 which included the following next steps for policymakers.! Strengthen K-12 standards! Provide supports to teachers and students! Make college and career readiness the focus on high school assessment and accountability systems! Use the ADP Algebra II Exam to determine if students are prepared to take credit-bearing college mathematics courses, specifically College Algebra or Pre-Calculus About the Algebra II Standard Setting Process Until now, the ADP Algebra II End-of-Course Exam results have been reported in terms of raw scores. This was true for the Spring 2008 and End-of-Fall 2008 administrations. A total of 76 points were possible. Following standard setting, results from the Spring 2009 administration, and future administrations, will be reported in terms of scaled scores and along three performance levels. What is Standard Setting? Standard setting is the process of taking the continuum of student performance on an assessment and separating it into performance levels. The goal of the standard setting is to define the scores on the continuum that separate students from one performance level to the next. Each performance level indicates a student s proficiency in Algebra II and his/her likely preparedness for first-year credit-bearing college mathematics courses. The ADP Algebra II Exam includes three performance levels:! Needs Preparation! Prepared! Well Prepared The knowledge, skills, and abilities required at each performance level are described in the Mapping to Performance Level Descriptors section (Tab 5) and are referred to as Performance Level Descriptors (PLDs). The PLDs were drafted by higher education mathematics faculty and further revised by Pearson, Achieve, and the Assessment Consortium. Who is Involved in Standard Setting? The ADP Algebra II standard setting panel includes:! Fifteen state department of education representatives (one representative from each of the 15 Assessment Consortium states)! Twelve mathematics and higher education representatives (selected by Achieve) All panelists have been involved with the exam during the development and/or administration. The panelists represent a combination of mathematics and policy leadership experience. 1-12

15 The standard setting meeting will be facilitated by Pearson, the developer/owner of the ADP Algebra II Endof-Course Exam. At the request of the ADP Assessment Consortium states, Achieve will attend standard setting as an observer and will have final responsibility for setting cut scores for the ADP Algebra I and II End-of-Course Exams based on recommendations from both standard setting groups. What is the Standard Setting Process for the ADP Algebra II Exam? Achieve has convened an expert group of advisors to provide technical advice and expertise on critical issues, including how to best define achievement levels and what process to use in setting proficiency standards. This group, called the Research Alliance, is comprised of experts in mathematics, assessment, design, K-12 education policy, and higher education. Based on a number of considerations including in particular the multiple purposes of the exam: to improve high school curriculum and instruction; to provide a common measure across states over time; and to serve as an indicator of readiness for first-year college credit-bearing mathematics courses the Research Alliance recommended delaying standard setting from July 2008 (after the first operational administration) to July 2009, so that various validity study evidence could be collected prior to standard setting and allow time for the states to incorporate the standards into their curriculum. It was the Research Alliance s recommendation that delaying standard setting would ground the standard setting in real data about the validity of the exam for indicating college readiness rather than speculation about what the student performance at that level should be. A modified briefing book approach, similar to the one outlined by Haertel (2002) 1 will be used to set standards on the exam. In this approach, policy makers are provided with a briefing book of all of the relevant data related to standard setting decisions and are asked to explicitly consider the policy implications of the full set of data. Haertel argued that stakeholder participation in a rational and coherent deliberative process is necessary to assure that the appropriate validity argument for performance standards can be satisfied. The briefing book itself (which includes evidence from a number of validity and alignment studies) will serve as the basis for the standard setting exercise. For the ADP Algebra II End-of-Course Exam standard setting, panelists will be provided with empirical data from the validity studies, in the form of a briefing book, to provide basis for their policy-based cut score recommendations. Panelists will also review the operational items from the spring 2009 test administration to provide context about the exam, but a traditional item mapping approach will not be followed. Why Not a More Traditional Approach to Standard Setting? Setting standards on a mathematics exam determining what it means to be proficient and what score is necessary to achieve at that level is typically done by mathematicians who review each item on the exam and its difficulty level and use that information to determine the mix of correct items that constitute basic, proficient, or advanced performance of the test content. In this case, proficient must also mean prepared to succeed in a first-year credit-bearing college mathematics course in post-secondary institutions. Therefore, standard setting will rely heaving on data that show the relationship between scores on the Algebra II exam and performance in postsecondary education. In this 1 Haertel, E. H. (2002). Standard setting as a participatory process: Implications for validation of standardsbased accountability programs. Educational Measurement: Issues and Practice, 21 (1),

16 case, an item-based exercise such as a modified Angoff or Bookmark procedure does not by itself provide a sufficient basis for quantifying performance level cuts. How Is College Readiness Defined? There is no universally accepted definition of college readiness. Twenty states have currently developed definitions of college readiness. These definitions aimed at high school students are couched in terms of courses, skills, standards, and/or test performance (Lloyd, 2009) 2. A primary purpose of the Algebra II standard setting is to define a score level on the test that indicates college readiness, that is, the level of performance that indicates a student is prepared for first-year credit-bearing college mathematics courses. For the research studies summarized in this briefing book, Pearson assumed working definitions of college readiness to assist with data interpretation. For example, in the Judgment studies, we asked participating college instructors to think of Prepared students as those that will ultimately earn a B or Better in relevant mathematics courses without remediation. The instructors made item level judgments about performance with these students in mind. For the predictive studies, we focused specifically on the Algebra II test scores as predictors of final course grades of B or higher. These practical definitions are useful in interpreting the results of the research studies and in thinking about their implications for the Algebra II standard setting. What are the Ground Rules for Standard Setting? The goal of the Algebra II standard setting is to obtain your judgments about where the Prepared and Well- Prepared cut scores should be placed. As a participant in the standard setting process, you will review the content of the Algebra II test and be exposed to the policy goals that serve as the basis for the test. Although you may be interested in commenting on the test questions or expressing personal opinions about the goals of the ADP program, the ground rules for the standard setting ask that you refrain from using the meeting time to critique test content or the policy basis for the program. Pearson will have content development staff available at the meeting who will accept written comments related to the content of specific questions should you feel compelled to provide this feedback. 2 Lloyd, S.C. (2009, June 11). Consensus on meaning of readiness remains elusive. Diplomas Count 2009, Education Week, 28(34), p

17 About the Research Used to Inform Standard Setting Pearson conducted a number of validity studies to better understand how the ADP Algebra II Exam fits into the current landscape of mathematics instruction and assessment across public high schools and two- and four-year public colleges. Three main types of validity studies will be used to inform standard setting: 1. Concurrent studies Student scores from the Spring 2008 ADP administration were matched to student scores to other state and national assessments to establish relationships, including those with existing measures of college readiness. 2. Cross-sectional studies The ADP Algebra II Exam was administered to students at the beginning of the semester of their college mathematics course and compared to their final grade in the course to determine how well a student s performance on the exam predicts his/her performance in the college math course. 3. Judgment studies Feedback was gathered from 133 college professors who teach College Algebra and Pre-Calculus courses regarding the relevance of the ADP Algebra II Exam standards to their course, draft performance level descriptors, and recommended cut scores. How is This Data included in the Briefing Book? The concurrent, cross-sectional, and judgment studies are described in sections 2-4 of the Briefing Book. Section 5 includes the Performance Level Descriptors and a study done by Pearson in which the content of each item appearing on the Spring 2009 exam was assigned to a Performance Level based on the Performance Level Descriptions. Section 6 includes a cross-walk of cut scores using the results from the various validity studies and the Performance Level mapping study. Appendix A includes additional information about the methodology used in conducting the validity studies and a complete copy of the research validity plan. Appendix B includes results from the content-based studies conducted by Achieve. What are the Results of the Validity Studies? Concurrent: What relationships exist between the ADP exam and state exams? There are moderate-to-strong relationships across states in the proportion of students who took both the state tests and the ADP exam. Differences across state tests are primarily due to differences in the matched student samples used for the studies Concurrent: What relationships exist between the ADP exam and the national exams? There is a strong relationship across states in the proportion of students who took both the national exams and the ADP exam. Differences in these relationships are primarily due to differences in the matched student samples used for the studies. Cross-Sectional: What ADP score predicts a B or better in College Algebra and Precalculus? Predictive studies suggested scores between 32 and 38 on the 76-point 2008 ADP exam associated with a 65% probability of earning a B or better. Differences in predictions were seen based on institution type (community college, 4-year typical, or 4-year selective) and course (College Algebra or Pre-Calculus). 1-15

18 Contrasting groups studies suggested a 2008 ADP exam score of 25 separates the distributions of students obtaining a B or better in College Algebra and Pre-Calculus from the distributions of students obtaining grades of C, D, or F. Judgment: What feedback was received from College Algebra and Pre-Calculus Professors, and what cut scores did they recommend? College Algebra and Pre-Calculus professors identified some clear content standards that were considered to be important and essential for incoming students to master prior to entry into credit-bearing courses. Median cut scores for the Prepared student ranged from 23 to 38 on the 2008 ADP exam. As expected, results varied based on type of course and institution. How Much Weight Should be Given to Each Study? While all of the studies include some limitations, the results of each study should be considered, along with the considerations, in reaching a cut score recommendation. Of the concurrent studies, the results involving the ACT are probably the strongest because of the large and representative sample and ACT s previouslyestablished benchmark college readiness score of 22. Of the two types of cross-sectional studies, the predictive studies are probably more meaningful than the contrasting groups studies, since they more directly focus on the relationship between ADP exam scores and criterion course grades of B or higher. The judgment studies are also worthy of consideration, given the representative sample of 133 professors representing 79 institutions and 20 different states. What Can Be Inferred from Converging Data? The converging data illustrate the rigor of the ADP exam. Both the empirically-based and judgment studies seem to support placing the Proficient cut score within a range of scores between 24 and 38 on the 2008 ADP exam. Your final recommendations for the ADP exam cut scores should be based on the goals of the exam, the specific information contained in this briefing book, discussions that occur during the standard setting meeting, and your own best judgment. 1-16

19 2. Concurrent Studies

20

21 Executive Summary - Concurrent Validity Studies Background Data! The concurrent validity studies provide data about how the ADP Algebra II End-of- Course Exam relates to other mathematics assessments.! Two types of exams were included in the concurrent validity studies: o National exams, specifically the math sections of the ACT, SAT, and PSAT assessments. o State exams, specifically mathematics assessments administered at the high school level in six different states.! The concurrent validity studies provide indirect information for considering the use of the ADP Algebra II exam to assess college readiness. For example, the SAT and ACT already provide implicit and explicit benchmarks of college readiness. The PSAT is used as a qualifying exam for the National Merit Scholarship program, and the state tests classify students into Proficient and Advanced performance categories.! The relevance of the various concurrent validity studies for the ADP Algebra II standard setting varies for a number of reasons. First, the exams studied vary in how closely they match the content measured by the Algebra II exam. Second, the exams differ in how well performance on them is thought to relate do college readiness. Finally, the amount and quality of the data collected across the various studies differed.! National exam scores matched with spring 2008 ADP Algebra II exam scores: o ACT scores were matched in Arkansas for 6,278 students and in Kentucky for 999 students. o SAT scores were matched in Indiana for 205 students and in Pennsylvania for 414 students. o PSAT scores were matched in Rhode Island for 954 students.! State exam scores matched with spring 2008 ADP Algebra II exam scores: o Hawaii State Assessment (HSA) scores were matched for 1,164 students. o Indiana Statewide Testing for Educational Progress-Plus (ISTEP+) scores were matched for 2,858 students. o Kentucky Core Content Test (KCCT) scores were matched for 923 students. o New Jersey High School Proficiency Assessment (HSPA) scores were matched for 206 students. o Pennsylvania System of School Assessment (PSSA) scores were matched for 3,015 students. o New England Common Assessment Program (NECAP) scores were matched for 882 students in Rhode Island. 2-1

22 Methods! Spring 2008 ADP Algebra II exam scores and scores on the national and state exams were linked using equipercentile linking, resulting in a concordance table for each exam.! For the ACT and SAT, linear regression was conducted to predict ACT/SAT scores exam scores from spring 2008 ADP exam scores.! For the ACT and SAT, logistic regression was conducted to investigate the probability of earning a particular ACT/SAT score, given an ADP exam score. Results! ACT o Strong linear relationship between ADP exam and ACT (r =.698). o The ACT college-readiness score of 22 is associated with spring 2008 ADP exam scores of 25 and 26. o An ADP exam core of 32 is associated with having a 65% chance of earning a 22 or better on the ACT. o An ADP exam score of 31 is associated with a predicted ACT score of 22.! SAT o Moderate correlation between ADP exam and SAT (r =.490). o Using the concordance table between ACT and SAT, the ACT collegereadiness score of 22 is associated with SAT scores of 520 and 530. These scores mapped to ADP exam scores of 23 and 24. o ADP exam scores of 36 and 39 are associated with a 65% chance of earning SAT scores of 520+ and 530+, respectively. o An ADP exam scores between 36 and 40 linearly predict SAT scores in the range.! PSAT o Strong linear relationship between ADP exam and PSAT (r =.616). o The level of PSAT math performance associated with being recognized as a National Merit Scholar corresponds to ADP exam scores of 56 to 59.! State exams o Mean scores on the spring 2008 ADP exam for the matched students varied considerably by state, ranging from 13.8 to o Correlations between ADP exam and the state exams also varied, ranging from.323 (New Jersey HSPA) to (Indiana STEP+). o Based on the concordance tables, the proficient levels on the state exams are associated with ADP exam scores ranging from 12 to 20. o Based on the concordance tables, the advanced levels on the state exams are associated with ADP exam scores ranging from 21 to

23 Considerations! The content match between the exams included in the concurrent studies and the Algebra II exam is limited. o The ACT, SAT, and PSAT contain varying amounts of algebra content, little of which is likely to be at the Algebra II level. o The state exams are primarily used for NCLB and/or as part of graduation. requirements, and are unlikely to measure Algebra II content.! The quality of the concurrent data varies across the studies. o The ACT sample is fairly representative of the ACT testing population. o The SAT sample is lower performing than the SAT testing population. o The PSAT sample is representative of the PSAT testing population in terms of mean performance, but is significantly less variable. o The state exam samples vary in their quality and the ability levels of the students matched to the ADP Algebra II exam.! The ACT concurrent validity study seems most relevant for the purposes of the ADP Algebra II standard setting. o The ACT sample is largest and ACT scores are highly correlated with ADP Algebra II scores. o ACT s definition of college-ready is having a 50% chance of earning a B or better and a 75% chance of earning a C or better in College Algebra. ACT s benchmark for this is an ACT mathematics score of 22 or higher.! The SAT concurrent validity study is relevant for the purposes of the ADP Algebra II standard setting; however, the SAT results should be interpreted with caution. o There is no established college-readiness score for the SAT. o The SAT sample was small and unrepresentative of the SAT testing population. o Correlations with ADP Algebra II scores in the concurrent validity study were lower than would be expected.! The PSAT study provides validity data for the ADP Algebra II exam, but is less relevant for the standard setting. o Although a high correlation between PSAT and ADP scores was found, there is not a clear relationship between PSAT and college-readiness. o The concordance for the benchmark score associated with eligibility for a National Merit Scholarship provides some indication of an upper level of performance on the ADP Algebra II exam; however, this level of performance likely exceeds the level of performance associated with college readiness. o PSAT data were limited to students in Rhode Island.! Although they provide validity evidence for the ADP Algebra II exam, the studies involving state exams are probably least relevant to setting the college readiness standards. o The tests largely measure different mathematics skills at a different level of rigor. o Connections between proficiency levels on the state tests and inferences of college readiness are unknown. 2-3