Linking Earlier Grades to STAR College Readiness Cut Scores

Transcription

1 Linking Earlier Grades to STAR College Readiness Cut Scores

2 Quick Reference Guide To The Star Assessments STAR Reading used for screening and progress-monitoring assessment is a reliable, valid, and efficient computer-adaptive assessment of general reading achievement and comprehension for grades STAR Reading provides nationally norm-referenced reading scores and criterion-referenced scores. A STAR Reading assessment can be completed without teacher assistance and repeated as often as weekly for progress monitoring. STAR Math used for screening, progress-monitoring, and diagnostic assessment is a reliable, valid, and efficient computer-adaptive assessment of general math achievement for grades STAR Math provides nationally normreferenced math scores and criterion-referenced evaluations of skill levels. A STAR Math assessment can be completed without teacher assistance and repeated as often as weekly for progress monitoring. Highly rated for progress monitoring by the National Center on Intensive Intervention Highly rated for screening and progress monitoring by the National Center on Response to Intervention Meet all criteria for scientifically based progress-monitoring tools set by the National Center on Student Progress Monitoring. Accelerating Learning for All, Renaissance Learning, the Renaissance Learning logo, Renaissance Place Real Time, STAR. STAR Math, and STAR Reading are trademarks of Renaissance Learning, Inc., and its subsidiaries, registered, common law, or pending registration in the United States and other countries by Renaissance Learning, Inc. All rights reserved. Printed in the United States of America. This publication is protected by U.S. and international copyright laws. It is unlawful to duplicate or reproduce any copyrighted material without authorization from the copyright holder. For more information, contact: RENAISSANCE LEARNING P.O. Box 8036 Wisconsin Rapids, WI (800) answers@renlearn.com 1 -

3 Project Purpose Educators face many challenges; chief among them is making decisions regarding how to allocate limited resources to best serve diverse student needs. A good assessment system supports teachers by providing timely, relevant information that can help address key questions about which students are on track to meet important goals and which students may need additional help. Assessments that can identify students early as being at-risk to miss academic goals can be especially useful because they can help inform instructional decisions that can improve student performance. This project focuses on the application of interim test results in middle school grades to inform educators about which students are on track for college readiness. Specifically, the purpose is to apply previously calculated statistical linkages between Renaissance Learning interim assessments 1 (STAR Reading and STAR Math) and the ACT EXPLORE subtests (ACT, 2011) to earlier grades. Previous research by ACT (2008) has indicated that the majority of eighth grade students are not on track to be ready for college courses by the time they graduate high school. Furthermore, eighth grade academic achievement was a major predictor of college readiness, more so than high school academics. These findings emphasize the importance of upper elementary and middle school grades for preparing students for success after high school. Accordingly, the goal of this project is to find 5 th and 6 th grade score ranges for STAR interim assessments that correspond to college readiness. Such ranges may be useful for the early identification of students at risk of failing to make college readiness goals in reading and math, empowering teachers to provide intervention during formative years when it could be most meaningful. Assessments This report extends previous research linking 7 th grade STAR performance to the 8 th grade college readiness benchmarks for the EXPLORE Reading, English, and Math subtests (Renaissance Learning, 2013c). EXPLORE Though marketed as an assessment for 8 th and 9 th graders, students generally take the EXPLORE in Fall of 8 th grade as part of ACT's College and Career Readiness System which also includes the PLAN in 10 th grade and ACT in 11 th grade. EXPLORE reports scale scores ranging from 1 to 25 to describe a student s degree of college readiness. The college readiness benchmark is 15 for the Reading subtest, 13 for English, and 17 for Math. Students for achieve the benchmark score or higher have a high probability of success in introductory college courses for that subject. In identifying the college readiness benchmark scores for their tests, ACT began by finding the ACT test score associated with a chance of earning a B or higher or about a 75% chance of obtaining a C or higher in credit-bearing first-year college courses commonly taken by first-year college students (e.g., English composition, college algebra, introductory social science courses). Having empirically derived ACT benchmarks based on college performance, EXPLORE benchmarks were then developed by identifying the EXPLORE scores associated with a probability of meeting each of the corresponding ACT benchmarks. For more information about the EXPLORE test and calculations for the college readiness benchmarks, see the EXPLORE Technical manual (ACT, 2011). STAR Reading and STAR Math Both STAR Reading and STAR Math are computer-administered, adaptive measures of general achievement in their respective subjects. Their adaptive nature permits these tests to be administered to students in grades 1 through 12. They are intended for use as interim assessments that can be administered at multiple points throughout the school year for purposes such as screening, placement, progress monitoring, and outcomes assessment. Renaissance Learning recommends that STAR tests be administered two to five times a year for most purposes, and more frequently when used in progress monitoring programs. STAR test item banks and 1 For an overview of the STAR tests and how they work, please see the Technical Manuals cited in the References section. -2 -

4 software make it possible to test as often as weekly for short term progress monitoring in programs such as RTI (response to intervention). STAR Reading and STAR Math fully automate every aspect of a testing program, including test administration, scoring, record-keeping, and report preparation. A core component of these assessment systems is a longitudinal database that contains permanent records of every test administered to a student, both within and across school years. STAR Reading and STAR Math are standardized, nationally normed, computer-adaptive, assessments. Both are, by design, brief. Both place a minimal burden on teacher time, as they can be self-administered, are automatically scored by their internal software, and generate a variety of reports. Furthermore, each student s test is adapted according to his or her previous responses, increasing the accuracy and reliability. Method Data Collection Analyses were conducted using longitudinal data collected from students who had taken STAR Reading and/or STAR Math on Renaissance Learning s Real Time platform. 2 For each student, the data file included 7 th grade STAR scores as well as any STAR scores from 6 th and/or 5 th grade. The dataset spanned seven school years (2005/2006 to 2012/2013) and included records from over 500,000 students. Procedure STAR college readiness ranges for earlier grades were developed by applying logistic regression to ascertain the predicted probability of achieving the 7 th grade STAR college readiness cut score for each STAR scaled score in 5 th and 6 th grade. This method chosen because it yielded ranges based on probability, instead of a single cut score. Ranges seem more appropriate than cut scores in applying college readiness standards to earlier grades, because student growth varies so much during these years. In using these results to evaluate whether a 5 th grader is on track to be college-ready, it is better to think about whether the student is in the right range of performance than whether the student is above or below a benchmark. Using logistic regression, tables were created listing the probability of achieving the 7 th grade college readiness cut score for each possible STAR score. This analysis required a sample of students who took both assessments at about the same time. Sample Each grade was divided into two samples. Linking was completed using a concurrent sample, which included all STAR tests taken 30 days before or after the mid-date of the end-of-year window (mid April to mid June). STAR tests taken outside the +/-30 day concurrent window were part of the predictive sample, which was used to evaluate the accuracy of using the linking results to predict end-of-year STAR performance using STAR tests taken earlier in the school year. In the predictive sample, STAR scores were projected to the mid-date of the concurrent testing window using national growth norms (Renaissance Learning, 2013a, 2013b). National growth norms are based on grade and initial performance, and are updated annually using a five-year period of data which includes millions of students. They provide typical growth rates for students based on starting STAR test score. For each STAR score in the predictive sample, the number of weeks between the STAR administration date and the concurrent window mid-date was calculated. Then the number of weeks was multiplied by the student s expected weekly scaled score growth (based on national growth norms). The expected growth was then added to the observed scaled score to determine the projected STAR score. If a student had multiple STAR tests in the predictive sample, then the projected scores were averaged. 2 Renaissance Place Real Time is a service that involves hosting schools data from the STAR tests and other products. For more information about Renaissance Place Real Time, see

5 Figures 1a and 1b display concurrent sample scatter plots for each subject and dataset. Figure 1a. Scatter Plots for Concurrent STAR Reading Scaled Scores STAR Reading 7th Grade Scaled Score STAR Reading 7th Grade Scaled Score STAR Reading 5th Grade Scaled Score STAR Reading 6th Grade Scaled Score Figure 1b. Scatter Plots for Concurrent STAR Math Scaled Scores STAR Math 7th Grade Scaled Score STAR Math 7th Grade Scaled Score STAR Math 5th Grade Scaled Score STAR Math 6th Grade Scaled Score -4 -

6 Tables 1a and 1b contain sample sizes and descriptive statistics for each subject, grade, and sample. Table 1a. Descriptive Statistics for STAR Reading Matched Scores Grade Sample n Earlier Grade Score 7 th Grade Score M SD M SD Grade 5 Concurrent 138, Predictive 334, Grade 6 Concurrent 286, Predictive 532, Table 1b. Descriptive Statistics for STAR Math Matched Scores Grade Sample n Earlier Grade Score 7 th Grade Score M SD M SD Grade 5 Concurrent 94, Predictive 78, Grade 6 Concurrent 167, Predictive 138, Results Scale Linkage Using concurrent samples, logistic regression was used to estimate the probability of achieving the 7 th grade college readiness cut score for each scaled score in the earlier grade. The result of the analysis was a set of tables yielding probabilities for each possible STAR score which were used to look up which earlier grade scores were associated with a 25%,, and 75% likelihood of achieving the college readiness cut score in 7 th grade (see Figures 2a through 2c). Figure 2a. Linkage of Earlier Grades to 7 th Grade College Readiness Cut Score for Reading of Achieving Grade 7 College Readiness Reading Score % 25% STAR Reading Grade 5 Scaled Score STAR Reading Grade 6 Scaled Score -5 -

7 Figure 2b. Linkage of Earlier Grades to 7 th Grade College Readiness Cut Score for English of Achieving Grade 7 College Readiness English Score % 25% STAR Reading Grade 5 Scaled Score STAR Reading Grade 6 Scaled Score Figure 2c. Linkage of Earlier Grades to 7 th Grade College Readiness Cut Score for Math of Achieving Grade 7 College Readiness Math Cut Score % 25% STAR Math Grade 5 Scaled Score STAR Math Grade 6 Scaled Score Earlier Grade Score Ranges for STAR College Readiness Cut Scores Table 2 displays the STAR range of scores in earlier grades that corresponds to the STAR 7 th grade cut score for each subject. These scores represent a range of probable college readiness that educators can use to guide decisions about instruction for students in late elementary and early middle school grades. -6 -

8 Table 2. STAR 7 th Grade College Readiness Cut Scores and Linked Earlier Grade Score Ranges Subject STAR 7th Grade Cut Score Linked 5th Grade STAR Scaled Score Range Linked 6th Grade STAR Scaled Score Range Prob 25 Prob 50 Prob 75 Prob 25 Prob 50 Prob 75 Reading English Math Classification Accuracy The predictive samples were used to explore the accuracy of the linking using STAR tests taken before or after the concurrent windows. Classification diagnostics were derived from counts of correct and incorrect classifications that could be made when using 5 th and 6 th grade STAR scores to predict whether or not a student would meet the STAR college readiness cut score in 7 th grade. Fifth- and sixth-grade scores associated with a chance of meeting the 7 th grade cut score were used in computing the classification accuracy estimates. The types of classifications are summarized in Table 3a and the classification diagnostic formulas are outlined in Table 3b. Table 3a. Schema for a Fourfold Table of Classification Diagnostic Data EXPLORE College Readiness Result Met Benchmark Below Benchmark Total STAR College Readiness Estimate Met Benchmark Below Benchmark True Positive (TP) False Negative (FN) False Positive (FP) True Negative (TN) Projected Ready (TP + FP) Projected Not (FN + TN) Total Observed Ready (TP + FN) Observed Not (FP + TN) N = TP+FP+FN+TN Table 3b. Descriptions of Classification Diagnostic Accuracy Measures Measure Formula Interpretation Overall classification accuracy Sensitivity Specificity Positive predictive value (PPV) Negative predictive value (NPV) Observed proficiency rate (OPR) Projected proficiency rate (PPR) TP + TN N TP TP + FN TN TN + FP TP TP + FP TN FN + TN TP + FN N TP + FP N Percentage of correct classifications Percentage of college-ready students identified as such using STAR Percentage of not ready students identified as such using STAR Percentage of students STAR finds college-ready who are collegeready according EXPLORE Percentage of students STAR finds not ready who actually are not ready according EXPLORE Percentage of students who achieve EXPLORE benchmark Percentage of students STAR finds college-ready Proficiency status projection error PPR - OPR Difference between projected and observed proficiency rates Classification accuracy diagnostics are presented in Table

9 On average, STAR scores from earlier grades correctly classified students as either college-ready or not (i.e., overall classification accuracy) 82% of the time. The forecasts ranged from 78% accuracy for 5 th grade Math to 84% accuracy for 6 th grade Reading and English. Sensitivity statistics (i.e., the percentage of college-ready students correctly forecasted) averaged 74%, ranging from 55% to 91%. Sensitivity is positively related to observed proficiency rate, with similar trends emerging across the two metrics. Specificity statistics (i.e., the percentage of students correctly forecasted as not ready for college) averaged 79%, ranging from 72% to 91%. Specificity is negatively related to observed proficiency rate, so linkings with higher observed proficiency rates tended to have lower specificity. False positive rates averaged 7%, with a range from 6% to 11%. False positive rates were slightly higher and more variable; they averaged and ranged from 6% to 16%. Positive predictive values averaged 81% and ranged from 75% to 86%, meaning that when earlier grade STAR scores forecasted students to meet the 7 th grade college readiness cut score, they actually did 81% of the time. Negative predictive values averaged 82% and ranged from 79% to 84%. When earlier grade STAR scores forecasted that students would not meet the 7 th grade college readiness cut score, they actually did not 82% of the time. Differences between the observed and projected proficiency rates (i.e., proficiency status projection error) indicated that earlier grade STAR scores tended to accurately predict college-readiness in 7 th grade. Positive values indicate over-prediction and negative values indicate under-prediction. Proficiency status projection errors averaged -3% and ranged from -9% to 5%. STAR scores in earlier grades tended to slightly overpredict college readiness for Reading and slightly under-predict college readiness for English and Math. Finally, the area under the ROC curve (AUC) is a summary measure of diagnostic accuracy. The average AUC was, with a range from 85% to 92%, indicating that earlier grade STAR scores did a very good job of discriminating between which students reached the 7 th grade college readiness cut scores and which did not. Table 4. Classification Diagnostics for Benchmark Forecasts Measure Reading English Math 5th Grade 6th Grade 5th Grade 6th Grade 5th Grade 6th Grade Overall classification accuracy 83% 84% 83% 84% 78% 81% Sensitivity 74% 77% 91% 89% 55% 6 Specificity 89% 89% 72% 77% 91% False positives 6% 6% 11% 9% 7% 6% False negatives 11% 6% 7% 16% 13% Positive predictive value 84% 84% 84% 86% 75% 76% Negative predictive value 82% 84% 83% 82% 79% 83% Observed proficiency rate 43% 42% 62% 61% 35% 32% Projected proficiency rate 38% 39% 67% 63% 26% 26% Proficiency status projection error -5% -4% 5% 2% -9% -7% Area under the ROC curve 91% 92% 91% 92% 85% 87% -8 -

10 Conclusions and Applications Logistic regression was used to estimate what STAR scores in 5 th and 6 th grade were associated with a 25% and 75% probability of meeting the college readiness cut scores for Reading, English, and Math in 7 th grade. Fifth- and sixth-grade scores identified in the linking are considered college readiness ranges that correspond to 7 th grade STAR college readiness cut scores. When projecting 5 th and 6 th grade STAR scores from earlier in the school year to predict 7 th grade performance, students were correctly classified as either college-ready or not 82% of the time, suggesting that the earlier grade college readiness ranges generated by the linking accurately reflect whether students are likely to reach the college-readiness cut score in 7 th grade. References ACT. (2008). The forgotten middle: Ensuring that all students are on target for college and career readiness before high school. Iowa City, IA: ACT, Inc. ACT. (2011). EXPLORE technical manual. Iowa City, IA: ACT, Inc. Renaissance Learning. (2013a). STAR Math: Technical manual. Wisconsin Rapids, WI: Author. Available from Renaissance Learning by request to research@renlearn.com Renaissance Learning. (2013b). STAR Reading: Technical manual. Wisconsin Rapids, WI: Author. Available from Renaissance Learning by request to research@renlearn.com Renaissance Learning. (2013c). Relating STAR Reading and STAR Math to ACT EXPLORE performance. Wisconsin Rapids, WI: Author. Available from Renaissance Learning by request to research@renlearn.com Independent technical reviews of STAR Reading and STAR Math U.S. Department of Education: National Center on Intensive Intervention. (2012). Review of progress monitoring tools [Review of STAR Math]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Intensive Intervention. (2012). Review of progress monitoring tools [Review of STAR Reading]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Response to Intervention. (2010). Review of progress-monitoring tools [Review of STAR Math]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Response to Intervention. (2010). Review of progress-monitoring tools [Review of STAR Reading]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Response to Intervention. (2011). Review of screening tools [Review of STAR Math]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Response to Intervention. (2011). Review of screening tools [Review of STAR Reading]. Washington, DC: Author. Available online from R