October 6, 2014 Converting Measures of Academic Progress ( ), Language Usage, and Math s to and Math scaled scores
Quick reference guide to the Assessments used for screening and progress-monitoring assessment is a reliable, valid, and efficient computer-adaptive assessment of general reading achievement and comprehension for grades 1 12. provides nationally norm-referenced reading scores and criterion-referenced scores. A assessment can be completed without teacher assistance in about 10 minutes and repeated as often as weekly for progress monitoring. Math used for screening, progress-monitoring, and diagnostic assessment is a reliable, valid, and efficient computer-adaptive assessment of general math achievement for grades 1 12. Math provides nationally norm referenced math scores and criterion-referenced evaluations of skill levels. A Math assessment can be completed without teacher assistance in less than 15 minutes and repeated as often as weekly for progress monitoring. and Math received the highest possible ratings for screening and progress monitoring by the National Center on Response to Intervention, are highly rated for progress monitoring by the National Center on Intensive Intervention, and meet all criteria for scientifically based progress-monitoring tools set by the National Center on Student Progress Monitoring. 2014 by Renaissance Learning, Inc. All rights reserved. Printed in the United States of America. All logos, designs, and brand names for Renaissance Learning s products and services, including but not limited to Renaissance Learning, Renaissance Place,, Assessments, Math, and are trademarks of Renaissance Learning, Inc., and its subsidiaries, registered, common law, or pending registration in the United States and other countries. All other product and company names should be considered the property of their respective companies and organizations. This publication is protected by U.S. and international copyright laws. It is unlawful to duplicate or reproduce any copyrighted material without authorization from the copyright holder. For more information, contact: RENAIANCE LEARNING P.O. Box 8036 Wisconsin Rapids, WI 54495-8036 (800) 338-4204 www.renaissance.com Page 2 of 13
Project Purpose The purpose of this project is to statistically link the Northwest Evaluation Association (NWEA) Measures of Academic Progress () and Assessments scales in order to facilitate the conversion of s to scaled scores. Linkages were completed between to, Language Usage to, and Math to Math. The resulting conversion table makes it possible for present or future users to translate their s to scaled scores. Assessments Measures of Academic Progress ( ) This report is concerned with the Measures of Academic Progress () assessments in reading, language usage, and math. assessments are K-12 computer adaptive interim assessments taken 3-4 times per year and are used for tracking student progress and growth in basic skills. Every item on a assessment is attached to a vertically aligned equal interval scale, called the RIT scale (Rasch Unit). s range from about 100 to 300 and use a vertical scale to create a grade-independent test score. and Math Both and Math are computer-administered, adaptive measures of general achievement in their respective subjects. They are intended for use as interim assessments that can be administered at multiple points throughout the school year for purposes such as screening, placement, progress monitoring, and outcomes assessment. Renaissance Learning recommends that tests be administered two to five times a year for most purposes, and more frequently when used for progress monitoring purposes. Recent changes to the test item banks and software make it possible to test as often as weekly, for short term progress monitoring in programs such as RTI (response to intervention). and Math fully automate every aspect of a testing program, including test administration, scoring, record-keeping, and report preparation. A core component of these assessment systems is a longitudinal database that contains permanent records of every test administered to a student, both within and across school years. Both are, by design, brief. They place a minimal burden on teacher time, as they can be self-administered, are automatically scored by their internal software, and generate a variety of reports. scores range from 0 to 1400 and use a vertical scale to create a grade-independent test score. Method Data Collection Analysis plans included the evaluation of correlations and statistical linkages between scores on Measures of Academic Progress () and for each test type. Such analyses require matched data, with student records that include both and test scores. Using a secure data-matching procedure compliant with the federal Family Educational Rights and Privacy Act (FERPA), staff from 2 districts provided Renaissance Learning with test scores for students who had taken and/or Math during the 2006 2007, 2007 2008, 2008 2009, 2009 2010, 2010 2011, 2011 2012, 2012 2013, and/or the 2013 2014 school year. Each record in the resulting data file included a student s scores as well as scores on any or Math tests taken during that same year and testing period. Linkages between the and scales and the Language Usage and scales were developed by applying equipercentile linking analysis (Kolen and Brennan, 2004). Because Math scale scores are a direct linear transformation of the underlying Rasch scores, and Math s are also Rasch scores, the two scales are assumed to be related by a linear transformation, and linear equating was identified as the preferable method for linking. Page 3 of 13
Sample These analyses require students take both assessments at about the same time. Linking was completed using a concurrent sample, which included all tests taken within 30 days before or after the mid-date of the administration window. The concurrent sample consisted of a total of 20,847 matched and reading scores, 17,360 matched Language Usage and scores, and 4,643 Math and Math scores. Of the concurrent sample, 10% of the scores were reserved as part of a holdout sample, which was used exclusively to evaluate the linking, and was not used to compute it. All students in the sample were in grades 1 through 8. Table 1a contains sample sizes and descriptive statistics for each subject. Table 1a. Descriptive Statistics for and Scores by Subject Test Subjects and and Language Usage Math and Math Sample Size Holdout Linking Total Mean Standard Deviation Mean Standard Deviation 2,020 18,827 20,847 428.36 260.381 194.46 20.492 1,737 15,627 17,360 439.22 235.569 197.37 17.977 452 4,191 4,643 604.38 164.166 202.27 22.563 Results Scale Linkage The result of the analysis is a set of graphs and tables yielding equivalent scores for each possible score. These results allow the user to look up the and Math score that corresponds to every possible, Language Usage, or Math score. Figure 1a. Linkage of to the RIT Scale Page 4 of 13
Figure 1b. Linkage of to the Language Usage RIT Scale Figure 1c. Linkage of Math to the Math RIT Scale Correlations Three sets of correlations were obtained from the sample: one between the scores and concurrent scores, another between scores and the score equivalents (obtained from the linking), and one between the observed scale scores and the scale score equivalents (obtained from the linking). Tables 2a through 2c display the correlations by subject and grade. For all subjects, the correlations between the and concurrent scores averaged.87 and ranged from.83 to.92. The correlations between scores and score equivalents were higher, as would be expected, averaging.97 and ranging from.95 to 1.00 The correlations between observed scale scores and the scale score equivalents averaged.90 and ranged from.86 to.92. Page 5 of 13
Table 2a. Pearson Correlations between Scale Scores and Scale Scores Observed Score Correlation With: Scale Score Correlation With: Grade Sample size Scale Concurrent Score Score s for Scale Scores s s Overall 20,848.87.96.90 1 2,007.79.96.84 2 4,077.83.98.84 3 3,757.82.98.84 4 3,487.81.97.83 5 2,962.79.96.82 6 1,650.77.95.81 7 1,728.79.96.82 8 1,180.76.96.80 Table 2b. Pearson Correlations between Scale Scores and Language Usage Scale Scores Language Usage Score Correlation Observed Scale With: Score Correlation With: Grade Sample size Concurrent Scale Scores Score s Scale Score s for Language Usage scores Overall 17,360.83.95.86 2 4,050.81.98.82 3 3,711.79.97.81 4 3,174.76.96.78 5 2,724.75.95.77 6 1,353.71.94.75 7 1,430.73.95.77 8 918.69.95.73 Table 2c. Pearson Correlations between Math Scale Scores and Math Scale Scores Observed Math Scale Score Math Score Correlation With: Sample Correlation With: Grade size Concurrent Math Math Score Math Scale Score s Scale Scores s for Math scores Overall 4,643.92 1.00.92 1 230.74 1.00.74 2 702.79 1.00.79 3 676.79 1.00.79 4 701.79 1.00.79 5 569.81 1.00.81 6 706.84 1.00.84 7 413.82 1.00.82 8 646.82 1.00.82 Page 6 of 13
Scaled Score s to s A principal purpose of the linkage between and test scores was to facilitate the conversion of RIT scores to scaled scores. Therefore, the score scale was linked to the score scale yielding a table of equivalent scores for each possible score. Tables 3a-3c display those scores for each subject. Because the linking was done using data that was limited to grades 1-8 and taken from only two districts, these cutscores should be considered approximations that can be updated with greater precision as more data become available in the future. Page 7 of 13
Table 3a. RIT Score and Scaled Score 100-135 0 136 5 137 22 138 23 139 24 140 34 141 39 142 42 143 45 144 49 145 54 146 56 147 58 148 59 149 60 150 61 151 63 152 64 153 66 154 67 155 69 156 71 157 73 158 75 159 78 160 80 161 83 162 85 163 88 164 91 165 96 166 101 167 106 168 117 169 130 170 140 171 150 172 162 173 172 174 181 175 192 176 201 177 212 178 221 179 230 180 238 181 247 182 256 183 267 184 275 185 285 186 293 187 304 188 315 189 323 190 335 191 345 192 356 193 365 194 374 195 385 196 397 197 408 198 422 199 435 200 446 201 456 202 463 203 474 204 488 205 499 206 512 207 524 208 538 209 555 210 567 211 583 212 600 213 618 214 634 215 654 216 676 217 700 218 724 219 757 220 787 221 816 222 845 223 872 224 900 225 920 226 946 227 967 228 988 229 1029 230 1064 231 1102 232 1136 233 1174 234 1199 235 1221 236 1247 237 1263 238 1291 239 1301 240 1312 241 1320 242 1326 243 1330 244 1336 245 1340 246 1343 247 1344 248 1345 249 1346 250-252 1346 253 1347 254-264 1348 265 1366 266 1367 267 1368 268 1369 269 1370 270 1371 271 1372 272 1373 273 1374 274 1375 275 1376 276 1377 277 1378 278 1379 279 1380 280 1381 281 1382 282 1383 283 1384 284 1385 285 1386 286 1387 287 1388 288 1389 289 1390 290 1391 291 1392 292 1393 293 1394 294 1395 295 1396 296 1397 297 1398 298 1399 299-300 1400 Page 8 of 13
Table 3b. Language Usage RIT Score and Scaled Score Language Usage 100-140 0 141 22 142-143 24 144-145 33 146 45 147 53 148 57 149 59 150 61 151 63 152 65 153 67 154 69 155 72 156 74 157 76 158 78 159 81 160 83 161 86 162 88 163 90 164 96 165 101 166 105 167 113 168 126 169 136 170 144 171 151 172 161 173 170 174 179 175 189 176 196 177 204 178 213 179 221 180 228 181 235 Language Usage 182 241 183 250 184 258 185 267 186 275 187 283 188 291 189 300 190 312 191 321 192 333 193 343 194 353 195 362 196 371 197 380 198 396 199 407 200 420 201 434 202 447 203 457 204 466 205 480 206 494 207 508 208 523 209 538 210 556 211 569 212 587 213 604 214 622 215 641 216 665 217 691 218 711 219 739 220 780 221 810 Language Usage 222 844 223 870 224 899 225 921 226 954 227 977 228 1023 229 1068 230 1117 231 1165 232 1194 233 1220 234 1252 235 1268 236 1309 237 1324 238 1332 239 1336 240 1340 241 1343 242 1345 243-246 1346 247 1348 248 1349 249 1350 250 1351 251 1352 252 1353 253 1354 254 1355 255 1356 256 1357 257 1358 258 1359 259 1360 260 1361 261 1362 262 1363 263 1364 264 1365 Language Usage 265 1366 266 1367 267 1368 268 1369 269 1370 270 1371 271 1372 272 1373 273 1374 274 1375 275 1376 276 1377 277 1378 278 1379 279 1380 280 1381 281 1382 282 1383 283 1384 284 1385 285 1386 286 1387 287 1388 288 1389 289 1390 290 1391 291 1392 292 1393 293 1394 294 1395 295 1396 296 1397 297 1398 298 1399 299-300 1400
Table 3c. Math RIT Score and Math Scaled Score Math RIT score 100-118 0 119 1 120 9 121 16 122 23 123 30 124 38 125 45 126 52 127 59 128 67 129 74 130 81 131 88 132 96 133 103 134 110 135 117 136 125 137 132 138 139 139 146 140 154 141 161 142 168 143 175 144 183 145 190 146 197 147 204 148 211 149 219 150 226 151 233 152 240 153 248 154 255 155 262 156 269 157 277 158 284 159 291 160 298 161 306 162 313 163 320 Math Math RIT score 164 327 165 335 166 342 167 349 168 356 169 364 170 371 171 378 172 385 173 393 174 400 175 407 176 414 177 422 178 429 179 436 180 443 181 451 182 458 183 465 184 472 185 480 186 487 187 494 188 501 189 509 190 516 191 523 192 530 193 538 194 545 195 552 196 559 197 567 198 574 199 581 200 588 201 596 202 603 203 610 204 617 205 625 206 632 207 639 208 646 209 653 Math Math RIT score 210 661 211 668 212 675 213 682 214 690 215 697 216 704 217 711 218 719 219 726 220 733 221 740 222 748 223 755 224 762 225 769 226 777 227 784 228 791 229 798 230 806 231 813 232 820 233 827 234 835 235 842 236 849 237 856 238 864 239 871 240 878 241 885 242 893 243 900 244 907 245 914 246 922 247 929 248 936 249 943 250 951 251 958 252 965 253 972 254 980 255 987 Math Math RIT score 256 994 257 1001 258 1009 259 1016 260 1023 261 1030 262 1038 263 1045 264 1052 265 1059 266 1067 267 1074 268 1081 269 1088 270 1095 271 1103 272 1110 273 1117 274 1124 275 1132 276 1139 277 1146 278 1153 279 1161 280 1168 281 1175 282 1182 283 1190 284 1197 285 1204 286 1211 287 1219 288 1226 289 1233 290 1240 291 1248 292 1255 293 1262 294 1269 295 1277 296 1284 297 1291 298 1298 299 1306 300 1313 Math Page 10 of 13
Holdout Mean Differences and Correlations Accuracy of the scale linkage was evaluated using a holdout sample, which contained 10% of the concurrent scores that were not used to compute the linking. The accuracy analysis investigates the correlations and the difference between the observed score summary statistics (mean, standard deviation, minimum, and maximum) and the score equivalents summary statistics. The results of the analysis show that the observed scaled scores are very similar to the score equivalents obtained from the linking. Across test subjects the difference between the observed score mean and the score equivalents mean is 1.28, standard deviation of scores 1.58, minimum score 25, and maximum score 25. Tables 4a and 4b display these statistics by subject. Table 4a. Difference between observed score and score equivalents statistics (Holdout Sample) Difference Scores Subject and and Language Usage Math and Math Holdout Sample Size Mean Standard Deviation of Minimum Maximum 2,020 2.13 2.337 39-1 1,737-0.75 1.827-6 8 452-0.96-0.588-29 65 Table 4b. Correlation between observed scores and score equivalents (Holdout sample) Correlation of Holdout Sample Observed scores with Subject score equivalents and.91 and Language Usage.88 Math and Math.91 Conclusions and Applications The equipercentile linking method was used to link the and scales and the Language Usage and scales, and the linear equating method was used to link the Math and Math scales. The result of each linkage analysis was an estimate of the approximately equivalent scaled score for each. Using the tables of linked scores, educators can now translate their s to scaled scores. Because the linking was done using data that was limited to grades 1-8 and taken from only two districts these cutscores should be considered approximations that can be updated with greater precision as more data become available in the future. Correlations indicated a strong relationship between the and tests. On average, the correlation between and concurrent scores (i.e., tests taken within +/- 30 days of the mid-date) was.87 for and,.83 for Language Usage and, and.92 for Math and Math. Similarly, there were high correlations between observed scores and score equivalents obtained from the linking. On average, the correlation was.90 between observed scores and score equivalents for scores was,.86 for observed scores and score equivalents for Language Usage scores, and.92 for observed Math scores and Math score equivalents for Math scores. Page 11 of 13
References Kolen, M. J. & Brennan, R. R. (2004). Test equating scaling and linking: Methods and practices. New York, NY: Springer Science+Business Media. Renaissance Learning. (2014a). Math: Technical manual. Wisconsin Rapids, WI: Author. Available from Renaissance Learning by request to research@renaissance.com Renaissance Learning. (2014b). : Technical manual. Wisconsin Rapids, WI: Author. Available from Renaissance Learning by request to research@renaissance.com Independent technical reviews of and Math U.S. Department of Education: National Center on Intensive Intervention. (2012a). Review of progress monitoring tools [Review of Math]. Washington, DC: Author. Available online from http://www.intensiveintervention.org/chart/progress-monitoring U.S. Department of Education: National Center on Intensive Intervention. (2012b). Review of progress monitoring tools [Review of ]. Washington, DC: Author. Available online from http://www.intensiveintervention.org/chart/progress-monitoring U.S. Department of Education: National Center on Response to Intervention. (2010a). Review of progress monitoring tools [Review of Math]. Washington, DC: Author. Available online from https://web.archive.org/web/20120813035500/http://www.rti4success.org/pdf/progressmonitoringgo M.pdf U.S. Department of Education: National Center on Response to Intervention. (2010b). Review of progress monitoring tools [Review of ]. Washington, DC: Author. Available online from https://web.archive.org/web/20120813035500/http://www.rti4success.org/pdf/progressmonitoringgo M.pdf U.S. Department of Education: National Center on Response to Intervention. (2011a). Review of screening tools [Review of Math]. Washington, DC: Author. Available online from http://www.rti4success.org/resources/tools-charts/screening-tools-chart U.S. Department of Education: National Center on Response to Intervention. (2011b). Review of screening tools [Review of ]. Washington, DC: Author. Available online from http://www.rti4success.org/resources/tools-charts/screening-tools-chart Page 12 of 13
Acknowledgments The following experts have advised Renaissance Learning in the development of the Assessments. Thomas P. Hogan, Ph.D., is a professor of psychology and a Distinguished University Fellow at the University of Scranton. He has more than 40 years of experience conducting reviews of mathematics curricular content, principally in connection with the preparation of a wide variety of educational tests, including the Stanford Diagnostic Mathematics Test, Stanford Modern Mathematics Test, and the Metropolitan Achievement Test. Hogan has published articles in the Journal for Research in Mathematics Education and Mathematical Thinking and Learning, and he has authored two textbooks and more than 100 scholarly publications in the areas of measurement and evaluation. He has also served as consultant to a wide variety of school systems, states, and other organizations on matters of educational assessment, program evaluation, and research design. James R. McBride, Ph.D., is vice president and chief psychometrician for Renaissance Learning. He was a leader of the pioneering work related to computerized adaptive testing (CAT) conducted by the Department of Defense. McBride has been instrumental in the practical application of item response theory (IRT) and since 1976 has conducted test development and personnel research for a variety of organizations. At Renaissance Learning, he has contributed to the psychometric research and development of Math,, and Early Literacy. McBride is co-editor of a leading book on the development of CAT and has authored numerous journal articles, professional papers, book chapters, and technical reports. Michael Milone, Ph.D., is a research psychologist and award-winning educational writer and consultant to publishers and school districts. He earned a Ph.D. in 1978 from The Ohio State University and has served in an adjunct capacity at Ohio State, the University of Arizona, Gallaudet University, and New Mexico State University. He has taught in regular and special education programs at all levels, holds a Master of Arts degree from Gallaudet University, and is fluent in American Sign Language. Milone served on the board of directors of the Association of Educational Publishers and was a member of the Literacy Assessment Committee and a past chair of the Technology and Literacy Committee of the International Association. He has contributed to both readingonline.org and Technology & Learning magazine on a regular basis. Over the past 30 years, he has been involved in a broad range of publishing projects, including the SRA reading series, assessments developed for Academic Therapy Publications, and software published by The Learning Company and LeapFrog. He has completed 34 marathons and 2 Ironman races. Page 13 of 13 R57878.141006