Converting Measures of Academic Progress (MAP ) Reading, Language Usage, and Math RIT scores to STAR Reading and STAR Math scaled scores

Transcription

1 October 6, 2014 Converting Measures of Academic Progress ( ), Language Usage, and Math s to and Math scaled scores

2 Quick reference guide to the Assessments used for screening and progress-monitoring assessment is a reliable, valid, and efficient computer-adaptive assessment of general reading achievement and comprehension for grades provides nationally norm-referenced reading scores and criterion-referenced scores. A assessment can be completed without teacher assistance in about 10 minutes and repeated as often as weekly for progress monitoring. Math used for screening, progress-monitoring, and diagnostic assessment is a reliable, valid, and efficient computer-adaptive assessment of general math achievement for grades Math provides nationally norm referenced math scores and criterion-referenced evaluations of skill levels. A Math assessment can be completed without teacher assistance in less than 15 minutes and repeated as often as weekly for progress monitoring. and Math received the highest possible ratings for screening and progress monitoring by the National Center on Response to Intervention, are highly rated for progress monitoring by the National Center on Intensive Intervention, and meet all criteria for scientifically based progress-monitoring tools set by the National Center on Student Progress Monitoring by Renaissance Learning, Inc. All rights reserved. Printed in the United States of America. All logos, designs, and brand names for Renaissance Learning s products and services, including but not limited to Renaissance Learning, Renaissance Place,, Assessments, Math, and are trademarks of Renaissance Learning, Inc., and its subsidiaries, registered, common law, or pending registration in the United States and other countries. All other product and company names should be considered the property of their respective companies and organizations. This publication is protected by U.S. and international copyright laws. It is unlawful to duplicate or reproduce any copyrighted material without authorization from the copyright holder. For more information, contact: RENAIANCE LEARNING P.O. Box 8036 Wisconsin Rapids, WI (800) Page 2 of 13

3 Project Purpose The purpose of this project is to statistically link the Northwest Evaluation Association (NWEA) Measures of Academic Progress () and Assessments scales in order to facilitate the conversion of s to scaled scores. Linkages were completed between to, Language Usage to, and Math to Math. The resulting conversion table makes it possible for present or future users to translate their s to scaled scores. Assessments Measures of Academic Progress ( ) This report is concerned with the Measures of Academic Progress () assessments in reading, language usage, and math. assessments are K-12 computer adaptive interim assessments taken 3-4 times per year and are used for tracking student progress and growth in basic skills. Every item on a assessment is attached to a vertically aligned equal interval scale, called the RIT scale (Rasch Unit). s range from about 100 to 300 and use a vertical scale to create a grade-independent test score. and Math Both and Math are computer-administered, adaptive measures of general achievement in their respective subjects. They are intended for use as interim assessments that can be administered at multiple points throughout the school year for purposes such as screening, placement, progress monitoring, and outcomes assessment. Renaissance Learning recommends that tests be administered two to five times a year for most purposes, and more frequently when used for progress monitoring purposes. Recent changes to the test item banks and software make it possible to test as often as weekly, for short term progress monitoring in programs such as RTI (response to intervention). and Math fully automate every aspect of a testing program, including test administration, scoring, record-keeping, and report preparation. A core component of these assessment systems is a longitudinal database that contains permanent records of every test administered to a student, both within and across school years. Both are, by design, brief. They place a minimal burden on teacher time, as they can be self-administered, are automatically scored by their internal software, and generate a variety of reports. scores range from 0 to 1400 and use a vertical scale to create a grade-independent test score. Method Data Collection Analysis plans included the evaluation of correlations and statistical linkages between scores on Measures of Academic Progress () and for each test type. Such analyses require matched data, with student records that include both and test scores. Using a secure data-matching procedure compliant with the federal Family Educational Rights and Privacy Act (FERPA), staff from 2 districts provided Renaissance Learning with test scores for students who had taken and/or Math during the , , , , , , , and/or the school year. Each record in the resulting data file included a student s scores as well as scores on any or Math tests taken during that same year and testing period. Linkages between the and scales and the Language Usage and scales were developed by applying equipercentile linking analysis (Kolen and Brennan, 2004). Because Math scale scores are a direct linear transformation of the underlying Rasch scores, and Math s are also Rasch scores, the two scales are assumed to be related by a linear transformation, and linear equating was identified as the preferable method for linking. Page 3 of 13

4 Sample These analyses require students take both assessments at about the same time. Linking was completed using a concurrent sample, which included all tests taken within 30 days before or after the mid-date of the administration window. The concurrent sample consisted of a total of 20,847 matched and reading scores, 17,360 matched Language Usage and scores, and 4,643 Math and Math scores. Of the concurrent sample, 10% of the scores were reserved as part of a holdout sample, which was used exclusively to evaluate the linking, and was not used to compute it. All students in the sample were in grades 1 through 8. Table 1a contains sample sizes and descriptive statistics for each subject. Table 1a. Descriptive Statistics for and Scores by Subject Test Subjects and and Language Usage Math and Math Sample Size Holdout Linking Total Mean Standard Deviation Mean Standard Deviation 2,020 18,827 20, ,737 15,627 17, ,191 4, Results Scale Linkage The result of the analysis is a set of graphs and tables yielding equivalent scores for each possible score. These results allow the user to look up the and Math score that corresponds to every possible, Language Usage, or Math score. Figure 1a. Linkage of to the RIT Scale Page 4 of 13

5 Figure 1b. Linkage of to the Language Usage RIT Scale Figure 1c. Linkage of Math to the Math RIT Scale Correlations Three sets of correlations were obtained from the sample: one between the scores and concurrent scores, another between scores and the score equivalents (obtained from the linking), and one between the observed scale scores and the scale score equivalents (obtained from the linking). Tables 2a through 2c display the correlations by subject and grade. For all subjects, the correlations between the and concurrent scores averaged.87 and ranged from.83 to.92. The correlations between scores and score equivalents were higher, as would be expected, averaging.97 and ranging from.95 to 1.00 The correlations between observed scale scores and the scale score equivalents averaged.90 and ranged from.86 to.92. Page 5 of 13

6 Table 2a. Pearson Correlations between Scale Scores and Scale Scores Observed Score Correlation With: Scale Score Correlation With: Grade Sample size Scale Concurrent Score Score s for Scale Scores s s Overall 20, , , , , , , , , Table 2b. Pearson Correlations between Scale Scores and Language Usage Scale Scores Language Usage Score Correlation Observed Scale With: Score Correlation With: Grade Sample size Concurrent Scale Scores Score s Scale Score s for Language Usage scores Overall 17, , , , , , , Table 2c. Pearson Correlations between Math Scale Scores and Math Scale Scores Observed Math Scale Score Math Score Correlation With: Sample Correlation With: Grade size Concurrent Math Math Score Math Scale Score s Scale Scores s for Math scores Overall 4, Page 6 of 13

7 Scaled Score s to s A principal purpose of the linkage between and test scores was to facilitate the conversion of RIT scores to scaled scores. Therefore, the score scale was linked to the score scale yielding a table of equivalent scores for each possible score. Tables 3a-3c display those scores for each subject. Because the linking was done using data that was limited to grades 1-8 and taken from only two districts, these cutscores should be considered approximations that can be updated with greater precision as more data become available in the future. Page 7 of 13

8 Table 3a. RIT Score and Scaled Score Page 8 of 13

9 Table 3b. Language Usage RIT Score and Scaled Score Language Usage Language Usage Language Usage Language Usage

10 Table 3c. Math RIT Score and Math Scaled Score Math RIT score Math Math RIT score Math Math RIT score Math Math RIT score Math Page 10 of 13

11 Holdout Mean Differences and Correlations Accuracy of the scale linkage was evaluated using a holdout sample, which contained 10% of the concurrent scores that were not used to compute the linking. The accuracy analysis investigates the correlations and the difference between the observed score summary statistics (mean, standard deviation, minimum, and maximum) and the score equivalents summary statistics. The results of the analysis show that the observed scaled scores are very similar to the score equivalents obtained from the linking. Across test subjects the difference between the observed score mean and the score equivalents mean is 1.28, standard deviation of scores 1.58, minimum score 25, and maximum score 25. Tables 4a and 4b display these statistics by subject. Table 4a. Difference between observed score and score equivalents statistics (Holdout Sample) Difference Scores Subject and and Language Usage Math and Math Holdout Sample Size Mean Standard Deviation of Minimum Maximum 2, , Table 4b. Correlation between observed scores and score equivalents (Holdout sample) Correlation of Holdout Sample Observed scores with Subject score equivalents and.91 and Language Usage.88 Math and Math.91 Conclusions and Applications The equipercentile linking method was used to link the and scales and the Language Usage and scales, and the linear equating method was used to link the Math and Math scales. The result of each linkage analysis was an estimate of the approximately equivalent scaled score for each. Using the tables of linked scores, educators can now translate their s to scaled scores. Because the linking was done using data that was limited to grades 1-8 and taken from only two districts these cutscores should be considered approximations that can be updated with greater precision as more data become available in the future. Correlations indicated a strong relationship between the and tests. On average, the correlation between and concurrent scores (i.e., tests taken within +/- 30 days of the mid-date) was.87 for and,.83 for Language Usage and, and.92 for Math and Math. Similarly, there were high correlations between observed scores and score equivalents obtained from the linking. On average, the correlation was.90 between observed scores and score equivalents for scores was,.86 for observed scores and score equivalents for Language Usage scores, and.92 for observed Math scores and Math score equivalents for Math scores. Page 11 of 13

12 References Kolen, M. J. & Brennan, R. R. (2004). Test equating scaling and linking: Methods and practices. New York, NY: Springer Science+Business Media. Renaissance Learning. (2014a). Math: Technical manual. Wisconsin Rapids, WI: Author. Available from Renaissance Learning by request to Renaissance Learning. (2014b). : Technical manual. Wisconsin Rapids, WI: Author. Available from Renaissance Learning by request to research@renaissance.com Independent technical reviews of and Math U.S. Department of Education: National Center on Intensive Intervention. (2012a). Review of progress monitoring tools [Review of Math]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Intensive Intervention. (2012b). Review of progress monitoring tools [Review of ]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Response to Intervention. (2010a). Review of progress monitoring tools [Review of Math]. Washington, DC: Author. Available online from M.pdf U.S. Department of Education: National Center on Response to Intervention. (2010b). Review of progress monitoring tools [Review of ]. Washington, DC: Author. Available online from M.pdf U.S. Department of Education: National Center on Response to Intervention. (2011a). Review of screening tools [Review of Math]. Washington, DC: Author. Available online from U.S. Department of Education: National Center on Response to Intervention. (2011b). Review of screening tools [Review of ]. Washington, DC: Author. Available online from Page 12 of 13

13 Acknowledgments The following experts have advised Renaissance Learning in the development of the Assessments. Thomas P. Hogan, Ph.D., is a professor of psychology and a Distinguished University Fellow at the University of Scranton. He has more than 40 years of experience conducting reviews of mathematics curricular content, principally in connection with the preparation of a wide variety of educational tests, including the Stanford Diagnostic Mathematics Test, Stanford Modern Mathematics Test, and the Metropolitan Achievement Test. Hogan has published articles in the Journal for Research in Mathematics Education and Mathematical Thinking and Learning, and he has authored two textbooks and more than 100 scholarly publications in the areas of measurement and evaluation. He has also served as consultant to a wide variety of school systems, states, and other organizations on matters of educational assessment, program evaluation, and research design. James R. McBride, Ph.D., is vice president and chief psychometrician for Renaissance Learning. He was a leader of the pioneering work related to computerized adaptive testing (CAT) conducted by the Department of Defense. McBride has been instrumental in the practical application of item response theory (IRT) and since 1976 has conducted test development and personnel research for a variety of organizations. At Renaissance Learning, he has contributed to the psychometric research and development of Math,, and Early Literacy. McBride is co-editor of a leading book on the development of CAT and has authored numerous journal articles, professional papers, book chapters, and technical reports. Michael Milone, Ph.D., is a research psychologist and award-winning educational writer and consultant to publishers and school districts. He earned a Ph.D. in 1978 from The Ohio State University and has served in an adjunct capacity at Ohio State, the University of Arizona, Gallaudet University, and New Mexico State University. He has taught in regular and special education programs at all levels, holds a Master of Arts degree from Gallaudet University, and is fluent in American Sign Language. Milone served on the board of directors of the Association of Educational Publishers and was a member of the Literacy Assessment Committee and a past chair of the Technology and Literacy Committee of the International Association. He has contributed to both readingonline.org and Technology & Learning magazine on a regular basis. Over the past 30 years, he has been involved in a broad range of publishing projects, including the SRA reading series, assessments developed for Academic Therapy Publications, and software published by The Learning Company and LeapFrog. He has completed 34 marathons and 2 Ironman races. Page 13 of 13 R