Using MCA-II Proficiency Level, Minnesota Growth, and Map Growth for Setting Individual Student s Growth Chi-Keung (Alex) Chan Minneapolis Public Schools & Christopher Desjardins University of Minnesota Twin Cities Presentation at the 2010 Minnesota Assessment Conference
Agenda Issues on Setting Goals Illustration of DPME Analyses MN Growth Model Mapping MN Growth onto MAP Scale Mapping Individual Trajectory onto MAP Triangulation & Setting Individuals Goals Strengths/Limitations & Future Research Student Growth Percentiles (SGPs)
Objectives of this Session Investigate t and discuss how to set individual student s learning goals by triangulating the MCA-II proficiency level, MN Growth, and MAP growth data; Discuss the strengths and limitations of this approach; Briefly introduce student growth percentiles (SGPs).
Issues on Setting Goals We always set smart or strategic goals for the districts, schools, subgroups, or grades arbitrarily without considering individual growth (e.g. 10% increase on proficiency for grade 5 reading); We always set arbitrary and short-term smart or strategic goals (e.g. 50 percent of students making one-year goal) without ensuring long-term success; We always set arbitrary smart or strategic goals without integrating various types of assessment data (e.g. should we use MAP or MCA-II growth to set the goals)?
Reflection 1. Are the goals we set really SMART? 2. Are the goals we set really STRATEGIC? 3. Are the goals we set really target on our students and ensure their long-term academic success? PAIR REFLECTION SHARE
Methodology Data: MPS students with 2007-08 and 2008-09 valid MCA-II scores, and 2008-09 growth data. Methods: DPME analyses MN growth: low, medium, high growth Mapping MN growth onto MAP Typical vs. catch-up (keep-up) growth
DPME Analyses A simple cross-tabulation analysis between the MCA-II proficiency level data by grade level over two years (2007-0808 and 2008-2009); 2009); Identify students who stay at the same level, move up to a higher h level, l or move down to a lower level; Identity students who need to catch up and who need to keep up.
DPME Analyses for 3 th to 4 th Grade MCA-II Reading 09 D P M E 08 D 14 5 0 0 73.7% 26.3% 0% 0% P 4 1 6 0 36.4% 9.1% 54.5% 0% M 0 3 5 3 0% 27.3% 45.5% 27.3% E 0 1 6 12 0% 5.3% 31.6% 63.2%
Diagonal Students Stay at Same Level 09 D P M E 08 D 14 73.7% P 1 9.1% M 5 45.5% E 12 63.2%
Areas above the Diagonal Line (Purple) Students Move Up to a Higher Level 09 D P M E 08 D 5 0 0 26.3% 0% 0% P 6 0 54.5% 0% M 3 27.3% E
08 D Areas below the Diagonal Line (Orange) Students Move Down to a Lower Level 09 D P M E P 4 36.4% M 0 3 0% 27.3% E 0 1 6 0% 5.3% 31.6%
Areas on the Left (Level D and P) Students Need to Catch Up 09 D P M E 08 D 14 5 73.7% 26.3% P 4 1 36.4% 9.1% M 0 3 0% 27.3% E 0 1 0% 5.3%
Areas on the Left (Level M and E) Students Need to Keep Up 09 D P M E 08 D 0 0 0% 0% P 6 0 54.5% 0.0% M 5 3 45.5% 27.3% E 6 12 31.6% 63.2%
Minnesota Growth Model For each grade and subject, students who took the MCA-II in 2008 and 2009 were grouped based on their scale score. Each group represented a very narrow range of scale scores. For each of these groups, a combined distribution of scale scores earned by the same students the following year (2008 or 2009) was generated, and summary statistics (mean and standard deviation) were calculated. Next, we calculated growth for 2008-09. Students with test scores in 2008 and 2009 whose 2009 scale score fell within a half standard deviation of the mean scale score for their group were placed in the Medium Growth group. Students whose scale scores fell below that range were placed in the Low Growth group. Students whose 2009 scores were more than one-half standard deviation above the mean for their group were placed in the "High Growth" group. If the second year scores are normally distributed within each first year score group, we would expect about 40% of students to fall into the Medium Growth range, and 30% to fall into each of the Low and High Growth ranges. DAC Update, MDE)
38.30% 30% Medium Growth
Minnesota Growth Target Ranges Source: http://education.state.mn.us/mde/data/data_downloads/account us/mde/data/data Downloads/Account ability_data/growth/index.htm
Minnesota Growth Target Ranges Target 1: Cut-scores between Medium and High growth for a particular scale score range at the prior grade. Target 2: Cut-scores between Medium and Low growth for a particular scale score range at the prior grade. Source: http://education.state.mn.us/mde/data/data_downloads/accountability_data/growth/in dex.htm
DPME Analyses for 3 th to 4 th Grade MCA-II Reading by MN Growth (Low Growth) 09 D P M E 08 D 4 0 0 0 100.0% 0% 0% 0% P 4 0 0 0 100.0% 0% 0% 0% M 0 1 0 0 0% 100.0% 0% 0% E 0 1 3 2 0% 16.7% 50.0% 33.3%
DPME Analyses for 3 th to 4 th Grade MCA-II Reading by MN Growth (Medium Growth) 09 D P M E 08 D 10 1 0 0 90.9% 9.1% 0% 0% P 0 1 0 0 0% 100.0% 0% 0% M 0 2 1 0 0% 66.7% 33.3% 0% E 0 0 3 2 0% 0% 60.0% 40.0%
DPME Analyses for 3 th to 4 th Grade MCA-II Reading by MN Growth (High Growth) 09 D P M E 08 D 0 4 0 0 0% 100.0% 0% 0% P 0 0 6 0 0% 0% 100.0% 0% M 0 0 4 3 0% 0% 57.1% 42.9% E 0 0 0 8 0% 0% 0% 100.0%
DPME Analyses & MN Growth Students with low growth are more likely to move down to a lower level; Students with medium growth are more likely to stay at the same level (especially at levels D and P) or move down to a lower level; Students with high growth are more likely to stay at the same level or move up to a higher level (especially at levels D and P);
Proficiency & MN Growth (Start at 320) 900 850 800 750 700 MCA A II Scale Score es 650 600 550 500 Level P Level M High Medium (320) Low Medium (320) 450 400 350 300 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Proficiency & MN Growth (Start at 350) 900 850 800 750 700 I Scale Scores MCA I 650 600 550 500 Level P Level M High Medium (350) Low Medium (350) 450 400 350 300 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Proficiency & MN Growth (Start at 380) 900 850 800 750 700 I Scale Scores MCA I 650 600 550 500 Level P Level M High Medium (380) Low Medium (380) 450 400 350 300 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Types of Growth with MN Growth Typical Growth: Between the range of the medium growth. Catch-up Growth: Growth that ensures students at Level D and P to be on-track to proficiency by 8 th grade. Keep-up Growth: Growth that ensures students at Level M and E to maintain at proficiency/same level by 8 th grade.
Examples Student A: Student A scored at Level D (328) at 3 rd grade and had low growth in 2008-0909 and continued scored at Level D (416) at 4 th grade. Student t B: Student B scored at Level M (355) at 3 rd grade and had medium growth in 2008-09 and maintained at Level M (455) at 4 th grade. What types of growth they need to be proficient by 8 th grade?
Student A (at Level D) 900 850 800 750 700 MCA I II Scale Scores 650 600 550 500 Level M High Medium Low Medium 450 400 350 300 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Catch-up Growth for Student A by 8 th Grade MCA I II Scale Scores 900 850 800 750 700 650 600 550 500 Set catch-up growth targets between the proficiency i line and the high medium growth line Level M High Medium Low Medium Catch up 450 400 350 300 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Student B (at Level M with Medium Growth) MCA II Scale Score es 900 Set keep-up growth targets between the 800 700 600 500 proficiency i line and the high medium growth line Level M High Medium Low Medium 400 300 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Mapping between MCA-II & MAP scores Using equipercentile method to map each spring MCA-II scores to spring MAP scores; Mapping the proficiency trend & growth trend mentioned above onto MAP scale; Mapping the projected individual s growth trajectory onto MAP scale and set the individual goals.
Mapping MCA-II Growth onto MAP Scale Proficiency Level 350 195 Low Medium (320) 320 168 High Medium (320) 320 168 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 450 204 420 168 430 181 550 207 517 168 534 190 650 214 618 170 641 203 750 219 717 183 745 213 850 221 816 175 848 218 Low Medium (350) 350 444 539 637 733 828 195 198 195 199 201 192 High Medium (350) 350 452 555 659 762 864 195 205 212 223 229 232 Low Medium (380) 380 214 467 217 560 218 656 220 753 221 849 220 High Medium (380) 380 214 478 225 579 233 677 237 777 240 876 243
Mapping MN Growth onto MAP Strand 250 240 230 220 IT Scores R 210 200 190 180 Level M High Medium (320) Low Medium (320) High Medium (350) Low Medium (350) High Medium (380) Low Medium (380) 170 160 150 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Mapping Individuals Trajectories onto MAP Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Proficiency Level 350 450 550 650 750 850 195 204 207 214 219 221 Student A s Low Medium Trajectory 328 176 416 162 516 167 617 169 716 180 815 173 Student A s High Medium Trajectory Student B s Low Medium Trajectory Student B s High Medium Trajectory 328 176 416 162 527 180 635 196 741 210 845 216 355 455 550 646 743 839 199 208 207 209 212 209 355 455 557 661 765 867 199 208 214 224 232 236
Mapping Individuals Trajectories onto MAP 250 240 230 220 RIT Scores 210 200 190 180 Level M High Medium (MN Growth) Low Medium (MN Growth) High Medium (B) Low Medium (B) 170 160 150 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Catch-up Growth for Person A on MAP scale 250 240 230 220 RIT Scores 210 200 190 Level M High Medium (A) Low Medium (A) Catch up (A) 180 170 160 150 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Catch-up Growth vs. NWEA Typical Growth 250 240 230 220 T Scores RI 210 200 190 180 Level M High Medium (A) Low Medium (A) Catch up (A) NWEA (start at 4th) NWEA (start at 3rd) 170 160 150 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Triangulation (3 rd to 4 th grade reading) Initial MN 3 rd Grade 4 th Grade RIT Types of Growth to be Proficiency growth MCA-II MCA-II Range at proficient at 8 th grade Level (3-4) Scale Score SS Range 4 th Grade Level D Low 320 < 420 < 168 Catch-up Level D Medium 320 420-430 168-181 Catch-up Level D High 320 > 430 > 181 Catch-up/High Medium Level P Low 340 < 433 < 185 Catch-up Level P Medium 340 433-442 185-196 Catch-up/High Medium Level P High 340 > 442 >196 High Medium Level M Low 350 < 444 < 198 Catch up/high Medium Level M Medium 350 444-452 198-205 Keep up/high Medium Level M High 350 > 452 > 205 Keep-up/High Medium Level E Low 380 < 467 < 217 Catch-up/ Keep up/high Medium Level E Medium 380 467-478 217-225 Keep up/high Medium Level E High 380 > 478 > 225 Keep up/high Medium
Setting Projected Individuals Goals Level Growth Gr 3 Gr 4 Gr5 Gr 6 Gr 7 Gr 8 (3-4) Student t A s D Low 328 38 416 527 57 638 749 79 850 Catch-Up Trajectory 176 162 180 200 218 221 Goals Student t B s M Mdi Medium 355 455 557 661 765 867 High Medium 199 208 214 224 232 236 Trajectory Goals
Strengths/Limitations & Future Research Strengths Limitations Future Research Emphasizes the importance and priority Are these individual goals aligned with the aggregated Investigate the alignment of these individual goals with of setting individual student s goals level goals or strategic goals? aggregated level goals or strategic goals Sets long-term goals to Are there any factors Investigate the impact of ensure academic success of every students positively/negatively related to the long-term academic success? factors that are related to long-term academic success Triangulates different types of assessment data for setting individual student s goals Sets simple, realistic and measureable goals How about K-2 and 9-12? Are MN growth trend and MAP growth trend equivalent? Map other assessment data onto MAP scale; Link MAP and MN growth scale Time-consuming & Develop a statistical moderate technical; program; Consider Partial arbitrary alternative approach - student growth percentiles (SGPs)
Student Growth Percentiles Betebenner, D. W. (2008). A primer on student growth percentiles. Retrieved from http://www.cde.state.co.us/cdedocs/research/pdf/aprimeronstu dentgrowthpercentiles.pdfpdf Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practices, 23(4), 42-51. National Center for the Improvement of Educational Assessment http://www.nciea.org/ Colorado Department of Education http://www.cde.state.co.us/research/growthmodel.htm
Student Growth Percentiles Normative quantification of individual id student growth (e.g. < 35%ile, low growth; 35%ile-64%ile, typical growth; 65%ile +, high growth); Can be used as criterion referenced in order to establish the quantity of adequate growth; Vertical scale is not required.
Conditional SGPs at 6 th Grade 230 220 210 RIT Scores 200 190 Level M Student C 39 %ile 68 %ile 99 %ile 180 170 Grade 3 Grade 4 Grade 5 Grade 6
Criterion-referenced SGPs 230 220 80%ile growth from 7 th to 8 th RIT Scores 210 200 80%ile growth from 6 th to 7 th Level M Student C 190 180 170 Student C needs to make 80%ile in two consecutive years (conditional on the prior achievement and prior growth rates) in order to catch-up. Likelihood for achieving two consecutive 80%ile growth is about 4%, are these catch-up goals realistic? Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Questions & Comments Contacts: Chi-Keung (Alex) Chan Alex.Chan@mpls.k12.mn.us 612-668-0578