Minnesota Reading Corps Mid-Year Evaluation 9/1/ /31/2008 Statewide Report

Transcription

1 Minnesota Reading Corps Mid-Year Evaluation 9/1/ /31/2008 Statewide Report Prepared By: Kerry Bollman, SSP, NCSP Academic Collaborative Planner, Reading Center Director Saint Croix River Education District Benjamin Silberglitt, PhD Senior Consultant, Assessment and Implementation Technology and Information Educational Services David Heistad, Ph.D Director, Research Evaluation and Assessment Minneapolis Public School District Page 1 of 20

2 Background of Minnesota Reading Corps Minnesota Reading Corps (MRC) is an AmeriCorps program that provides trained literacy tutors (Members) for children age three to grade three. Some MRC Members work with preschoolers and focus on integrating talking, reading, and writing into all activities. Other Members provide supplemental literacy skills tutoring for children in kindergarten to third grade. Still others recruit, train, and manage community volunteers to expand the capacity of the program. MRC Members and community volunteers are trained in specific research-based, leveled literacy instructional protocols, and are supported by expert coaches. Members use reliable, valid assessment tools to monitor student progress on a regular basis, and with help from their expert coaches, use data from assessments to inform tutoring strategies for each student. Use of specific research based instructional techniques and technically adequate assessment tools for decision making make the MRC program both highly unique and highly coveted across the literacy landscape. The vision of the Minnesota Reading Corps is to impact literacy in the state of MN through children, AmeriCorps members and communities as follows: All children in MN, ages 3 to grade 3, who qualify for MRC, will have access to MRC and will meet reading standards by third grade. AmeriCorps members, through the training, development and service opportunity provided by MRC, will pursue education related careers and/or continue to be ambassadors for children's literacy throughout their lives. Schools and community institutions/organizations, through their experiences with MRC, will understand and adopt the MRC methods for increasing literacy; those institutions will, in turn, promote MRC literacy methods to their colleagues. Evaluation Design The evaluation for the Minnesota Reading Corps program year includes six broad questions as listed below. Additions to the current year evaluation include more thorough analysis of performance for participating students relative to their performance level at the start of tutoring, ongoing follow-up on progress for students who previously participated in MRC services, analysis of performance on the 2009 Minnesota Comprehensive Assessment II for participating MRC students, and enhanced measurement of Pre-K site systems change. 1. What is the current impact of the MRC on the state of Minnesota in terms of numbers of students and programs receiving support? 2. Are the data collection tools used with children being implemented with fidelity? 3. Are the interventions used with children being implemented with fidelity? Page 2 of 20

3 4. Is the performance of MRC-participating students in terms of their literacy improvement consistent with expectations? 5. Are the organizations with which the MRC is working changing to adopt the practices of the MRC? 6. What is the impact of the MRC experience on the AmeriCorps Members? Assessment Data Collection The assessment data listed below will be collected during the school year. Minnesota Reading Corps Members collect all student assessment data. Internal and Master Coaches collect fidelity data for assessments and interventions. Internal Coaches, Site Supervisors, and Members complete survey data regarding perceptions of systems change and impact. Internal and Master Coaches collect Pre-K environmental observation data. Student Data for Pre-school Programs: Age 3 on or before Sept 1 st * Fall (Oct 6-21) Winter (Jan 19-30) Spring (April 20 May 1) IGDI Rhyming IGDI Rhyming IGDI Rhyming IGDI Picture Naming IGDI Picture Naming IGDI Picture Naming IGDI Alliteration IGDI Alliteration IGDI Alliteration Age 4 on or before Sept 1 st Age 5 on or before Sept 1 st but not enrolled in K IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency Program participants are strongly encouraged, but not required, to collect benchmark data on three-year-old students in classrooms served by Minnesota Reading Corps Members. Page 3 of 20

4 Student Data for K-3 Programs: Fall (Sept 8 19) Winter (Jan 12-23) Spring (May 4-15) Letter Naming Fluency Letter Naming Fluency Letter Sound Fluency Letter Sound Fluency Nonsense Word Fluency Nonsense Word Fluency Grade 1 Letter Naming Fluency Nonsense Word Fluency Oral Reading Fluency (3 Letter Sound Fluency Oral Reading Fluency (3 passages) Nonsense Word Fluency passages) Grade 2 Oral Reading Fluency (3 Oral Reading Fluency (3 Oral Reading Fluency (3 passages) passages) passages) Grade 3 Oral Reading Fluency (3 Oral Reading Fluency (3 Oral Reading Fluency (3 passages) passages) passages) Kindergarten Letter Naming Fluency Letter Sound Fluency Additional Data Observations of assessment fidelity three times per year Observations of intervention fidelity nine times per year in K-3 programs only Early Language and Literacy Classroom Observation two times per year in Pre-K programs only End of year MRC Member surveys End of year MRC site surveys Evaluation Report What is the current impact of the MRC on the state of Minnesota in terms of numbers of students and programs receiving support? In the tables below, the number of Minnesota Reading Corps Members, full or part time, serving during the school year, who collected data for students during one or more benchmark assessment windows and submitted the data for evaluation is recorded, along with the number of students receiving MRC services for whom data are recorded. Numbers of participating students are compiled according to the following criteria: Number of preschool students with data from 1 benchmark window Number of preschool students with data from 3 benchmark windows (end of year) Number of K-3 students benchmarked as a follow up from previous year s tutoring Number of K-3 students with at least 1 week of data on at least 1 measure Number of K-3 students with at least 5 weeks of data on at least 1 measure Number of K-3 students with at least 10 weeks of data on at least 1 measure Number of K-3 students with at least 20 weeks of data on at least 1 measure Page 4 of 20

5 Table 1: Pre-Kindergarten Participation Region Number of Members Number of Students with 1 or More Assessments in Window Number of Students with 3 Assessments in Window Central St. Cloud Metro St. Paul North Central Grand Rapids North East - Duluth North West - Moorhead Southern - Rochester Total 103 1,639 0 Table 2: Kindergarten-Grade 3 Participation Region Number of Members Benchmarked as Follow up 1 or More Assessments 5 or More Assessments on at least 1 Measure 10 or More Assessments on at least 1 Measure Central St. Cloud Metro St. Paul ,137 1, North Central Grand Rapids 20 or more Assessments on at least 1 Measure North East - Duluth North West - Moorhead Southern - Rochester Total 222 1,247 4,347 3,492 1,525 0 In order to more fully describe the population of children served by the Minnesota Reading Corps program, data regarding gender, ethnicity, special education entitlement and primary language spoken were collected by Reading Corps Members. Demographic data for students participating in the Minnesota Reading Corps program with at least one data point collected through the year are summarized below. It is noted that the ethnicity categories used for the evaluation reflect revisions to categories used in the Federal No Child Left Behind law since the time of the MRC evaluation report. Page 5 of 20

6 Table 3: Pre-Kindergarten Grade 3 Participant Demographic Data Gender Ethnicity Special Education Entitlement Primary Language Pre-K 52% Male 47% Female 1% Unknown 2% American Indian / Alaska 19% African American 5% Asian 0% Filipino 14% Hispanic / Latino 0% Pacific Islander 1% Unknown 6% Multiple / No Response 31% White 22% Not Reported 92% General Education 8% Students with IFSPs 66% English as Primary Language 29% ELL 5% Unknown K-3 48% Male 44% Female 8% Unknown 2% American Indian / Alaska 13% African American 3% Asian 0.03% Filipino 5% Hispanic / Latino 0.1% Pacific Islander 32% Unknown 1% Multiple / No Response 42% White 97% General Education 2% Title 1 1% Students with IEPs 94% English as Primary Language 6% ELL Are the data collection tools used with children being implemented with fidelity? Analysis of fidelity with which student assessment data are conducted is a critical initial aspect to the evaluation of the MRC program so that results from evaluation of these data may be reported with confidence. In order to accomplish this, a series of Accuracy of Implementation Rating Scales (AIRS) have been compiled from each Minnesota Reading Corps (MRC) site. MRC Internal Coaches and new Master Coaches were trained in August 2008 to administer and score assessment measures, and to conduct observations of Reading Corps members as they administer and score these measures. The AIRS are structured observational protocols that provide an opportunity for observers to certify that each aspect of a standardized administration for each assessment measure has been fully conducted. Master and/or Internal Coaches completed a minimum of 1 AIRS for each Reading Corps Member for each type of assessment the member conducted at least 3 times each year, around the benchmark data collection periods. The table below documents the number of AIRS assessments compiled and percent fidelity documented for each measure. In addition to reporting these data in aggregate format to document high fidelity of assessment procedures across the state, this observation system also provided Members with immediate feedback regarding the quality of their own assessment skills, and an opportunity to receive clarification or re-training as needed in a timely manner. Page 6 of 20

7 Table 4: Fidelity of Assessment Data Collection Procedures Measure Picture Naming Alliteration Rhyming Letter Naming Letter Sounds Nonsense Words Oral Reading Fluency Total AIRS Collected Fidelity Range 24% - 28% - 11% - 15% - 23% - 23% - 48% - Reported Median % Fidelity 97% 95% Reported Mean % Fidelity 91% 83% 86% 90% 86% 92% 94% Reported Standard Deviation Are the interventions used with children being implemented with fidelity? As with the assessment tools, analysis of the level of fidelity with which the student intervention protocols are followed is a critical initial aspect to the evaluation of the MRC program so that results of student growth analysis may be attributed to accurate implementation of intervention scripts. In order to accomplish this, a series of intervention integrity observations have been compiled from each MRC site. MRC Internal and Master Coaches were trained in August 2008 to evaluate implementation integrity for each of the MRC interventions. The integrity checklists provide an opportunity for observers to certify that each aspect of a standardized administration for each intervention has been fully conducted. Master and/ or Internal Coaches completed a minimum of 1 intervention integrity checklist for each MRC member during each monthly visit, for a possible total of 9 checklists per Member per year. The interested reader is referred to Ehrhardt, Barnett, Lentz, Stollar, & Reifin, (1996) for a description of how to use scripts to improve intervention integrity. The table below documents the number of integrity checklists compiled and percent fidelity documented for each intervention: Page 7 of 20

8 Table 5: Fidelity of Intervention Implementation Procedures Intervention Repeated Reading Duet Reading Newscaster Reading Repeated Reading w/ Question Generation Stop Go Pencil Tap Great Leaps Letter Sound Identification Word Blending Phoneme Blending Phoneme Segmenting Total Fidelity Checks Collected Fidelity Range Reported Median % Fidelity Reported Mean % Fidelity % - 21% - 55% % - 60% - 6% - 33% - 20% - 63% - 56% - 93% 94% 91% 88% 83% 94% 90% 91% 90% 96% 90% 78% 91% 92% 96% 88% Reported Standard Deviation Is the performance of MRC-participating students in terms of their literacy improvement consistent with expectations? The following sections document growth and achievements of children age 3 to grade 3 who participated in the MRC program during the school year. It is important to acknowledge that MRC participating students are also supported by a variety of resources, most notably the instruction and guidance provided by their schools and families. This evaluation is not intended to address or control for the variables related to these resources, nor to suggest that student progress or lack thereof must be attributed solely to the service being provided through the efforts of the MRC program. This design s purpose is to focus on the desired literacy outcomes for all children. Pre-Kindergarten Student Performance The five measurement tools utilized for the Pre-K Reading Corps program are listed below. For a description of the technical characteristics of these assessments, the interested reader is referred to the Minnesota Reading Corps program evaluation. For each assessment tool, a target score was identified as the goal for the end of the year. These target scores were based on the target scores used in Minneapolis Public Schools for incoming kindergarten students, and upon 50 th %ile scores for incoming kindergarten students within school districts served by Saint Croix River Education District. Prior to use of these targets for the current project, these targets were reviewed by the researchers at the University of Minnesota who originally created the Individual Growth and Development Indicators. The measures and target scores for this project are listed below: Page 8 of 20

9 Measure Spring Target Score Rhyming 12 Picture Naming 26 Alliteration 8 Letter Sound Fluency 8 Letter Naming Fluency 14 Pre-Kindergarten student performance on fall IGDI measures is listed in the table below for all students with a birth-date reported, and data collected during the fall assessment window. Score ranges are also reported. Students score NA when they do not complete the sample items sufficiently to warrant participation in the assessment. A score of 0 indicates adequate performance on the sample items, but no accurate responses during the assessment. Table 6: Pre-Kindergarten Participant Performance on IGDIs: Fall Benchmark Measure Picture Naming Alliteration Rhyming Letter Naming Fluency Letter Sound Fluency Output Three Year Olds Four Year Olds Five Year Olds Fall Number Students Tested 349 1, Range of Scores NA-33 NA Fall (Number) Percent Students Above Spring Target (17) 4.9% (182) 17.08% (11) 31.4% Fall Number Students Tested 349 1, Range of Scores NA-11 NA-20 NA-9 Fall (Number) Percent Students Above Spring Target (2) 0.6% (50) 4.69% (2) 5.7% Fall Number Students Tested 349 1, Range of Scores NA-16 NA-20 NA-17 Fall (Number) Percent Students Above Spring Target (6) 1.7% (102) 9.58% (6) 17.1% Fall Number Students Tested 349 1, Range of Scores NA-38 NA-56 NA-42 Fall (Number) Percent Students Above Spring Target (14) 4.0% (231) 21.69% (13) 37.1% Fall Number Students Tested 349 1, Range of Scores NA-12 NA-41 NA-20 Fall (Number) Percent Students Above Spring Target (3) 0.9% (92) 8.64% (3) 8.6% The figure below shows the normative performance of all 4-year-old students participating in the IGDI measures during the fall benchmark window of the school year. As a clarification, it is noted that students who did not successfully complete the sample items on each assessment measure in order to continue on to the actual assessment were given a score of NA which is recorded in this figure as a -6. The NA score is distinguished from a score of 0, which reflects a performance in which the student s appropriate responses to the sample items warranted continuation with the assessment in accordance with standardized procedure, but responses during the timed assessment yielded no correct answers. Page 9 of 20

10 Figure 1: Normative Performance of 4-Year-Olds on IGDI Measures: Fall * Green Diamond = 90 th %ile score; Mid-Blue X = 75 th %ile score; Navy Triangle = 50 th %ile score; Light Blue Circle= 25 th %ile score; Red Square = 10 th %ile score. A cross-cohort analysis of performance by 4-year-old students has been compiled across the six years of the Minnesota Reading Corps program. The following figure shows the percent of 4-yearold MRC participants meeting the assessments spring target scores at fall, winter, and spring assessment times across years. It is noted that the data was analyzed by an outside agency, and only includes students enrolled in Headstart MRC classrooms. Across successive school years, fairly stable fall performance is noted, along with an overall increase in percent of students meeting spring target scores by the spring benchmark window. Data from the current year again show performance roughly equivalent with or slightly above that of previous years on fall measures. Page 10 of 20

11 Figure 2: Cross-Cohort Percent Above Target on Early Literacy Measures Pre-Kindergarten Matched Sample Analysis In order to address the evaluation question regarding growth across years for students participating in the Minnesota Reading Corps program, analysis has been completed for a subset of MRC participating students who were enrolled in PICA Head Start programming during the school year, and continued on to elementary school within Minneapolis Public School (MPS) system as kindergarteners in fall of For comparison purposes, an additional group of students was identified who did not receive intervention from a Reading Corps member as preschoolers, but also enrolled in MPS in fall of Minneapolis Beginning of Kindergarten Assessment Beginning of Kindergarten literacy was assessed using the Minneapolis Beginning of Kindergarten Assessment (BKA). This assessment is individually administrated by retired school teachers who are trained to be reliable data collectors. The assessment includes 5 literacy domains: phonological awareness (initial letter sounds and rhyming), alphabetic principles (letter names and letter sounds), vocabulary (picture naming), concepts of print (front of the book, right to left sweep, etc.), and oral comprehension. Page 11 of 20

12 The BKA has scores with a high level of reliability (Betts J., Heistad D., Pickart M., and Anderson, A., 2005). The overall total test score, called the Early Literacy Composite has internal consistency reliability at or above Similarly high internal consistency reliability estimates have been found among the three norm groups (below 5 ½, between 5 ½ and 6, and above 6 years of age as measured by age at time of BKA). The test-retest reliability was found to be The scores on the BKA have been found to have strong validity evidence (content, criterion and construct). The predictive validity of the BKA to 1 st grade oral reading (0.80), 1 st grade comprehension (0.66) and to 2 nd grade reading (0.66) as measured by the Northwest Achievement Levels Tests (NALT) is also moderately strong. The BKA Early Literacy Composite is also highly correlated with the End of Kindergarten Assessment (EKA) Early Literacy Composite (0.74). Matched Sample Methodology In order to locate matched samples for comparison, files of all kindergarten students were sorted hierarchically on these variables in the MPS data system in fall 2008 using the following sort order: Home Language Special Education Status Special Education Disability Category Free or reduced price Lunch Racial Ethnic Category English Language Learner Status Gender Residential Zip Code Student Birth date (note: matched students needed to be the same age within 6 months for a match. Students in the comparison groups were matched on at least 8 of 9 variables. The best match was chosen by computer as the student directly above or directly below the Reading Corps student with the most matches. Ties (e.g. if the student above and below in the file each had a match on 9 variables) were broken based on birth date closest to the Reading Corps student. Matching between Reading Corps students and non-reading Corps students were as follows: 48 pairs (73%) had 9 out of 9 variables matched 18 pairs (27%) had 8 out of 9 variables matched Demographic data for students in this analysis are listed below: 52% Male 96% Free/reduced lunch 0% Special Education students 56% ELL students 51% Lives with both parents Page 12 of 20

13 Table 7: Racial/Ethnic Background of Study Participants Racial/Ethnic Background Reading Corps Non- Reading Corps Matched Sample American Indian 3 (4.5%) 3 (4.5%) African American 37 (56%) 37 (56%) Asian 1 (1.5%) 1 (1.5%) Hispanic 23 (35%) 23 (35%) White 2 (3%) 2 (3%) Total Table 8: Home Language of Study Participants Home Language Reading Corps Non- Reading Corps Matched Sample English 26 (39%) 27 (41%) Hmong 1 (1.5%) 1 (1.5%) Spanish 25 (38%) 25 (38%) Somali 14 (21%) 13 (19%) Total Results of Matched Sample Analysis The Dependent t-test comparing mean total literacy scores on the Minneapolis Public Schools Beginning Kindergarten Assessment (BKA) for students in Reading Corps relative to students not in Reading Corps was statistically significant t (65) = 2.96; p=.004. The effect size (in standard deviation units) for Reading Corps vs. non-reading Corps was.53 which is a moderate effect. In conclusion, the difference between Reading Corps students and non-reading Corps students was a large difference which was statistically significant at a 99.5% confidence level. The Minneapolis Public School system has identified a score of 80 as the target for the literacy portion of the Beginning Kindergarten Assessment, representing a performance that predicts future success in reading development and adequate performance on state assessments of reading. As seen in Figure 3, the average total literacy score of for Reading Corps participating students within the PICA Headstart program who continued on to Kindergarten within the Minneapolis Public Schools system well exceeds this target score. Page 13 of 20

14 Figure 3: Performance on Minneapolis BKA for Reading Corps and Non-Reading Corps Students Kindergarten-Grade 3 Student Performance The four assessment tools utilized for the K-3 Reading Corps program are listed below. For a description of the technical characteristics of these assessments, the interested reader is referred to the Minnesota Reading Corps program evaluation. For each assessment tool, a target score was identified as the goal for the beginning, middle, and end of the year. These target scores were based on research conducted at the St. Croix River Education District, which documented the predictive and concurrent validity of these measures with the Minnesota Comprehensive Assessment II. As a result of the strong correlations between performance on the selected fluency measures and on the MCAII, a series of cut scores has been identified. The table below specifies assessments given at each grade level and the cut scores for each assessment during several points throughout the school year. These cut scores, or target scores, define levels of performance on the fluency measures that strongly predict future success on the grade 3 Minnesota Comprehensive Assessment II. For example, a student who reads a Grade 1 passage in the winter of first grade at a rate of 20 words read correctly per minute has an 75% chance of earning a score in the meets standards or exceeds standards ranges on the 3 rd grade MCAII two years later. Page 14 of 20

15 Grade Measure Fall Target Winter Target Spring Target K Letter Sound Fluency Nonsense Word Fluency N/A 1 Oral Reading Fluency N/A Oral Reading Fluency Oral Reading Fluency The target scores as listed above for each assessment used as a part of ongoing student literacy measurement in Reading Corps grow across years from age 3 to grade 3, define a pathway to success. Through consideration of the inherent growth that would occur for a child who met each of the targets, an expectation of growth rate at each grade level can be defined. For example, the fall grade 2 target score is 43 on oral reading fluency. The spring grade 2 target score on this measure is 90. To grow from 43 to 90 in one academic year, a student would need to gain 1.31 words correct per minute per week on the oral reading fluency assessment. Thus, 1.31 words growth per week becomes the expectation for 2 nd grade growth rates. Because our targets are connected to the statewide assessment rather than normative performance of other students in local districts, we have a consistent and meaningful comparison across the state. Students participating in the Minnesota Reading Corps program are monitored frequently. The primary purpose of this progress monitoring is to enable those providing support to the student the ability to evaluate the effectiveness of current reading instruction, and to make data-based decisions regarding changes in instruction. For the purposes of outcomes evaluation, the progress monitoring data also provides a means for comparing the rate of growth of participating Minnesota Reading Corps students to the expected grade level growth rate. Students are selected for participation in the Reading Corps program because they are identified as having below grade level skills in reading. These students who achieve higher growth rates than those indicated by our targets are catching up to grade level expectation by making more than one year s growth in one year s time. For the state-wide Minnesota Reading Corps project, one measure of our success is the extent to which our participating students are achieving this primary goal. In the table below, a comparison between weekly growth rate expectations and the average weekly growth rates on program assessment measures of children participating in K-3 Reading Corps programs who have at least 3 data points collected per measure is listed. The current analysis includes all data collected between 9/1/2008 and 12/31/2008. Notably in all grade levels, the average growth rate of Minnesota Reading Corps participants exceeded the target growth rate (note that in Grade 1, while mean growth of participants in the second half of the year, on Oral Reading Fluency, was slightly below target growth, the mean growth in the first half of the year, on Nonsense Word Fluency, exceeded target growth). In other words, the average growth rate for Minnesota Reading Corps participants exceeded a rate of one year s growth in one year s time. This is significant as it demonstrates that participating students are actually catching up to grade level expectations. Page 15 of 20

16 Table 9: Kindergarten-Grade 3 Participant Growth Grade K Grade 1 Grade 2 Grade 3 Letter Sound Fluency Nonsense Word Fluency (Fall to Winter) Oral Reading Fluency (Winter to Spring)** Oral Reading Fluency Oral Reading Fluency Reading Corps Mean Growth Rate N/A Target Growth Rate Number of Students 466 1,196 N/A 1,176 1,167 * Only students with 3 or more data points on the given measure were included in growth rate calculations ** Students in this group may have also participated in Grade 1 (NWF) The table below examines K-3 students who have at least 3 data points collected per measure, represented by the Total # of Students. The table presents the percentage of these students whose individual growth rates exceeded the target growth rates for that grade level and measure. This calculation represents the portion of the population participating in a Reading Corps intervention that had growth rates in excess of one year of growth in one years time. Percentages are given by region, and overall. Table 10: Kindergarten Grade 3 Percentage of Students Above Growth Targets by Region Grade K Grade 1 Grade 1 Grade 2 Grade 3 (LSF) (NWF) (R-CBM)** (R-CBM) (R-CBM) TOTAL* Central % Above Target 88.50% 91.01% N/A 80.95% 98.32% 89.97% St. Cloud Total # of Students N/A Metro % Above Target 89.58% 91.02% N/A 75.16% 94.43% 86.89% St. Paul Total # of Students N/A ,191 North Central % Above Target % N/A 91.18% 96.49% 95.50% Grand Rapids Total # of Students 0 20 N/A North East % Above Target 87.50% 94.81% N/A 75.42% 96.00% 89.28% - Duluth Total # of Students N/A North West % Above Target 96.30% 95.41% N/A 73.96% 97.33% 89.25% - Moorhead Total # of Students N/A Southern % Above Target 88.89% 91.57% N/A 86.43% 95.54% 91.12% - Rochester Total # of Students N/A % Above Target 89.48% 92.14% N/A 77.72% 95.63% 88.61% TOTAL Total # of Students 466 1,196 N/A 1,176 1,167 4,005 * TOTAL represents the total number of slopes analyzed, not the total number of students, as students in Grade 1 may have participated in two categories ** Students in this group may have also participated in Grade 1 (NWF) In the table below, the percent of K-3 MRC participating students with at least 3 progress monitor data points who have graduated from tutoring (exit) so far this year is reported. A 3 data point minimum was selected in order to remove from the analysis students who were erroneously included in the data set for a short time, but who did not have at least a reasonable minimum length of service prior to discontinuation. Page 16 of 20

17 Table 11: Number and Percentage of students with 3 weeks of progress monitoring data who exit Region Grade Number Exited Percent Exited Central - Saint Cloud K % Central - Saint Cloud % Central - Saint Cloud % Central - Saint Cloud % Metro St. Paul K % Metro St. Paul % Metro St. Paul % Metro St. Paul % North Central - Grand Rapids % North Central - Grand Rapids % North Central - Grand Rapids % North East - Duluth K % North East - Duluth % North East - Duluth % North East - Duluth % North West - Moorhead K % North West - Moorhead % North West - Moorhead % North West - Moorhead % Southern - Rochester K % Southern - Rochester % Southern - Rochester % Southern - Rochester % TOTAL K % TOTAL % TOTAL % TOTAL % TOTAL TOTAL % In order to evaluate longer-term outcomes for students previously enrolled in the Minnesota Reading Corps program, additional analysis has been conducted on current performance of students who participated in MRC during the school year. These participants were divided into two groups. The first group included all MRC participants who graduated from tutoring prior to the end of the school year based on program exit criteria, or had a spring benchmark score that met or exceeded the spring target score for that grade level (Success in Prior Year = 1). The second group included all MRC participants who did not meet either of those criteria (Success in Prior Year = 0). In the table below, average scores for the fall benchmark period during the school year is reported as well as the percent of these students whose current year scores meet or exceed current enrollment grade targets. Page 17 of 20

18 Table 12: fall mean scores and percent above target for participants Region Grade Measure Success in Prior Year? Number Tested Average Score (Fall Benchmark) Percent Above Target (Fall Benchmark) Central - Saint Cloud 1 NWF % Central - Saint Cloud 1 NWF % Central - Saint Cloud 2 R-CBM % Central - Saint Cloud 2 R-CBM % Central - Saint Cloud 3 R-CBM % Central - Saint Cloud 3 R-CBM % Metro St. Paul 1 NWF % Metro St. Paul 1 NWF % Metro St. Paul 2 R-CBM % Metro St. Paul 2 R-CBM % Metro St. Paul 3 R-CBM % Metro St. Paul 3 R-CBM % North Central - Grand Rapids 1 NWF % North Central - Grand Rapids 1 NWF % North Central - Grand Rapids 2 R-CBM % North Central - Grand Rapids 2 R-CBM % North Central - Grand Rapids 3 R-CBM % North Central - Grand Rapids 3 R-CBM % North East - Duluth 1 NWF % North East - Duluth 1 NWF % North East - Duluth 2 R-CBM % North East - Duluth 2 R-CBM % North East - Duluth 3 R-CBM % North East - Duluth 3 R-CBM % North West - Moorhead 1 NWF % North West - Moorhead 1 NWF % North West - Moorhead 2 R-CBM % North West - Moorhead 2 R-CBM % North West - Moorhead 3 R-CBM % North West - Moorhead 3 R-CBM % Southern - Rochester 1 NWF % Southern - Rochester 1 NWF % Southern - Rochester 2 R-CBM % Southern - Rochester 2 R-CBM % Southern - Rochester 3 R-CBM % Southern - Rochester 3 R-CBM % Page 18 of 20

19 The table below displays data from above in aggregate form, with the data collapsed by grade level to show overall percent of students who met or exceeded current fall targets by level of success during the school year, by region, and across the state. In total 797 participating MRC students in this analysis either met exit criteria or scored at or above spring target during the school year % of these students continued to score at or above target on current fall measures during the school year. By contrast, 441 participating MRC students did not meet exit criteria or spring target scores during the school year. Only 8.84% of these students scored at or above target on current fall measures during the school year: Table 13: Percent of students who met or exceeded fall 2008 targets by success in prior years Region Success in Prior Year? Number Tested Percent Above Target (Fall Benchmark) Central - Saint Cloud % Central - Saint Cloud % Metro St. Paul % Metro St. Paul % North Central - Grand Rapids % North Central - Grand Rapids % North East - Duluth % North East - Duluth % North West - Moorhead % North West - Moorhead % Southern - Rochester % Southern - Rochester % TOTAL % TOTAL % Are the organizations with which the MRC is working changing to adopt the practices of the MRC? At the pre-k level, Internal and Master coaches utilize the Early Literacy and Language Classroom Observation tool (ELLCO) to evaluate changes in the quality of the classroom environment in MRC Pre-K classrooms across the school year. This tool is offered through Brookes Publishing, and a technical report is available regarding the psychometric properties of this instrument on the company website at Observations focus on 5 key literacy elements: classroom structure, curriculum, the language environment, books and book reading, and print and early writing supports. In the fall, coaches review observation data with teachers and MRC Members serving in the participating classrooms, and set goals for improving the literacy and language environment. Coaching during the school year is related to making progress toward these identified classroom goals. Spring observations are completed to identify and measure changes across the year in classroom environment. The tables below list numbers of participating classrooms as well as fall scores on the ELLCO tool. Page 19 of 20

20 Table 14: Participating Classrooms in ELLCO tool Assessment Region Number of Classrooms with ELLCO Score Central St. Cloud 4 Metro St. Paul 34 North Central Grand Rapids 12 North East Duluth 2 North West Moorhead 13 Southern Rochester 19 Total 84 Table 14: Median Scores on ELLCO tool Scale Possible Total Score Median Fall Score Classroom Structure Subtotal Curriculum Subtotal Language Environment Subtotal Books and Book Reading Subtotal Print and Early Writing Subtotal Median Spring Score Internal Coaches and Site Supervisors will participate in electronic surveys toward the end of the school year in order to share their perceptions about change in their local systems resulting from participation in the MRC program. These data will be compiled in the final report. What is the impact of the MRC experience on the AmeriCorps Members? Minnesota Reading Corps members will participate in an electronic survey toward the end of the school year in order to share their perceptions of impact of the MRC program. These data will be compiled for the final report. Page 20 of 20