Minnesota Reading Corps State-Wide Evaluation

Transcription

1 Minnesota Reading Corps State-Wide Evaluation KERRY BOLLMAN, SSP, NCSP Instructional Services Coordinator, Reading Center Director Saint Croix River Education District BENJAMIN SILBERGLITT, PhD Director, Software Applications Technology and Information Educational Services DAVID PARKER, PhD School Psychologist, Master Coach South Washington County Schools, Minnesota Reading Corps MINNESOTA READING CORPS STATE-WIDE EVALUATION 1

2 Table of Contents BACKGROUND OF THE MINNESOTA READING CORPS 5 EVALUATION DESIGN 6 ASSESSMENT DATA COLLECTION 7 EVALUATION REPORT 9 1. CURRENT IMPACT 9 GROWTH IN DATA REPORTED 12 DEMOGRAPHICS FIDELITY OF ASSESSMENT DATA COLLECTION FIDELITY OF INTERVENTION IMPLEMENTATION 28 GROWTH IN FIDELITY REPORTING STUDENT OUTCOMES 40 PRE-KINDERGARTEN STUDENT PERFORMANCE 40 SUPPLEMENTAL INTERVENTION IN MRC PRE-K PROGRAM 48 CLASSROOM OUTCOMES FOR PRE-K SITES 51 KINDERGARTEN-GRADE 3 STUDENT PERFORMANCE 52 PERFORMANCE ON THE STATEWIDE READING ASSESSMENT SYSTEMS CHANGE IMPACT ON AMERICORPS MEMBERS ACTION RESEARCH: RESULTS OF PILOT STUDIES Appendix 85 MINNESOTA READING CORPS STATE-WIDE EVALUATION 2

3 Tables and Figures TABLE 1: OVERALL MEMBER COUNTS AND SERVICE HOURS, BY REGION AND TOTAL... 9 TABLE 2: PRE-KINDERGARTEN PARTICIPATION TABLE 3: NUMBER OF K-3 STUDENTS BENCHMARKED AS FOLLOW-UP TO PRIOR YEARS' TUTORING.. 11 TABLE 4: NUMBER OF STUDENTS WITH AT LEAST 1 DATA POINT, ANY MEASURE TABLE 5: NUMBER OF STUDENTS WITH AT LEAST 5 WEEKS OF DATA (R-CBM, NWF, OR LSF) TABLE 6: NUMBER OF STUDENTS WITH AT LEAST 10 WEEKS OF DATA (R-CBM, NWF, OR LSF) TABLE 7: NUMBER OF STUDENTS WITH AT LEAST 20 WEEKS OF DATA (R-CBM, NWF, OR LSF) FIGURE 1: GROWTH IN STUDENT DATA REPORTED TABLE 8: MINNESOTA READING CORPS PARTICIPATION: ALL STUDENTS TABLE 9: PARTICIPATION LEVELS IN MRC TUTORING TABLE 10: PARTICIPATION LEVELS IN K-3 MRC TUTORING BY REGION TABLE 11: PRE-KINDERGARTEN - GRADE 3 PARTICIPANT DEMOGRAPHIC DATA TABLE 12: FIDELITY OF ASSESSMENT DATA COLLECTION PROCEDURES TABLE 13: FIDELITY OF INTERVENTION IMPLEMENTATION PROCEDURES FIGURE 2: GROWTH IN FIDELITY REPORTING TABLE 14: KINDERGARTEN PARTICIPANT PERFORMANCE ON IGDIS: FALL BENCHMARK TABLE 15: PRE-KINDERGARTEN PARTICIPANT PERFORMANCE ON IGDIS: WINTER BENCHMARK TABLE 16: PRE-KINDERGARTEN PARTICIPANT PERFORMANCE ON IGDIS: SPRING BENCHMARK FIGURE 3: PERCENTAGE OF STUDENTS ON OR ABOVE, NEAR, AND FAR FROM TARGET: FALL, WINTER, AND SPRING FIGURE 4: NORMATIVE PERFORMANCE OF 4-YEAR-OLDS ON IGDI MEASURES: FALL, WINTER, AND SPRING TABLE 17: NORMATIVE PERFORMANCE OF 4-YEAR-OLDS ON IGDI MEASURES: FALL, WINTER, AND SPRING TABLE 18: FLOOR EFFECT ISSUES WITH PRE-KINDERGARTEN ASSESSMENTS - PERCENTAGE OF STUDENTS NOT COMPLETING SAMPLE IGDIS ITEMS OR SCORING ZERO ON LETTER NAMING/SOUND FLUENCY BY MEASURE AND SEASON (ALL PREK STUDENTS) TABLE 19: PRE-KINDERGARTEN STUDENT GROWTH TABLE 20: IGDI FALL-SPRING GROWTH BY HOURS PER WEEK IN CORE INSTRUCTION: 4-YEAR-OLDS TABLE 21: IGDI FALL-SPRING GROWTH BY HOURS PER WEEK IN CORE INSTRUCTION: ALL PRE- KINDERGARTEN STUDENTS TABLE 22: PARTICIPATION IN PRE-K SUPPLEMENTAL INTERVENTIONS: NUMBER OF STUDENTS AND NUMBER OF MINUTES PER WEEK TABLE 23: IGDI FALL-SPRING GROWTH BY PARTICIPATION IN SUPPLEMENTAL INSTRUCTION: 4-YEAR- OLDS TABLE 24: IGDI FALL-SPRING GROWTH BY PARTICIPATION IN SUPPLEMENTAL INSTRUCTION: ALL PRE-KINDERGARTEN STUDENTS FIGURE 5: CROSS-COHORT PERCENT ABOVE TARGET ON EARLY LITERACY MEASURES TABLE 25: PERCENT OF PRE-K STUDENTS MOVING FROM BELOW TO AT OR ABOVE TARGET TABLE 26: ELLCO PERFORMANCE IN THE FALL AND SPRING TABLE 27: KINDERGARTEN - GRADE 3 PARTICIPANT GROWTH TABLE 28: AVERAGE LINEAR GROWTH RATES, BY REGION FIGURE 7: GRADE 1 NONSENSE WORD FLUENCY GROWTH CURVE ESTIMATES BY REGION FIGURE 8: GRADE 1: ORAL READING FLUENCY GROWTH CURVE ESTIMATES BY REGION MINNESOTA READING CORPS STATE-WIDE EVALUATION 3

4 FIGURE 9: GRADE 2: ORAL READING FLUENCY GROWTH CURVE ESTIMATES BY REGION FIGURE 10: GRADE 3: ORAL READING FLUENCY GROWTH CURVE ESTIMATES BY REGION TABLE 29: KINDERGARTEN - GRADE 3 PERCENTAGE OF STUDENTS ABOVE GROWTH TARGETS BY REGION TABLE 30: AVERAGE TUTORING PARTICIPATION BY GRADE LEVEL TABLE 31: AVERAGE TUTORING PARTICIPATION FOR K-3 BY GRADE LEVEL AND REGION TABLE 32: PERCENTAGE OF STUDENTS WITH 3 WEEKS OF PROGRESS MONITORING DATA WHO EXIT TABLE 33: PERCENTAGE OF STUDENTS WITH 3 WEEKS OF PROGRESS MONITORING DATA WHO EXIT BY REGION FIGURE 11: PERCENTAGE OF STUDENTS SUCCESSFULLY EXITED: TO COMPARISON TABLE 34: PERCENTAGE WHO EXIT: FALL BENCHMARK TIER 2 VS. TIER TABLE 35: PERCENTAGE WHO EXIT: FALL BENCHMARK TIER 2 VS. TIER 3 BY REGION FIGURE 12: PERCENT OF STUDENTS IN EACH TIER WHO EXITED SUCCESSFULLY TABLE 36: PERCENTAGE WHO EXIT WHO ALSO MEET OR EXCEED SPRING BENCHMARK TABLE 37: PERCENTAGE WHO EXIT WHO ALSO MEET OR EXCEED SPRING BENCHMARK BY REGION FIGURE 13: CHANGE IN PERCENTAGE OF STUDENTS EXITING AND MEETING SPRING BENCHMARK TARGETS FIGURE 14: PERCENT OF STUDENTS WITH SUCCESSFUL EXIT OR BENCHMARK SCORE AND SUCCESS ON STATEWIDE READING ASSESSMENTS GRADE 3: COMPARISON OF TO TABLE 39: OUTCOMES OF SECOND GRADE MRC PARTICIPATING STUDENTS ON SPRING GRADE 3 STATEWIDE READING ASSESSMENT TABLE 40: OUTCOMES OF FIRST GRADE MRC PARTICIPATING STUDENTS ON SPRING GRADE 3 STATEWIDE READING ASSESSMENT TABLE 41: OUTCOMES OF KINDERGARTEN MRC PARTICIPATING STUDENTS ON SPRING GRADE 3 STATEWIDE READING ASSESSMENT TABLE 42: TOTAL NUMBER OF STUDENTS MONITORED IN MRC PROGRAMS, WITH THIRD GRADE STATEWIDE READING ASSESSMENT IN TABLE 45: INTERNAL COACH SYSTEMS CHANGE SURVEY RESULTS TABLE 46: MRC MEMBER IMPACT SURVEY RESULTS TABLE 47: K-FOCUS MRC MEMBER IMPACT SURVEY RESULTS TABLE 48: COMPARISON OF PERCENT OF STUDENTS MEETING INDIVIDUAL GROWTH RATES BY PARTICIPATION IN KINDERGARTEN FOCUS MODEL TABLE 49: AVERAGE WEEKLY GROWTH RATE OF ALL MRC PARTICIPATING STUDENTS IN KINDERGARTEN FOCUS MODEL * FIGURE 15: GROWTH CURVE ESTIMATES, BY K-FOCUS PARTICIPATION TABLE 50: THREE-YEAR HISTORY OF TOTAL NUMBERS OF KINDERGARTEN STUDENTS SERVED IN SITES IMPLEMENTING K-FOCUS BY YEAR SITE BEGAN IMPLEMENTING K-FOCUS TABLE 51: MATCHED PAIRS T-TESTS OF WORD CONSTRUCTION TREATMENT VS. COMPARISON GROUPS TABLE 52: FAMILY LITERACY PILOT PROGRAM PARTICIPATION RATES MINNESOTA READING CORPS STATE-WIDE EVALUATION 4

5 BACKGROUND OF THE MINNESOTA READING CORPS Minnesota Reading Corps (MRC) is an AmeriCorps program that provides trained literacy tutors (Members) for children age three to grade three. Some MRC Members work with preschoolers and focus on integrating talking, reading, and writing into all classroom activities. Other Members provide supplemental literacy skills tutoring for children in kindergarten to third grade. Still others recruit, train, and manage community volunteers to expand the capacity of the program. MRC Members and community volunteers are trained in specific researchbased literacy instructional protocols, and supported by expert coaches. Members use reliable, valid assessment tools to monitor student progress on a regular basis, and with help from their expert coaches, use data from assessments to inform tutoring strategies for each student. Use of specific research based instructional techniques and technically adequate assessment tools for decision making make the MRC program both highly unique and highly coveted across the literacy landscape. The vision of the Minnesota Reading Corps is to impact literacy in the state of MN through children, AmeriCorps members and communities as follows: All children in MN, ages 3 to grade 3, who qualify for MRC, will have access to MRC and will meet reading standards by third grade. AmeriCorps members, through the training, development and service opportunity provided by MRC, will pursue education related careers and/or continue to be ambassadors for children's literacy throughout their lives. Schools and community institutions/organizations, through their experiences with MRC, will understand and adopt the MRC methods for increasing literacy; those institutions will, in turn, promote MRC literacy methods to their colleagues. MINNESOTA READING CORPS STATE-WIDE EVALUATION 5

6 EVALUATION DESIGN The evaluation for the Minnesota Reading Corps program year includes six broad questions as listed below. Additions to the current year evaluation include the evaluation of a new intervention designed to support early word reading, and a family literacy pilot program. 1. What is the current impact of the MRC on the state of Minnesota in terms of numbers of students and programs receiving support? 2. Are the data collection tools used with children being implemented with fidelity? 3. Are the interventions used with children being implemented with fidelity? 4. Is the performance of MRC-participating students in terms of their literacy improvement consistent with expectations? 5. Are the organizations with which the MRC is working changing to adopt the practices of the MRC? 6. What is the impact of the MRC experience on the AmeriCorps Members? MINNESOTA READING CORPS STATE-WIDE EVALUATION 6

7 Assessment Data Collection The assessment data listed below was collected during the school year. Minnesota Reading Corps members collect all student assessment data. Internal and master coaches collect fidelity data for assessments and interventions. Internal coaches, site supervisors, and members complete survey data regarding perceptions of systems change and impact. Internal and master coaches collect Pre-K environmental observation data. Student Data for Pre-school Programs: Age 3 on or before Sept 1 st Fall Winter Spring IGDI Rhyming IGDI Picture Naming IGDI Alliteration IGDI Rhyming IGDI Picture Naming IGDI Alliteration IGDI Rhyming IGDI Picture Naming IGDI Alliteration Age 4 on or before Sept 1 st Age 5 on or before Sept 1 st but not enrolled in K IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency MINNESOTA READING CORPS STATE-WIDE EVALUATION 7

8 Student Data for K-3 Programs: Kindergarten Letter Naming Fluency Letter Sound Fluency Grade 1 Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Grade 2 Oral Reading Fluency (3 passages) Grade 3 Oral Reading Fluency (3 passages) Fall Winter Spring Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Fluency Nonsense Word Fluency Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Letter Naming Fluency Letter Sound Fluency Nonsense Word Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Additional Data: Observations of assessment fidelity three times per year Observations of intervention fidelity between three and nine times per year in K-3 programs, monthly in Pre-K programs. Early Language and Literacy Classroom Observation two times per year in Pre-K programs only End of year MRC Member surveys End of year MRC site surveys MINNESOTA READING CORPS STATE-WIDE EVALUATION 8

9 EVALUATION REPORT 1. Current Impact What is the current impact of the MRC on the state of Minnesota in terms of numbers of students and programs receiving support? In the tables below, the number of Minnesota Reading Corps members, by position type, serving during the school year, who collected data for students during one or more benchmark assessment windows and submitted the data for evaluation is recorded. In addition, the number of students receiving MRC services for whom data are recorded is reported. Numbers of participating students are compiled according to the following criteria: Number of preschool students with data from 1 benchmark window Number of preschool students with data from 3 benchmark windows Number of K-3 students benchmarked as a follow up from previous year s tutoring Number of K-3 students with at least 1 week of data on at least 1 measure Number of K-3 students with at least 5 weeks of data on at least 1 measure Number of K-3 students with at least 10 weeks of data on at least 1 measure Number of K-3 students with at least 20 weeks of data on at least 1 measure The following table provides a count of Minnesota Reading Corps participating Members who served during the school year in Pre-K and K-3 programs. The total number of hours served is also displayed. Overall counts, regional counts, and counts-by-position are provided. Table 1: Overall Member Counts and Service Hours, by Region and Total Region Position Number of Members Number of Hours Served Central K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps MINNESOTA READING CORPS STATE-WIDE EVALUATION 9

10 Volunteer Coordinator Metro K-3 Literacy Tutor Family Literacy Member Kindergarten-Focused Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Northcentral K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Northeast K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Northwest K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Southeast K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Southwest K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Grand Total MINNESOTA READING CORPS STATE-WIDE EVALUATION 10

11 Table 2: Pre-Kindergarten Participation Region Number of Members Number of Students with 1 or More Assessments in 1 Window Number of Students with 1 or more Assessments in 3 Windows Central Metro 118 3, North Central North East North West South East South West Total 207 5,953 4,750 Table 3: Number of K-3 Students benchmarked as follow-up to prior years' tutoring Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Table 4: Number of Students with at least 1 data point, any measure Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL MINNESOTA READING CORPS STATE-WIDE EVALUATION 11

12 Table 5: Number of Students with at least 5 weeks of data (R-CBM, NWF, or LSF) Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Table 6: Number of Students with at least 10 weeks of data (R-CBM, NWF, or LSF) Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Table 7: Number of Students with at least 20 weeks of data (R-CBM, NWF, or LSF) Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Growth in Data Reported The figure below compares total numbers of students served with data recorded for the MRC program during the , , , and school years. Significant growth in total amount of student data recorded is noted across pre-k and k-3 programs. MINNESOTA READING CORPS STATE-WIDE EVALUATION 12

13 Figure 1: Growth in Student Data Reported 14,000 12,000 10,000 8,000 6,000 4, ,000 0 # Pre-K Students with Data in 1 or More Benchmark Windows # Pre-K Students with Data in 3 Benchmark Windows # K-3 Students with 5 or More Data Points # K-3 Students with 20 or More Data Points # K-3 Students Benchmarked as Follow up to Last Year's Tutoring In order to more fully acknowledge the impact on student participation in the Minnesota Reading Corps, the following tables summarize the total number of students identified as having participated in the MRC program, regardless of data presence or tutoring time: Table 8: Minnesota Reading Corps Participation: All Students Grade Active Exited Moved Re-Enrolled Referred to other services Grand Total PreK PreK PreK PreK PreK Age Unknown MINNESOTA READING CORPS STATE-WIDE EVALUATION 13

14 PreK Total K TOTAL Further analysis was completed to identify the average participation dosage for students in both the Pre-K and the K-3 MRC programs. Average number of weeks, sessions per week, and minutes per week of tutoring service provided is summarized below, first in total across the state, and then broken down for the K-3 program by region. Increases in both number of sessions per week and minutes per week is noted in K-3 for the school year relative to Table 9: Participation Levels in MRC Tutoring Number of Students Average Sessions per Week Average Total Minutes Average Minutes per Week Average Average Total Grade Weeks Sessions Minutes PreK PreK PreK PreK PreK Unknown 81 N/A N/A N/A N/A N/A N/A PreK Total K TOTAL Table 10: Participation Levels in K-3 MRC Tutoring by Region Grade North East Number of Students Average Weeks Average Sessions Average Sessions per Week Average Total Minutes Average Minutes per Week Total Minutes K MINNESOTA READING CORPS STATE-WIDE EVALUATION 14

15 TOTAL North Central K TOTAL Metro K TOTAL North West K TOTAL South East K TOTAL South West K TOTAL Central K TOTAL MINNESOTA READING CORPS STATE-WIDE EVALUATION 15

16 Demographics In order to more fully describe the population of children served by the Minnesota Reading Corps program, data regarding gender, ethnicity, and English Learner Status were collected. These demographic data for participating students were entered into the MRC OnCorps Database by members. Table 11: Pre-Kindergarten - Grade 3 Participant Demographic Data Gender Ethnicity English Learner Status Pre-K 51.37% Male 48.57% Female 0.07% Not Reported 2.35% American Indian / Alaska Native 19.57% African American 8.45% Asian 12.54% Hispanic / Latino 0.50% Pacific Islander 2.65% Unknown / No Response 6.43% Multiple 46.68% White 0.83% Not Reported 75.24% English as Primary Language 24.11% ELL 0.65% Not Reported K % Male 46.02% Female 4.65% Not Reported 2.55% American Indian / Alaska Native 18.51% African American 8.24% Asian 0.02% Filipino 10.10% Hispanic / Latino 0.29% Pacific Islander 2.27% Unknown/No Response 2.58% Multiple 48.85% White 6.59% Not Reported 66.71% English as Primary Language 14.88% ELL 18.41% Not Reported MINNESOTA READING CORPS STATE-WIDE EVALUATION 16

17 2. Fidelity of Assessment Data Collection What is the Research Base for MRC Program Assessments? As listed in the above, the assessment tools used to determine literacy progress of MRC-participating students include the following measures: Picture Naming Fluency Alliteration Fluency Rhyming Fluency Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Oral Reading Fluency These tools were selected for use in the MRC because of their well-established statistical reliability and validity for screening and progress monitoring purposes. Picture Naming, Alliteration, and Rhyming measures were developed through the University of Minnesota, and are commonly referred to as Individual Growth and Development Indicators (IGDIs) of literacy. Letter Naming, Letter Sounds, and Nonsense Words are measures of early literacy skills thoroughly researched by many groups, but most famously packaged by two assessment programs: DIBELS and AIMSweb. Oral Reading Fluency provides an assessment of connected text reading. Early and ongoing research on this measure has also been conducted at the University of Minnesota. All these measures fit under the umbrella of Curriculum-Based Measurement (CBM), and are fluency based assessments, meaning that students respond to an unlimited number of items within a fixed amount of time, and the number of correct responses is counted. The information that follows summarizes empirical findings related to the statistical reliability and validity of the measures used in the Minnesota Reading Corps. Picture Naming Fluency: r=.44 to.78 1 month alternate form reliability r=.67 test-retest 3-week reliability r=.47 to.75 with PPVT-3 and.63 to.81 with PLS-3 MINNESOTA READING CORPS STATE-WIDE EVALUATION 17

18 r=.32 to.37 with DIBELS Letter Naming Fluency and.44 to.49 with DIBELS Initial Sound Fluency r=.41 (longitudinal) and.60 (cross sectional) between scores and chronological age, with correlations of.63,.32, and.48 for typically developing, HeadStart, and ECSE populations respectively Sources: McConnell, S.R., Priest, J.S., Davis, S.D., & McEvoy, M.A. (2002). Best Practices in Measuring Growth and Development for Preschool Children, In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology (4 th ed., Vol. 2, ). Washington DC: National Association of School Psychologists. Missall, K.N., & McConnell, S.R. (April, 2004). Psychometric Characteristics for Individual Growth and Development Indicators: Picture Naming, Rhyming, and Alliteration (Technical Report). Minneapolis, MN: University of Minnesota. Accessed online at July 27, Missall, K.N. et. al. (2007). Examination of Predictive Validity of Preschool Early Literacy Skills. School Psychology Review 36 (3) Missall, K. N., McConnell, S. R., & Cadigan, K. (2006). Early literacy development: Skill growth and relations between classroom variables for preschool children. Journal of Early Intervention, 29, Phaneuf, R. L., & Silberglitt, B. (2003). Tracking preschoolers' language and preliteracy development using a general outcome measurement system. Topics in Early Childhood Special Education, 23, Priest, J. S., McConnell, S. R., Walker, D., Carta, J. J., Kaminski, R. A., McEvoy, M. A., Good, R., Greenwood, C. R., & Shinn, M. R. (2001). General growth outcomes for young children: Developing a foundation for continuous progress measurement. Journal of Early Intervention, 24, Wackerle, Alisha K. (2007). Test review: Selection of general growth outcomes for children between birth and age eight. Assessment for Effective Intervention. 33(1), MINNESOTA READING CORPS STATE-WIDE EVALUATION 18

19 Alliteration: r=.46 to.80 test-retest reliability over 3 weeks r=.40 to.57 with PPVT-3 r=.34 to.55 with Clay s Concepts about Print r=.75 to.79 with TOPA r=.39 to.71 with DIBELS Letter Naming Fluency r=.61 with chronological age Sources: McConnell, S.R., Priest, J.S., Davis, S.D., & McEvoy, M.A. (2002). Best Practices in Measuring Growth and Development for Preschool Children, In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology (4 th ed., Vol. 2, ). Washington DC: National Association of School Psychologists. Missall, K.N., & McConnell, S.R. (April, 2004). Psychometric Characteristics for Individual Growth and Development Indicators: Picture Naming, Rhyming, and Alliteration (Technical Report). Minneapolis, MN: University of Minnesota. Accessed online at July 27, Missall, K.N. et. al. (2007). Examination of Predictive Validity of Preschool Early Literacy Skills. School Psychology Review 36 (3) Missall, K. N., McConnell, S. R., & Cadigan, K. (2006). Early literacy development: Skill growth and relations between classroom variables for preschool children. Journal of Early Intervention, 29, Phaneuf, R. L., & Silberglitt, B. (2003). Tracking preschoolers' language and preliteracy development using a general outcome measurement system. Topics in Early Childhood Special Education, 23, Priest, J. S., McConnell, S. R., Walker, D., Carta, J. J., Kaminski, R. A., McEvoy, M. A., Good, R., Greenwood, C. R., & Shinn, M. R. (2001). General growth outcomes for young children: Developing a foundation for continuous progress measurement. Journal of Early Intervention, 24, VanDerHeyden, A.M., Snyder, P.A., Broussard, C., & Ramsdell, K. (2007). Measuring Response to Early Literacy Intervention with Preschoolers at Risk. Topics in Early Childhood Special Education 27(4), MINNESOTA READING CORPS STATE-WIDE EVALUATION 19

20 Wackerle, Alisha K. (2007). Test review: Selection of general growth outcomes for children between birth and age eight. Assessment for Effective Intervention. 33(1), Rhyming: r=.83 to.89 test-retest reliability over 3 weeks r=.56 to.62 with PPVT-3 r=.54 to.64 with Clay s Concepts about Print r=.44 to.62 with TOPA r=.44 to.63 with IGDI Picture Naming and.43 with IGDI Alliteration r=.48 to.59 with DIBELS Letter Naming Fluency r=.44 to.68 with DIBELS Initial Sound Fluency r=.46 with chronological age Sources: McConnell, S.R., Priest, J.S., Davis, S.D., & McEvoy, M.A. (2002). Best Practices in Measuring Growth and Development for Preschool Children, In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology (4 th ed., Vol. 2, ). Washington DC: National Association of School Psychologists. Missall, K.N., & McConnell, S.R. (April, 2004). Psychometric Characteristics for Individual Growth and Development Indicators: Picture Naming, Rhyming, and Alliteration (Technical Report). Minneapolis, MN: University of Minnesota. Accessed online at July 27, Missall, K.N. et. al. (2007). Examination of Predictive Validity of Preschool Early Literacy Skills. School Psychology Review 36 (3) Missall, K. N., McConnell, S. R., & Cadigan, K. (2006). Early literacy development: Skill growth and relations between classroom variables for preschool children. Journal of Early Intervention, 29, Phaneuf, R. L., & Silberglitt, B. (2003). Tracking preschoolers' language and preliteracy development using a general outcome measurement system. Topics in Early Childhood Special Education, 23, MINNESOTA READING CORPS STATE-WIDE EVALUATION 20

21 Priest, J. S., McConnell, S. R., Walker, D., Carta, J. J., Kaminski, R. A., McEvoy, M. A., Good, R., Greenwood, C. R., & Shinn, M. R. (2001). General growth outcomes for young children: Developing a foundation for continuous progress measurement. Journal of Early Intervention, 24, VanDerHeyden, A.M., Snyder, P.A., Broussard, C., & Ramsdell, K. (2007). Measuring Response to Early Literacy Intervention with Preschoolers at Risk. Topics in Early Childhood Special Education 27(4), Wackerle, Alisha K. (2007). Test review: Selection of general growth outcomes for children between birth and age eight. Assessment for Effective Intervention. 33(1), Letter Naming Fluency: r=.94 inter rater reliability r=.90 2 week test retest reliability r=.88 1 month alternate reliability r=.93 alternate forms reliability r=.70 with WJ-R Readiness Cluster r=.70 with WJ Psychoeducational Battery r=.53 to.58 with CTOPP Composite Predictive r=.65 with WJ Total Reading Cluster Predictive r=.71 with R-CBM ELL Predictive r =.67 with a composite of DIBELS NWF and R-CBM Sources: Assessment Committee Report for Reading First. (2002). Analysis of Reading Assessment Measures. Retrieved February 21, 2007, from Good, R.H., Kaminski, R.A., Shinn, M. Bratten, J., Shinn, M., & Laimon, L. (in preparation). Technical Adequacy and Decision Making Utility of DIBELS (Technical Report). Eugene, OR: University of Oregon Good, R.H. III., Kaminski, R.A., Simmons, D., Kame enui, E.J. (2001). Using Dynamic Indicators of Basic Early Literacy Skills (DIBELS) in an outcomes-driven model: Steps to reading outcomes. Unpublished manuscript, University of Oregon at Eugene. MINNESOTA READING CORPS STATE-WIDE EVALUATION 21

22 Elliot, J., Lee, S.W., Tolefson, N. (2001). A Reliability and Validity Study of the Dynamic Indicators of Basic Early Literacy Skills Modified. School Psychology Review, 30 (1), Haager, D. & Gersten, R (April, 2004). Predictive Validity of DIBELS for English Learners in Urban Schools. DIBELS Summit conference presentation, Albuquerque, NM Hintz, J.M., Ryan, A.L., & Stoner, G. (2003). Concurrent Validity and Diagnostic Accuracy of DIBELS and the CTOPP. School Psychology Review Kaminski, R.A. & Good, R.H. (1996). Toward a Technology for Assessment Basic Early Literacy Skills. School Psychology Review, 25, Rouse, H., Fantauzzo, J.W. (2006). Validity of the Dynamic Indicators of Basic Early Literacy Skills as an Indicator of Early Literacy for Urban Kindergarten Children. School Psychology Review 35 (3) Letter Sound Fluency: r=.83 2-week test-retest reliability r=.80 alternate form reliability r=.79 with Letter Naming Fluency Predictive r=.72 with R-CBM Sources: Elliott, J., Lee, S.W., & Tollefson, N. (2001). A Reliability and Validity Study of the Dynamic Indicators of Basic Early Literacy Skills Modified. School Psychology Review, 30 (1), Fuchs, L., Fuchs D. (2004). Determining Adequate Yearly Progress from Kindergarten through Grade 6 with Curriculum Based Measurement. Assessment for Effective Intervention 29 (4) Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School-Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, MINNESOTA READING CORPS STATE-WIDE EVALUATION 22

23 Scott, S.A., Sheppard, J., Davidson, M.M., & Browning, M.M. (2001). Prediction of First Graders Growth in Oral Reading Fluency Using Kindergarten Letter Naming Fluency. Journal of School Psychology, 39(3), Ritchey, K.D (2008). Assessing Letter Sound Knowledge: A Comparison of Letter Sound Fluency and Nonsense Word Fluency. Exceptional Children 74 (4) Nonsense Word Fluency: r=.83 one month alternate form reliability r=.36 to.59 with WJ-R Readiness Cluster Predictive r=.82 with Spring R-CBM in Spring of grade 1 Predictive r =.65 with oral reading and.54 with maze in grade 3 Ell Predictive r=.63 with a composite of DIBELS NWF and R-CBM Sources: Burke, M. D., Hagan-Burke, S. (2007). Concurrent criterion-related validity of early literacy indicators for middle of first grade. Assessment for Effective Intervention. 32(2), Good, R.H., Kaminski, R.A., Shinn, M. Bratten, J., Shinn, M., & Laimon, L. (in preparation). Technical Adequacy and Decision Making Utility of DIBELS (Technical Report). Eugene, OR: University of Oregon Good, R.H., Kaminski, R.A., Simmons, D., & Kame-enui, E.J. (2001). Using DIBELS in an Outcomes Driven Model: Steps to Reading Outcomes. Unpublished manuscript, University of Oregon, Eugene. Haager, D. & Gersten, R (April, 2004). Predictive Validity of DIBELS for English Learners in Urban Schools. DIBELS Summit conference presentation, Albuquerque, NM Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School-Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, MINNESOTA READING CORPS STATE-WIDE EVALUATION 23

24 Kaminski, R.A. & God, R.H. (1996). Toward a Technology for Assessment Basic Early Literacy Skills. School Psychology Review, 25, Ritchey, K.D (2008). Assessing Letter Sound Knowledge: A Comparison of Letter Sound Fluency and Nonsense Word Fluency. Exceptional Children 74 (4) Rouse, H., Fantauzzo, J.W. (2006). Validity of the Dynamic Indicators of Basic Early Literacy Skills as an Indicator of Early Literacy for Urban Kindergarten Children. School Psychology Review 35 (3) Vanderwood, M.., Linklater, D., Healy, K. (2008). Predictive Accuracy of Nonsense Word Fluency for English Language Learners. School Psychology Review 37 (1) Oral Reading Fluency: r=.92 to.97 test retest reliability r=.89 to.94 alternate form reliability r=.82 to.86 with Gates-MacGinite Reading Test r=.83 to Iowa Test of Basic Skills r =.88 to Stanford Achievement Test r=.73 to.80 to Colorado Student Assessment Program r=.67 to Michigan Student Assessment Program r=.73 to North Carolina Student Assessment Program r=74 to Arizona Student Assessment Program r=.61 to.65 to Ohio Proficiency Test, Reading Portion r=.58 to.82 with Oregon Student Assessment Program (SAT 10) Sources: Barger, J. (2003). Comparing the DIBELS Oral Reading Fluency indicator and the North Carolina end of grade reading assessment (Technical Report). Ashville, NC: North Carolina Teacher Academy. Baker S. et. al,. (2008). Reading Fluency as a Predictor of Reading Proficiency in Low-Performing, High-Poverty Schools. School Psychology Review 37 (1) MINNESOTA READING CORPS STATE-WIDE EVALUATION 24

25 Burke, M. D., Hagan-Burke, S. (2007). Concurrent criterion-related validity of early literacy indicators for middle of first grade. Assessment for Effective Intervention. 32(2), Deno, S. L., Mirkin, P. K., & Chiang, B. (1982). Identifying valid measures of reading. Exceptional Children, Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School-Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, Hintze, J.M, et al (2002). Oral Reading Fluency and Prediction of Reading Comprehension in African American and Caucasian Elementary School Children. School Psychology Review, 31 (4) Hintze, J. M. & Silberglitt, B. (in press). A Longitudinal Examination of the Diagnostic Accuracy and Predictive Validity of R-CBM and High-Stakes Testing. School Psychology Review. Marston, D., Fuchs, L., & Deno, S. (1987). Measuring pupil progress: a comparison of standardized achievement tests and curriculum-related measures. Diagnostique, 11, Marston, D. (1989). Curriculum-based measurement: What is it and why do it? In M. R. Shinn (Ed.), Curriculum-based measurement: Assessing special children (pp ). New York: Guilford Press. McGlinchey, M. T., & Hixson, M. D. (2004). Contemporary research on curriculumbased measurement: Using curriculum-based measurement to predict performance on state assessments in reading. School Psychology Review, 33(2), Schilling, S. G., Carlisle, J. F., Scott, S. E., & Zeng, J. (2007). Are fluency measures accurate predictors of reading achievement? The Elementary School Journal, 107(5), Silberglitt, B. & Hintze, J. M. (in press). Formative Assessment Using Oral Reading Fluency Cut Scores to Track Progress Toward Success on State-Mandated MINNESOTA READING CORPS STATE-WIDE EVALUATION 25

26 Achievement Tests: A Comparison of Methods. Journal of Psychoeducational Assessment. Shaw, R., & Shaw, D. (2002). DIBELS Oral Reading Fluency-Based Indicators of the third-grade reading skills for Colorado State Assessment Program (CSAP) (Technical Report). Eugene, OR: University of Oregon. Shinn, M., Good, R., Knutson, N., Tilly, W., & Collins, A. (1992). Curriculum-based measurement of oral reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, Stage, S. A., & Jacobsen, M. D. (2001). Predicting student success on a statemandated performance-based assessment using oral reading fluency. School Psychology Review, 30(3), Tindal, G., Germann, G., & Deno, S. (1983). Descriptive research on the Pine County Norms: A compilation of findings (Research Report No. 132). Minneapolis, MN: University of Minnesota Institute for Research on Learning Disabilities. Vander Meer, C. D., Lentz, F. E., & Stollar, S. (2005). The relationship between oral reading fluency and Ohio proficiency testing in reading (Technical Report). Eugene, OR: University of Oregon. Wilson, J. (2005). The relationship of Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Oral Reading Fluency to performance on Arizona Instrument to Measure Standards (AIMS). Tempe, AZ: Tempe School District No. 3. Are the data collection tools used with children being implemented with fidelity? Analysis of fidelity with which student assessment data are conducted is a critical initial aspect to the evaluation of the MRC program so that results from evaluation of these data may be reported with confidence. In order to accomplish this, a series of Accuracy of Implementation Rating Scales (AIRS) have been compiled from each Minnesota Reading Corps (MRC) site. MRC Internal Coaches and Master Coaches were trained in August 2011 to administer and score assessment measures, and to conduct observations of Reading Corps members as they administer and score these measures. The AIRS MINNESOTA READING CORPS STATE-WIDE EVALUATION 26

27 are structured observational protocols that provide an opportunity for observers to certify that each aspect of a standardized administration for each assessment measure has been fully conducted. Master and/or Internal Coaches completed a minimum of 1 AIRS for each Reading Corps Member for each type of assessment the member conducted at least 3 times each year, near the benchmark data collection periods. The table below documents the total number of AIRS assessments compiled, the total number of AIRS assessments completely filled out on forms, and percent fidelity documented of those completely filled out on forms for each measure. In addition to reporting these data in aggregate format to document high fidelity of assessment procedures across the state, this observation system also provided Members with immediate feedback regarding the quality of their own assessment skills, and an opportunity to receive clarification or re-training as needed in a timely manner. MINNESOTA READING CORPS STATE-WIDE EVALUATION 27

28 Table 12: Fidelity of Assessment Data Collection Procedures Measure Rhyming Alliteration Picture Naming PK Letter Naming PK Letter Sounds K-3 Letter Naming K-3 Letter Sounds Nonsense Words Oral Reading Fluency Total AIRS Collected Total Complete AIRS Fidelity Range Reported 0% - 100% 11% - 100% 0% - 100% 36%- 100% 0% - 100% 46% - 100% 38% - 100% 0% - 100% 15% - 100% Median % Fidelity Reported Mean % Fidelity Reported Standard Deviation 100% 100% 100% 100% 100% 100% 100% 100% 100% 99% 99% 99% 99% 99% 98% 98% 98% 98% Fidelity of Intervention Implementation Are the interventions used with children being implemented with fidelity? As with the assessment tools, analysis of the level of fidelity with which the student intervention protocols are followed is a critical initial aspect to the evaluation of the MRC program so that results of student growth analysis may be attributed to accurate implementation of intervention scripts. In order to accomplish this, a series of intervention integrity observations have been MINNESOTA READING CORPS STATE-WIDE EVALUATION 28

29 compiled from each MRC site. MRC internal and master coaches were trained in August 2011 to evaluate implementation integrity for each of the MRC interventions. The integrity checklists provide an opportunity for observers to certify that each aspect of a standardized administration for each intervention has been fully conducted. Master and/ or internal coaches completed a minimum of 1 intervention integrity checklist for each MRC member during each monthly visit, for a possible total of 9 checklists per member per year. The interested reader is referred to Ehrhardt, Barnett, Lentz, Stollar, & Reifin, (1996) for a description of how to use scripts to improve intervention integrity. The first 10 intervention protocols listed are available to all students involved in the K-3 MRC program. The next 7 are offered in a sub-set of K-3 MRC sites whose Members have been nominated and provided additional training on implementation of small group intervention protocols. The final group of intervention protocols are those offered in the Pre-K MRC program. The table below documents the number of integrity checklists compiled and percent fidelity documented for each intervention: Table 13: Fidelity of Intervention Implementation Procedures Intervention Total Fidelity Median Mean Standard Complete Range Percent Percent Deviation Fidelity Reported Fidelity Fidelity Checks Reported Reported Duet Reading % - 100% 100% 96% 0.08 Newscaster Reading % - 100% 100% 96% 0.08 Repeated Reading with % - 100% 94% 87% 0.15 Comprehension Strategy Practice Stop / Go % - 100% 100% 97% 0.66 Pencil Tap % - 100% 100% 94% 0.11 Great Leaps % - 100% 96% 92% 0.13 Letter Sound Identification % - 100% 100% 95% 0.11 Word Blending % - 100% 100% 96% 0.10 Phoneme Blending % - 100% 100% 96% 0.09 Phoneme Segmenting % - 100% 100% 97% 0.07 Word Construction %-100% 95% 91% 0.15 Partner Reading (Paired) 44 78% - 100% 100% 97% 0.06 Phoneme Blending (Paired) 88 36% - 100% 100% 95% 0.11 Phoneme Segmenting (Paired) 62 46% - 100% 100% 94% 0.12 Letter Sound Identification % - 100% 100% 94% 0.11 (Paired) Blending Words (Paired) 64 64% - 100% 100% 95% 0.09 Repeated Read Aloud (K Focus) % - 100% 89% 85% 0.15 Pre-K Oral Language % - 100% 100% 97% 0.07 MINNESOTA READING CORPS STATE-WIDE EVALUATION 29

30 Pre-K Phonological Awareness % - 100% 100% 96% 0.06 Pre-K Phonological Awareness %-100% 100% 97% 0.08 Pre-K Phonological Awareness %-100% 100% 97% 0.05 Pre-K Phonological Awareness %-100% 100% 90% 0.13 Pre-K Repeated Read Aloud % - 100% 91% 83% 0.19 Pre-K Repeated Read Aloud %-100% 91% 86% 0.19 Pre-K Repeated Read Aloud %-100% 91% 74% 0.34 Pre-K Repeated Read Aloud %-100% 100% 95% 0.10 Pre-K Sign In % - 100% 100% 95% 0.10 Pre-K Visual Discrimination % - 100% 100% 94% 0.09 Pre-K Visual Discrimination %-100% 100% 94% 0.08 Pre-K Visual Discrimination %-100% 100% 96% 0.07 Pre-K Visual Discrimination %-100% 100% 97% 0.04 Pre-K Visual Discrimination %-100% 100% 98% 0.07 Pre-K Visual Discrimination %-100% 100% 100% 0.13 Growth in Fidelity Reporting The chart below compares total numbers of fidelity observations recorded for the MRC program during the , , , and school years for interventions offered across more than one year. Significant growth in total complete observations recorded is noted, mirroring growth in the program. MINNESOTA READING CORPS STATE-WIDE EVALUATION 30

31 Figure 2: Growth in Fidelity Reporting What is the Research Base for MRC Program Interventions? The interventions identified for use in the MRC program are each designed to provide additional practice that is supplemental to the core reading instructional program offered by the local school site. This practice is provided with the intention of building automaticity and fluency of important reading skills that have already been introduced by local classroom teachers. It is important to note that MRC participation is in addition to, not in replacement of, a comprehensive core reading instructional program, and that the MRC program should in no way be viewed as a substitute for high quality core instruction. MRC provides important additional guided practice time in reading for students who need this support. For further discussion regarding the benefit of supplemental support to students at risk for reading failure, see Harn (2008). For a discussion of benefit of well-matched interventions, see Wagner et al (2006). The chosen interventions share a common theme in focus on building fluency for basic reading skills such as phonemic awareness, letter sound knowledge, MINNESOTA READING CORPS STATE-WIDE EVALUATION 31

32 decoding skill, and sight word recognition. Fluency is interpreted in this program as incorporating rate, accuracy, and prosody, or expression. Richard Allington, Former president of the International Reading Association writes: "There are a substantial number of rigorously designed research studies demonstrating (1) that fluency can be developed, most readily through a variety of techniques that involve rereading texts and (2) that fostering fluency has reliable positive impacts on comprehension and performance. Thus when fluency is an instructional goal, as it should be for struggling readers, we have a wealth of research to guide our instructional planning. (Allington, 2001) For further discussion on the relationship between oral reading fluency and comprehension skills, the interested reader is referred to Tenenbaum & Wolking (1989) and Sindelar, Monda, & O Shea, (1990). A unique feature of MRC is the consistent use of research-based intervention protocols with participating students to provide this additional support. MRC members select from a menu of research-based supplemental reading interventions for use with participating MRC students as listed below. For each intervention protocol, a description of the research base, and/or sources of empirical evidence of intervention effectiveness are listed. Repeated Reading with Comprehension Strategy Practice Nelson, J. S., Alber, S. R., & Grody, A. (2004). Effects of systematic error correction and repeated readings on reading accuracy and proficiency of second graders with disabilities. Education and Treatment of Children, 27, Staubitz, J. E., Cartledge, G., Yurick, A., & Lo, Y. (2004). Repeated reading for students with emotional or behavioral disorders: Peerand trainermediated instruction. Behavior Disorders, 31, Therrien, W. J. (2004). Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education, 25, MINNESOTA READING CORPS STATE-WIDE EVALUATION 32

33 Moyer, S.B. (1982). Repeated reading. Journal of Learning Disabilities, 45, Rashotte, C.A., & Torgeson, J.K. (1985). Repeated reading and reading fluency in learning disabled children. Reading Research Quarterly. 20, Samuels, S. J. (1979). The method of repeated reading. The Reading Teacher, 32, Samuels, S.J., (1987). Information processing abilities and reading. Journal of Learning Disabilities, 20(1), Sindelar, P.T., Monda, L.E., & O Shea, L.J. (1990). Effects of repeated reading on instructional and mastery level readers. Journal of Educational Research, 83, Therrien, W.J. (2004). Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education. 25(4) Morrow, L. M. (1985). Retelling stories: A strategy for improving young children s comprehension, concept of story structure, and oral language complexity. The Elementary School Journal, 85, Duet Reading Aulls, M.W., (1982). Developing Readers in Today s Elementary Schools. Allyn & Bacon: Boston. Blevins, W. (2001). Building Fluency: Lessons and Strategies for Reading Success. New York: Scholastic Professional Books. Dowhower, S.L. (1991). Speaking of prosody: Fluency s unattended bedfellow. Theory into Practice, 30 (3), Mathes, P.G., Simmons, D.C., & Davis, B.I. (1992). Assisted reading techniques for developing reading fluency. Reading Research and Instruction, 31, MINNESOTA READING CORPS STATE-WIDE EVALUATION 33

34 Weinstein, G., & Cooke, N. L. (1992). The effects of two repeated reading interventions on generalization of fluency. Learning Disability Quarterly, 15, Newscaster Reading Armbruster, B.B., Lehr, F., & Osborn, J. (2001). Put reading first: The research building blocks for teaching children to read. Washington, DC: US Department of Education, National Institute for Literacy. Dowhower. S.L. (1987). Effects of repeated reading on second-grade transitional readers fluency and comprehension. Reading Research Quarterly. 22, (listening to a tape) Heckelman, R.G. (1969). A neurological-impress method of remedial reading instruction. Academic Therapy, 4, Daly, E. J., III, & Martens, B. (1994). A comparison of three interventions for increasing oral reading performance: Application of the instructional hierarchy. Journal of Applied Behavior Analysis, 29, Skinner, C. H., Adamson, K. L., Woodward, J. R., Jackson, R. R., Atchison, L. A., & Mims, J. W. (1993). The effects of models rates of reading on students reading during listening previewing. Journal of Learning Disabilities, 26, Rasinski, T.V. (2003). The fluent reader: Reading strategies for building word recognition, fluency, and comprehension. New York, NY: Scholastic Professional Books. Searfoss, L. (1975). Radio Reading. The Reading Teacher, 29, Stahl S. (2004). What do we Know About Fluency?: Findings of the National Reading Panel. In McCardle, P., & Chhabra, V. (Eds) The Voice of Evidence in Reading Research. Brookes: AU. Stop Go MINNESOTA READING CORPS STATE-WIDE EVALUATION 34

35 Blevins, W. (2001). Building Fluency: Lessons and Strategies for Reading Success. New York: Scholastic Professional Books. Rasinski, T., & Padak, N. (1994). Effects of fluency development on urban second-graders. Jorunal of Education Research, 87. Rasinski, T.V. (2003). The fluent reader: Reading strategies for building word recognition, fluency, and comprehension. New York, NY: Scholastic Professional Books. Pencil Tap Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Education Research. 77(1), Howell, K., W., & Nolet. V., (2000). Curriculum-Based Evaluation: Teaching and Decision Making 3 rd Ed. Belmont, CA: Wadsworth. Lysakowski, R.S., & Walberg, H.J. (1982). Instructional effects of cues, participation, and corrective feedback: A quantitative synthesis. American Educational Research Journal Vol 19(4), Tenenbaum, G., & Goldring, E. (1989). A meta-analysis fo the effecta of enhanced instruction: Cues, participation, reinforcement and feedback and correctives on motor skill learning. Journal of Research & Development in Education. Vol 22(3) Great Leaps Mercer, Cecil D., Campbell, Kenneth U., Miller, W. David, Mercer, Kenneth D., and Lane, Holly B. Effects of a Reading Fluency Intervention for Middle Schoolers with Specific Learning Disabilities. Learning Disabilities Research and Practice, 15(4), Meyer, Marianne. Repeated Reading: An Old Standard is Revisited and Renovated. Perspectives, 28(1), MINNESOTA READING CORPS STATE-WIDE EVALUATION 35

36 Letter Sound Identification Adams, M.J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press. Adams, M.J. (2001).Alphabetic anxiety and explicit, systematic phonics instruction: A cognitive science perspective. In S.B. Neuman & D.K. Dickinson (eds.), Handbook of Early Literacy Research (pp ). New York: Guilford Press. Chard, D.J., & Osborn, J. (1999). Word Recognition: Paving the road to successful reading. Intervention in school and clinic, 34(5), Word Blending Adams, M.J. (2001).Alphabetic anxiety and explicit, systematic phonics instruction: A cognitive science perspective. In S.B. Neuman & D.K. Dickinson (eds.), Handbook of Early Literacy Research (pp ). New York: Guilford Press. Goswami, U. (2000). Causal connections in beginning reading: The importance of rhyme. Journal or Research in Reading, 22(3) Greaney, K.T., Tunmer, W.E., & Chapman, J.W., (1997). Journal of Educational Psychology, 89(4) Phoneme Blending Adams, M.J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press. Bos, C.D., & Vaughn, S. (2002). Strategies for teaching students with learning and behavioral problems (5 th Ed.). Boston: Allyn & Bacon. Ehri, L.C., Nunees, S.R., & Willows, D.M. (2001). Phonemic awareness instruction helps children learn to read: Evidence from the National Reading Panel s meta-analysis. Reading Research Quarterly, 36(3) MINNESOTA READING CORPS STATE-WIDE EVALUATION 36

37 Elkonin, D.B. (1973). U.S.S.R. In J. Downing (Ed.), Comparative Reading (pp ). New York: MacMillan. National Reading Panel. (2000). Teaching children to read: An evidencebased assessment of the scientific research literature on reading and its implications for reading instruction. Bethesda, MA: National Institutes of Health. Santi, K.L., Menchetti, B.M., & Edwards, B.J. (2004). A comparison of eight kindergarten phonemic awareness programs based on empirically validated instructional principals. Remedial and Special Education, Vol 25(3) Smith, C.R. (1998). From gibberish to phonemic awareness: Effective decoding instruction. Exceptional Children, Vol 30(6) Smith, S.B., Simmons, D.C., & Kame enui, E, J. (1998). Phonological Awareness: Research bases. In D.C. Simmons & E.J. Kame enui (Eds.), What Reading research tells us about children with diverse learning needs: Bases and basics. Mahwah, NJ: Lawrence Erlbaum Associates. Snider, V. E. (1995). A primer on phonemic awareness: What it is, why it is important, and how to teach it. School Psychology Review, 24, Phoneme Segmentation Adams, M.J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press. Blachman, B. A. (1991). Early intervention for children s reading problems: Clinical applications of the research on phonological awareness. Topics in Language Disorders, 12, Bos, C.D., & Vaughn, S. (2002). Strategies for teaching students with learning and behavioral problems (5 th Ed.). Boston: Allyn & Bacon. Ehri, L.C., Nunees, S.R., & Willows, D.M. (2001). Phonemic awareness instruction helps children learn to read: Evidence from the National MINNESOTA READING CORPS STATE-WIDE EVALUATION 37

38 Reading Panel s meta-analysis. Reading Research Quarterly, 36(3) National Reading Panel. (2000). Teaching children to read: Anevidencebased assessment of the scientific research literature on reading and its implications for reading instruction. Bethesda, MA: National Institutes of Health. Santi, K.L., Menchetti, B.M., & Edwards, B.J. (2004). A comparison of eight kindergarten phonemic awareness programs based on empirically validated instructional principals. Remedial and Special Education, Vol 25(3) Smith, C.R. (1998). From gibberish to phonemic awareness: Effective decoding instruction. Exceptional Children Vol 30(6) Smith, S.B., Simmons, D.C., & Kame enui, E, J. (1998). Phonological Awareness: Research bases. In D.C. Simmons & E.J. Kame enui (Eds.), What Reading research tells us about children with diverse learning needs: Bases and basics. Mahwah, NJ: Lawrence Erlbaum Associates. Snider, V. E. (1995). A primer on phonemic awareness: What it is, why it is important, and how to teach it. School Psychology Review, 24, Partner Reading Fuchs, D., Fuchs, L., & Burish, P. (2000). Peer-Assisted Learning Strategies: An Evidence-Based Practice to Promote Reading Achievement. Learning Disabilities Research and Practice, 15(2), Koskinen, P. & Blum, I. (1986). Paired repeated reading: A classroom strategy for developing fluent reading. The Reading Teacher, 40(1), Rathvon, N. (2008) Effective School Interventions, Second Edition. New York, NY, Guilford Press Vaughn, S., Chard, D., Bryant, D., Coleman, M., Tyler, B., Linan Thompson, S., & K. Kouzekanani. (2000) Fluency and comprehension interventions for third grade students. Remedial and Special Education, 21(6), MINNESOTA READING CORPS STATE-WIDE EVALUATION 38

39 Word Construction McCandliss, B.D., Beck, I., Sandak, R., & Perfetti, C. (2003). Focusing attention on decoding for children with poor reading skills: A study of the Word Building intervention. Scientific Studies of Reading.7(1), Repeated Read Aloud McGee, Lea M., & Schickedanz, Judith A. (2007). Repeated interactive read-alouds in preschool and kindergarten. The Reading Teacher. 60(8), Lonigan, C. J., Anthony, J. L., Bloomield, B. G., Dyer, S. M., & Samwel, C. S. (1999). Effects of two shared-reading interventions on emergent literacy skills of at-risk preschoolers. Journal of Early Intervention, 22(4), Whitehurst, G. J., Arnold, D. S., Epstein, J. N., Angell, A. L., Smith, M., & Fischel, J. E. (1994). A picture book reading intervention in day care and home for children from low-income families. Developmental Psychology, 30(5), Whitehurst, G. J., Epstein, J. N., Angell, A. L., Payne, A. C., Crone, D. A., & Fischel, J. E. (1994). Outcomes of an emergent literacy intervention in Head Start. Journal of Educational Psychology, 86(4), MINNESOTA READING CORPS STATE-WIDE EVALUATION 39

40 4. Student Outcomes Is the performance of MRC participating students in terms of their literacy improvement consistent with expectations? The following sections document growth and achievements of children age 3 to grade 3 who participated in the MRC program in the school year. It is important to acknowledge that MRC participating students are also supported by a variety of resources, most notably the instruction and guidance provided by their schools and families. This evaluation is not intended to address or control for the variables related to these resources, nor to suggest that student progress or lack thereof must be attributed solely to the service being provided through the efforts of the MRC program. This design s purpose is to focus on the desired literacy outcomes for all children. Pre-Kindergarten Student Performance The five measurement tools utilized for the Pre-K Reading Corps program are listed below. For a description of the technical characteristics of these assessments, the interested reader is referred to the Minnesota Reading Corps program evaluation. For each assessment tool, a target score was identified as the goal for the end of the year. These target scores were based on the target scores used in Minneapolis Public Schools for incoming kindergarten students, and upon 50 th %ile scores for incoming kindergarten students within school districts served by Saint Croix River Education District. Prior to use of these targets for the current project, these targets were reviewed by the researchers at the University of Minnesota who originally created the Individual Growth and Development Indicators. The measures and target scores for this project are listed below: Measure Spring Target Score Rhyming 12 Picture Naming 26 Alliteration 8 Letter Sound Fluency 8 Letter Naming Fluency 14 MINNESOTA READING CORPS STATE-WIDE EVALUATION 40

41 Pre-Kindergarten student performance on fall, winter, and spring IGDI measures is listed in the tables below for all students with a birth-date reported, and data collected during the appropriate assessment window. Score ranges are also reported. Students score NA when they do not complete the sample items sufficiently to warrant participation in the assessment. A score of 0 indicates adequate performance on the sample items, but no accurate responses during the assessment. Table 14: Kindergarten Participant Performance on IGDIs: Fall Benchmark Measure Output Three Year Olds Four Year Olds Five Year Olds Picture Fall Number Students Tested 955 4, Naming Range of Scores NA-37 NA-44 NA-47 Fall (Number) Percent Students Above Spring Target (65) 6.81% (909) 20.25% (74) 37.56% Alliteration Fall Number Students Tested 937 4, Range of Scores NA-14 NA-31 NA-22 Fall (Number) Percent Students Above Spring Target (13) 1.39% (417) 9.28% (38) 19.59% Rhyming Fall Number Students Tested 954 4, Range of Scores NA-21 NA-28 NA-25 Fall (Number) Percent Students Above Spring Target (12) 1.26% (535) 11.98% (51) 26.42% Letter Naming Fall Number Students Tested 646 5, Fluency Range of Scores NA-40 NA-63 NA-68 Fall (Number) Percent Students Above Spring Target (36) 5.57% (1,212) 21.73% (87) 38.33% Letter Sound Fall Number Students Tested 255 4, Fluency Range of Scores NA-21 NA-110 NA-41 Fall (Number) Percent Students Above Spring Target (10) 3.92% (628) 14.17% (53) 27.18% MINNESOTA READING CORPS STATE-WIDE EVALUATION 41

42 Table 15: Pre-Kindergarten Participant Performance on IGDIs: Winter Benchmark Measure Output Three Year Olds Four Year Olds Five Year Olds Picture Winter Number Students Tested 917 4, Naming Range of Scores NA-43 NA-50 NA-47 Winter (Number) Percent Students Above Spring Target (237) 25.85% (2,037) 46.94% (122) 61.93% Alliteration Winter Number Students Tested 884 4, Range of Scores NA-24 NA-30 NA-30 Winter (Number) Percent Students Above Spring Target (67) 7.58% (1,179) 27.55% (79) 40.31% Rhyming Winter Number Students Tested 916 4, Range of Scores NA-25 NA-44 NA-29 Winter (Number) Percent Students Above Spring Target (99) 10.81% (1,570) 36.29% (91) 46.19% Letter Naming Winter Number Students Tested 373 4, Fluency Range of Scores NA-50 NA-79 NA-76 Winter (Number) Percent Students Above Spring Target (85) 22.79% (2,428) 55.83% (124) 62.94% Letter Sound Winter Number Students Tested 270 4, Fluency Range of Scores NA-34 NA-77 NA-49 Winter (Number) Percent Students Above Spring Target (26) 9.63% (1,696) 39.61% (92) 47.18% Table 16: Pre-Kindergarten Participant Performance on IGDIs: Spring Benchmark Measure Output Three Year Olds Four Year Olds Five Year Olds Picture Spring Number Students Tested 806 3,8, Naming Range of Scores NA-41 NA-58 NA-47 Spring (Number) Percent Students Above Spring Target (398) 49.38% (2,795) 72.09% (139) 79.89% Alliteration Spring Number Students Tested 784 3, Range of Scores NA-27 NA-38 NA-34 Spring (Number) Percent Students Above Spring Target (142) 18.11% (2,047) 52.94% (109) 63.01% Rhyming Spring Number Students Tested 797 3, Range of Scores NA-28 NA-37 NA-35 Spring (Number) Percent Students Above Spring Target (179) 22.46% (2,330) 60.02% (119) 68.39% Letter Naming Spring Number Students Tested 291 3, Fluency Range of Scores NA-63 NA-95 NA-73 Spring (Number) Percent Students Above (119) (3,020) (141) MINNESOTA READING CORPS STATE-WIDE EVALUATION 42

43 Letter Sound Fluency Spring Target 40.89% 78.10% 81.03% Spring Number Students Tested 243 3, Range of Scores NA-43 NA-105 NA-61 Spring (Number) Percent Students Above (61) (2,512) (126) Spring Target 25.10% 65.78% 72.83% The figure below shows the percentage of students who were on or above, near, and far below target by each benchmark period. Data are displayed for each early literacy measure. Figure 3: Percentage of Students On or Above, Near, and Far from Target: Fall, Winter, and Spring The figure and table below show the normative performance of all 4-year-old students participating in the IGDI measures during each benchmark window during the school year. As a clarification, it is noted that students who did not successfully complete the sample items on each assessment measure in order to continue on to the actual assessment were given a score of NA which is recorded in this figure as a -1. MINNESOTA READING CORPS STATE-WIDE EVALUATION 43

44 Figure 4: Normative Performance of 4-Year-Olds on IGDI Measures: Fall, Winter, and Spring Items Correct Letter Naming Spring Letter Naming Winter Letter Naming Fall Alliteration Spring Alliteration Winter Alliteration Fall Picture Naming Spring Picture Naming Winter Picture Naming Fall Letter Sound Spring Letter Sound Winter Letter Sound Fall Rhyming Spring Rhyming Winter Rhyming Fall Green Diamond = 90 th %ile score; Mid-Blue X = 75 th %ile score; Navy Triangle = 50 th %ile score; Light Blue Diamond= 25 th %ile score; Red Square = 10 th %ile score. MINNESOTA READING CORPS STATE-WIDE EVALUATION 44

45 Table 17: Normative Performance of 4-Year-Olds on IGDI Measures: Fall, Winter, and Spring 90th %ile 75th %ile 50th %ile 25th %ile 10 th %ile Number Students in Norm Group Rhyming - Fall Rhyming - Winter Rhyming - Spring Letter Sounds - Fall Letter Sounds - Winter Letter Sounds - Spring Picture Naming - Fall Picture Naming - Winter Picture Naming - Spring Alliteration - Fall Alliteration - Winter Alliteration - Spring Letter Naming - Fall Letter Naming - Winter Letter Naming - Spring NA NA NA NA 2 NA NA NA NA As these data indicate, there may be significant floor effect issues with the IGDI assessments for this population. This means that the assessments do not accurately measure the skills of some students because these students are not yet able to demonstrate the skills expected on the assessment. Assessments for which a floor effect exists may not fully capture growth that is occurring for students with low skills. The table below provides the percentage of students in fall and spring who received a score of NA on each IGDI assessment, or a score of zero on Letter Naming and Letter Sound Fluency. MINNESOTA READING CORPS STATE-WIDE EVALUATION 45

46 Table 18: Floor Effect Issues with Pre-Kindergarten Assessments - Percentage of Students Not Completing Sample IGDIs Items or scoring zero on Letter Naming/Sound Fluency By Measure and Season (All PreK Students) Fall Spring Rhyming 34.42% 9.17% Letter Naming 8.02% 2.50% Picture Naming 7.16% 0.63% Alliteration 43.64% 14.39% Letter Sounds 16.25% 4.89% The number of children who demonstrated growth across the school year from fall to spring on each assessment measure is reported below. All students included in this analysis had a valid birthdate and a valid score for at least one measure in both fall and spring assessment windows. Table 19: Pre-Kindergarten Student Growth 3 Year Olds 4 Year Olds Number Percent Number Percent Showed Growth on 1 or More Measures % % Showed Growth on 2 or More Measures % % Showed Growth on 3 or More Measures % % Showed Growth on 4 or More Measures % % Showed Growth on 5 Measures % % Total Students Additional analysis was completed for the school year to evaluate the effects of length of instructional program on student growth as measured by the IGDIs. Average fall to spring growth for students who were 4 years old on September 1, 2011 is reported below for students in programs meeting 2-9 hours per week, hours per week, and 16+ hours per week. Table 20: IGDI Fall-Spring Growth by Hours per Week in Core Instruction: 4-Year-Olds 2-9 Hrs/Week Hrs/Week 16+ Hrs/Week Unreported Grand Total Rhyming Growth* Average Standard Deviation N Letter Naming Growth Average MINNESOTA READING CORPS STATE-WIDE EVALUATION 46

47 Standard Deviation N Picture Naming Growth Average Standard Deviation N Alliteration Growth* Average Standard Deviation N Letter Sounds Growth* Average Standard Deviation N * - Significant floor effects may have affected the results for these measures For comparison purposes, the same data are provided below for all Pre-K enrolled students, regardless of age. Table 21: IGDI Fall-Spring Growth by Hours per Week in Core Instruction: All Pre-Kindergarten Students 2-9 Hrs/Week Hrs/Week 16+ Hrs/Week Unreported Grand Total Rhyming Growth* Average Standard Deviation N Letter Naming Growth Average Standard Deviation N Picture Naming Growth Average Standard Deviation N Alliteration Growth* Average Standard Deviation N Letter Sounds Growth* Average Standard Deviation N * - Significant floor effects may have affected the results for these measure MINNESOTA READING CORPS STATE-WIDE EVALUATION 47

48 Supplemental Intervention in MRC Pre-K Program During the school year, some pre-k students in each classroom were identified to receive supplemental intervention support in addition to the core instructional program. These additional small group interactions within the classroom were designed to provide additional opportunities to practice important early literacy skills. In the analysis reported below, student participation rates in supplemental interventions are provided, including number of students participating, and the average and standard deviation of the total number of minutes of supplemental intervention received during the school year. Table 22: Participation in Pre-K Supplemental Interventions: Number of Students and Number of Minutes per Week Four Year-Olds only All PreK Students Average Total Minutes Standard Deviation of Total Minutes Number of Participants IGDI Growth Data were further analyzed to explore differences in IGDI growth by participation in supplemental interventions. Growth statistics are disaggregated by whether the student participated in the intervention ( None if no participation) and the total number of minutes of intervention received over the school year (1-99 minutes, minutes, 200+ minutes) in the tables below. Results are reported first for students who were 4 years old on September 1, 2011, then for all pre-k participating students. Table 23: IGDI Fall-Spring Growth by Participation in Supplemental Instruction: 4-Year-Olds 1-99 Minutes Minutes 200+ Minutes None Grand Total Rhyming Growth* Average Standard Deviation N Letter Naming Growth Average Standard Deviation MINNESOTA READING CORPS STATE-WIDE EVALUATION 48

49 N Picture Naming Growth Average Standard Deviation N Alliteration Growth* Average Standard Deviation N Letter Sounds Growth* Average Standard Deviation N * - Significant floor effects may have affected the results for these measures Table 24: IGDI Fall-Spring Growth by Participation in Supplemental Instruction: All Pre-Kindergarten Students 1-99 Minutes Minutes 200+ Minutes None Grand Total Rhyming Growth* Average Standard Deviation N Letter Naming Growth Average Standard Deviation N Picture Naming Growth Average Standard Deviation N Alliteration Growth* Average Standard Deviation N Letter Sounds Growth* Average Standard Deviation N * - Significant floor effects may have affected the results for these measures MINNESOTA READING CORPS STATE-WIDE EVALUATION 49

50 A cross-cohort analysis of performance by 4-year-old students has been compiled across eight years of the Minnesota Reading Corps program. The following figure shows the percent of 4-year-old MRC participants meeting the assessments spring target scores at fall, winter, and spring assessment times across years. It is noted that the data was analyzed by an outside agency, and only includes students enrolled in Headstart MRC classrooms. Across successive school years, fairly stable fall performance is noted, along with an overall increase in percent of students meeting spring target scores by the spring benchmark window. Data from the current year show an increase in performance relative to last year in 5 of 5 measures, and highest percentage above target by spring to date in 5 of 5 measures. Figure 5: Cross-Cohort Percent Above Target on Early Literacy Measures 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% % Rhyming Picture Naming Alliteration Letter Sounds Letter Names % 44.70% 26.80% 20.30% 38.90% % 57.10% 35.60% 32.10% 44.30% % 35.00% 22.00% 32.00% 44.00% % 50.90% 47.30% 39.40% 49.80% % 58.90% 43.40% 49.80% 63.30% % 54.70% 35.02% 44.90% 62.38% % 58.72% 38.90% 44.80% 65.46% % 70.51% 48.24% 56.79% 72.83% % 72.09% 52.94% 65.78% 78.10% MINNESOTA READING CORPS STATE-WIDE EVALUATION 50

51 Analysis was completed to determine the percent of pre-k students with fall scores below target who moved to target or higher than target in each measure by spring. For each measure, students were included if they were age 3-5 on September 1, 2011, and if they had in window scores recorded in fall and spring for each of the measures. Current year data demonstrates an increase across all five measures for percent of students moving from below to at or above target relative to the school year. The table below summarizes the findings. Table 25: Percent of Pre-K Students Moving from Below to At or Above Target Measure Number of Pre-K students with Fall Scores Below Target and a Spring Score Percent of Pre-K Students with Fall Scores Below Target and Spring Scores At or Above Target Rhyming 4, % Letter Naming Fluency 2, % Picture Naming 3, % Alliteration 3, % Letter Sound Fluency 3, % Classroom Outcomes for Pre-K Sites Two times per year, Master or Internal coaches completed a systematic observation in each classroom using the Early Language and Literacy Classroom Observation (ELLCO) tool. This observation tool is designed to address the classroom environmental factors that support language and literacy development for pre-kindergarten students. The table below illustrates performance in fall and spring for pre-kindergarten classroom assessed using this measure. Total possible score is 95 points. Table 26: ELLCO Performance in the Fall and Spring Percentile Fall ELLCO Total Score Spring ELLCO Total Score 90 th th th th th MINNESOTA READING CORPS STATE-WIDE EVALUATION 51

52 Kindergarten-Grade 3 Student Performance The four assessment tools utilized for the K-3 Reading Corps program are listed below. A description of the technical characteristics of these assessments, is listed above. For each assessment tool, a target score was identified as the goal for the beginning, middle, and end of the year. These target scores were based on research conducted at the St. Croix River Education District, which documented the predictive and concurrent validity of these measures with the Minnesota Comprehensive Assessment II. As a result of the strong correlations between performance on the selected fluency measures and on the statewide reading assessment, a series of cut scores has been identified. The table below specifies assessments given at each grade level and the cut scores for each assessment during several points throughout the school year. These cut scores, or target scores, define levels of performance on the fluency measures that strongly predict future success on the grade 3 statewide reading assessment. For example, a student who reads a Grade 1 passage in the winter of first grade at a rate of 22 words read correctly per minute has a 75% chance of earning a score in the meets standards or exceeds standards ranges on the 3 rd grade statewide reading assessment two years later. Grade Measure Fall Target Winter Target Spring Target K Letter Sound Fluency Nonsense Word Fluency N/A 1 Oral Reading Fluency N/A Oral Reading Fluency Oral Reading Fluency The target scores as listed above for each assessment used as a part of ongoing student literacy measurement in Reading Corps grow across years from age 3 to grade 3, define a pathway to success. Through consideration of the inherent growth that would occur for a child who met each of the targets, an expectation of growth rate at each grade level can be defined. For example, the fall grade 2 target score is 43 on oral reading fluency. The spring grade 2 target score on this measure is 90. To grow from 43 to 90 in one academic year, a student would need to gain 1.31 words correct per minute per week on the oral reading fluency assessment. Thus, 1.31 words growth per week becomes the expectation for 2 nd grade growth rates. Because our targets are connected to the state- wide assessment rather than normative performance of other MINNESOTA READING CORPS STATE-WIDE EVALUATION 52

53 students in local districts, we have a consistent and meaningful comparison across the state. Students participating in the Minnesota Reading Corps program are monitored frequently. The primary purpose of this progress monitoring is to enable those providing support to the student the ability to evaluate the effectiveness of current reading instruction, and to make data-based decisions regarding changes in instruction. For the purposes of outcomes evaluation, the progress monitoring data also provide a means for comparing the rate of growth of participating Minnesota Reading Corps students to the expected grade level growth rate. Students are selected for participation in the Reading Corps program because they are identified as having below grade level skills in reading. These students who achieve higher growth rates than those indicated by our targets are catching up to grade level expectation by making more than one year s growth in one year s time. For the state-wide Minnesota Reading Corps project, one measure of our success is the extent to which our participating students achieve this primary goal. In the table below, a comparison between weekly growth rate expectations and the average weekly growth rates on program assessment measures of children participating in K-3 Reading Corps programs who have at least 3 data points collected per measure is listed. The current analysis includes all data collected between 9/1/2011 and 6/30/2012. In all grade levels, the average growth rate of Minnesota Reading Corps participants exceeded the target growth rate. Note that in Grade 1, while mean growth of participants in the second half of the year, on Oral Reading Fluency, was slightly below target growth, the mean growth in the first half of the year, on Nonsense Word Fluency, exceeded target growth. In other words, the average growth rate for Minnesota Reading Corps participants exceeded a rate of one year s growth in one year s time. This demonstrates that participating students are actually catching up to grade level expectations. MINNESOTA READING CORPS STATE-WIDE EVALUATION 53

54 Table 27: Kindergarten - Grade 3 Participant Growth Grade K (Letter Sound Fluency) Grade 1 (Nonsense Word Fluency) Grade 1 (R-CBM)** Grade 2 (R-CBM) Grade 3 (R-CBM) MRC Mean Growth Rate Target Growth Rate Number of Students * Only students with 3 or more data points on the given measure were included in growth rate calculations ** Students in this group may have also participated in Grade 1 NWF Table 28: Average Linear Growth Rates, By Region Grade K (LSF) Grade 1 (NWF) Grade 1 (R-CBM) Grade 2 (R-CBM) Grade 3 (R-CBM) North East North Central Metro North West South East South West Central TOTAL The average linear growth is reported above, and used throughout the evaluation. This is done to provide an easily interpretable metric that can be used to provide group comparisons and can be readily compared to growth rates of benchmark target scores. Research suggests, however, that growth rates on many general outcome measures are typically non-linear across a given school year. The following graphs are provided to allow for closer examination of this non-linear growth pattern, as well as for visual examination of differences in these patterns across districts. Figures 5 through 9 depict the growth curve estimates for participating MRC students. These visual displays more accurately capture the quadratic nature of growth on the assessment measures. On each chart the bold black lines represent target scores at each season of the school year. The solid red line represents the aggregate growth curve for all students, and the remaining lines represent the growth curve for students from each district. MINNESOTA READING CORPS STATE-WIDE EVALUATION 54

55 Figure 6: Grade K: Letter Sound Fluency Growth Curve Estimates by Region 80 Letter Sounds Correct per Minute NE NC Metro NW SE SW Central TOTAL TARGET -40 Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Week 17 Week 19 Week 21 Week 23 Week 25 Week 27 Week 29 Week 31 Week 33 Week 35 MINNESOTA READING CORPS STATE-WIDE EVALUATION 55

56 Figure 7: Grade 1 Nonsense Word Fluency Growth Curve Estimates by Region Correct Sounds per Minute NE NC Metro NW SE SW Central TOTAL TARGET 0 Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Week 17 Week 19 Week 21 Week 23 Week 25 Week 27 Week 29 Week 31 Week 33 Week 35 MINNESOTA READING CORPS STATE-WIDE EVALUATION 56

57 Figure 8: Grade 1: Oral Reading Fluency Growth Curve Estimates by Region Words Correct per Minute NE NC Metro NW SE SW Central TOTAL TARGET 0 Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Week 17 Week 19 Week 21 Week 23 Week 25 Week 27 Week 29 Week 31 Week 33 Week 35 MINNESOTA READING CORPS STATE-WIDE EVALUATION 57

58 Figure 9: Grade 2: Oral Reading Fluency Growth Curve Estimates by Region Words Correct per Minute NE NC Metro NW SE SW Central TOTAL TARGET 0 Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Week 17 Week 19 Week 21 Week 23 Week 25 Week 27 Week 29 Week 31 Week 33 Week 35 MINNESOTA READING CORPS STATE-WIDE EVALUATION 58

59 Figure 10: Grade 3: Oral Reading Fluency Growth Curve Estimates by Region Words Correct per Minute NE NC Metro NW SE SW Central TOTAL TARGET 0 Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Week 17 Week 19 Week 21 Week 23 Week 25 Week 27 Week 29 Week 31 Week 33 Week 35 MINNESOTA READING CORPS STATE-WIDE EVALUATION 59

60 The table below examines K-3 students who have at least 3 data points collected per measure, represented by Total Number of Students. The table presents the percentage of these students whose individual growth rates exceeded the target growth rates for that grade level and measure. This calculation represents the portion of the population participating in a Reading Corps intervention that had growth rates in excess of one year of growth in one year s time. Percentages are given by region, and overall. Table 29: Kindergarten - Grade 3 Percentage of Students Above Growth Targets by Region Region Grade K (LSF) Grade 1 (NWF) Grade 1 (R-CBM)** Grade 2 (R-CBM) Grade 3 (R-CBM) TOTAL * % Above Target 84.21% 86.42% 39.17% 66.24% 65.66% 67.19% North East Total Number of Students % Above Target 92.00% 81.01% 55.74% 70.93% 84.35% 77.24% North Central Total Number of Students % Above Target 86.34% 79.46% 42.61% 64.07% 66.85% 69.88% Metro Total Number of Students % Above Target 94.51% 83.11% 53.77% 84.95% 76.06% 80.17% North West Total Number of Students % Above Target 91.30% 82.11% 53.68% 79.93% 75.90% 76.77% South East Total Number of Students % Above Target 84.11% 79.77% 52.63% 70.30% 64.64% 70.50% South West Total Number of Students % Above Target 93.84% 87.80% 57.50% 80.58% 73.08% 79.05% Central Total Number of Students TOTAL % Above Target 87.46% 80.97% 46.26% 68.69% 68.68% 71.77% Total Number of Students * TOTAL represents the total number of slopes analyzed, not the total number of students, as students in Grade 1 may have participated in two categories ** Students in this group may have also participated in Grade 1 (NWF) Regarding student performance during the school year, data were examined to determine the typical number of weeks that a successful MRC program participant could expect to receive tutoring sessions before MINNESOTA READING CORPS STATE-WIDE EVALUATION 60

61 graduating out of the program. The results of this analysis are displayed below, overall and by region for both pre-k and k-3 populations. Table 30: Average Tutoring Participation by Grade Level Number of Students Average Sessions per Week Average Total Minutes Average Minutes per Week Average Average Total Grade Weeks Sessions Minutes PreK PreK PreK PreK PreK Unknown 81 N/A N/A N/A N/A N/A N/A PreK Total K TOTAL Table 31: Average Tutoring Participation for K-3 by Grade Level and Region Number of Students Average Weeks Average Sessions Average Sessions per Week Average Total Minutes Average Minutes per Week Total Minutes Grade North East K TOTAL North Central K TOTAL Metro K MINNESOTA READING CORPS STATE-WIDE EVALUATION 61

62 TOTAL North West K TOTAL South East K TOTAL South West K TOTAL Central K TOTAL Starting in the school year, exit criteria were refined for the K-3 program to require that students have both 3-5 consecutive data points above the aim line on their progress monitoring graphs, and that they have at least one data point above the nearest upcoming benchmark target. It was anticipated that this additional exit requirement would result in longer length of service to students in general prior to exiting, and that those who successfully exited the program would also meet spring target expectations and ultimately meet standards on the grade 3 statewide reading assessment in greater numbers. The following tables provide data related to results of this change in exit criteria. In the tables below, the percent of K-3 MRC participating students with at least 3 progress monitor data points who graduated from tutoring (exit) during the year is reported, first across the state, and then by region. Members are asked to report student exit status in two locations: on individual student progress monitor graphs kept within the AIMSweb data system, and within the MINNESOTA READING CORPS STATE-WIDE EVALUATION 62

63 OnCorps data management system. For this and following analyses involving student exit status, status was based on reporting within the OnCorps system. A 3 data point minimum was selected in order to remove from the analysis students who did not have at least a reasonable minimum length of service prior to discontinuation. Table 32: Percentage of Students with 3 Weeks of Progress Monitoring Data Who Exit Grade Number of Students Number Exited Percent Exited K % % % % K-3 Total % TOTAL % Table 33: Percentage of Students with 3 Weeks of Progress Monitoring Data Who Exit by Region Region Grade Number of Students Number Exited Percent Exited North East K % North East % North East % North East % North East TOTAL % North Central K % North Central % North Central % North Central % North Central TOTAL % Metro K % Metro % Metro % Metro % Metro TOTAL % North West K % North West % North West % North West % North West TOTAL % South East K % MINNESOTA READING CORPS STATE-WIDE EVALUATION 63

64 South East % South East % South East % South East TOTAL % South West K % South West % South West % South West % South West TOTAL % Central K % Central % Central % Central % Central TOTAL % The figure below captures the change in percentage of students who successfully exited the program between the and school years. Interestingly, the percentage of students exited returned back to levels at or above that of the school year even through the more stringent exit criteria that were implemented in were employed again in MINNESOTA READING CORPS STATE-WIDE EVALUATION 64

65 Figure 11: Percentage of Students Successfully Exited: to Comparison 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% % 0.00% Kindergart en Grade 1 Grade 2 Grade 3 Total % 52.48% 43.83% 46.72% 50.16% % 46.65% 37.38% 43.58% 46.66% % 51.53% 40.51% 45.98% 52.27% Further examination of exit rates for participating students was conducted through consideration of initial performance level in the fall of the school year. For this analysis, students were divided into 2 groups based on their fall benchmark assessment level: Those beginning the school year somewhat below grade level standards (Tier 2), and those beginning the school year significantly below grade level standards (Tier 3). Category assignment was defined by target scores that reference success on the grade 3 statewide reading assessment. Students in the somewhat below (Tier 2) category were those expected to have between a 25-75% chance of meeting standards on the grade 3 statewide reading assessment. Students in the significantly below (Tier 3) category were those expected to have less than a 25% chance of meeting standards on the statewide reading assessment based on current performance on the assessment measures. Exit rates for these two groups of students are summarized in the table below. This is an important question as it relates to level of students for whom the MRC intervention program is most effective. MINNESOTA READING CORPS STATE-WIDE EVALUATION 65

66 Table 34: Percentage Who Exit: Fall Benchmark Tier 2 vs. Tier 3 Grade Number Fall Tier 2 Percent Tier 2 Exited Number Fall Tier 3 Percent Tier 3 Exited K % % % % % % % K-3 Total % % Table 35: Percentage Who Exit: Fall Benchmark Tier 2 vs. Tier 3 by Region Region Grade Number Fall Tier 2 Percent Tier 2 Exited Number Fall Tier 3 Percent Tier 3 Exited North East K % North East % % North East % % North East % % North East TOTAL % % North Central K % North Central % % North Central % % North Central % % North Central TOTAL % % Metro K % Metro % % Metro % % Metro % % Metro TOTAL % % North West K % North West % % North West % % North West % % North West TOTAL % % South East K % South East % % South East % % South East % % South East TOTAL % % South West K % MINNESOTA READING CORPS STATE-WIDE EVALUATION 66

67 South West % % South West % % South West % % South West TOTAL % % Central K % Central % % Central % % Central % % Central TOTAL % % In keeping with program design, the vast majority of participating students in the Minnesota Reading Corps program have fall assessment scores that fall in the tier 2 range. Given the large differences in number of students in tier 2 and 3 fall performance groups, comparisons of outcomes between the groups are difficult to make. However, the vast difference in success rate for the small group falling below tier 2 as compared to those falling within tier 2 is noted. The following figure displays a graphical representation of the total percentage of students with exit information who successfully exited the MRC program based on their Tier designation at time of fall benchmark data. Figure 12: Percent of Students in Each Tier Who Exited Successfully 70.00% 60.00% 50.00% Percent Exited 40.00% 30.00% 20.00% 10.00% 0.00% Tier 2 Tier 3 Percent Exited 61.53% 29.18% MINNESOTA READING CORPS STATE-WIDE EVALUATION 67

68 During the school year, MRC members continued to collect benchmark data three times per year for all students who had previously participated in the program. This additional data collection allows an analysis of the percentage of students who exit the MRC program who also meet or exceed the spring benchmark scores at the end of the year. This is an important analysis as it allows for study of the current exit criteria for the program. Results are summarized in the tables below. As can be seen in the tables, across the state there is a reported range of 63%- 76% of students who exit the program and also meet spring benchmark targets. Overall, just over 69% of students who exited the program prior to the end of the school year exceeded the spring benchmark target across the state. Table 36: Percentage Who Exit Who Also Meet or Exceed Spring Benchmark Grade Number Exited Number Exited with Spring Benchmark Number Above Spring Benchmark Percent Above Spring Benchmark K % % % % K-3 Total % Table 37: Percentage Who Exit Who Also Meet or Exceed Spring Benchmark by Region Number Exited with Spring Benchmark Number Above Spring Benchmark Percent Above Spring Benchmark Region Grade Number Exited North East K % North East % North East % North East % North East TOTAL % North Central K % North Central % North Central % North Central % North Central TOTAL % MINNESOTA READING CORPS STATE-WIDE EVALUATION 68

69 Metro K % Metro % Metro % Metro % Metro TOTAL % North West K % North West % North West % North West % North West TOTAL % South East K % South East % South East % South East % South East TOTAL % South West K % South West % South West % South West % South West TOTAL % Central K % Central % Central % Central % Central TOTAL % The figure below captures the change in percentage of students who successfully exited the program and met spring benchmark targets between the and school years. As was expected, an increase in the percentage of students meeting spring targets resulted from the increased stringency of the exit criteria. MINNESOTA READING CORPS STATE-WIDE EVALUATION 69

70 Figure 13: Change in Percentage of Students Exiting and Meeting Spring Benchmark Targets 90.00% Percent Meeting Spring Target 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Kindergarte n Grade 1 Grade 2 Grade 3 Total % 61.25% 30.62% 81.31% 57.43% % 75.09% 71.74% 63.56% 68.92% % 75.94% 70.76% 63.04% 69.08% MINNESOTA READING CORPS STATE-WIDE EVALUATION 70

71 Performance on the Statewide Reading Assessment For students participating in the MRC program as third graders during the school year, as second graders during the school year, as first graders during the school year, and/or as kindergarteners during the school year, analysis was completed to compare their outcome in the MRC program with their performance on the spring 2012 grade 3 statewide reading assessment. For most students in the K-3 MRC sample, this is the Minnesota Comprehensive Assessment: II of reading (MCAII). For students identified as English Language Learners, the statewide reading assessment given is the Test of Emerging Academic English (TEAE). Student scores on the grade 3 MCAII and TEAE are considered equivalent, and are reported in aggregate form throughout this report. The tables below summarize these data, first for grade 3 participants, then for grade 2 participants, then the grade 1 participants, and finally the kindergarten participants. Table 38: Outcomes of Third Grade MRC Participating Students on Spring Grade 3 Statewide Reading Assessment District Total with MCA Results Total Pass MCA Percent Pass MCA Number Successful Exit or Spring Benchmark and have MCA Results Of Successful Exit or Benchmark, Percent Success MCA North East % % North Central % % Metro % % North West % % South East % % South West % % Central % % TOTAL % % The following figure provides comparison data on third grade student success on the statewide reading assessment in the through school years. In most regions and for the state in total, a greater percentage of students both exited the MRC program and demonstrated at least grade level performance on the statewide reading assessment in grade 3 in the school year relative to MINNESOTA READING CORPS STATE-WIDE EVALUATION 71

72 Figure 14: Percent of Students with Successful Exit or Benchmark Score and Success on Statewide Reading Assessments Grade 3: Comparison of to % Percent Exited and Successful on MCA 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% North East North Central Metro North West South East South West Central TOTAL % 65.71% 63.11% 70.34% 71.61% 81.63% 77.91% 67.72% % 63.08% 77.34% 79.66% 82.98% 88.98% 83.45% 78.86% % 82.35% 75.12% 81.75% 81.62% 85.08% 85.26% 78.40% MINNESOTA READING CORPS STATE-WIDE EVALUATION 72

73 Table 39: Outcomes of Second Grade MRC Participating Students on Spring Grade 3 Statewide Reading Assessment District Total with MCA Results Total Pass MCA Percent Pass MCA Number Successful Exit or Spring Benchmark and have MCA Results Of Successful Exit or Benchmark, Percent Success MCA North East % % North Central % % Metro % % North West % % South East % % South West % % Central % % TOTAL % % Table 40: Outcomes of First Grade MRC Participating Students on Spring Grade 3 Statewide Reading Assessment District Total with MCA Results Total Pass MCA Percent Pass MCA Number Successful Exit or Spring Benchmark and have MCA Results Of Successful Exit or Benchmark, Percent Success MCA North East % % North Central % % Metro % % North West % % South East % % South West % % Central % % TOTAL % % MINNESOTA READING CORPS STATE-WIDE EVALUATION 73

74 Table 41: Outcomes of Kindergarten MRC Participating Students on Spring Grade 3 Statewide Reading Assessment District Total with MCA Results Total Pass MCA Percent Pass MCA Number Successful Exit or Spring Benchmark and have MCA Results Of Successful Exit or Benchmark, Percent Success MCA North East % % North Central % % Metro % % North West % % South East % % South West Central % % TOTAL % % Table 42: Total Number of Students Monitored in MRC Programs, with Third Grade Statewide Reading Assessment in District Total with MCA Results Total Pass MCA Percent Pass MCA Number Successful Exit or Spring Benchmark in most recent year of participation and have MCA Results Of Successful Exit or Benchmark in most recent year or participation, Percent Success MCA North East % % North Central % % Metro % % North West % % South East % % South West % % Central % % TOTAL % % MINNESOTA READING CORPS STATE-WIDE EVALUATION 74

75 5. Systems Change Are the organizations with which the MRC is working changing to adopt the practices of the MRC? In the spring of the program year, an electronic survey was sent to all internal coaches requesting their feedback regarding systems change impact the MRC program had on their local site. Internal coaches are employees of the host districts who work regularly to support their local members in the MRC role. Overall, 39.0% of respondents indicated that participation in the MRC program was very influential in prompting systems change at the local site. 46.5% indicated that participation in MRC was somewhat influential in prompting systems change. The following results summarize additional feedback received from 274 internal coaches regarding perceptions of specific areas of systems change in MRC sites. Table 45: Internal Coach Systems Change Survey Results The following questions are designed to capture your perceptions regarding changes to your local system resulting from participation in the MRC program. For each item below, select the response the best describes your site. Answer Options Prior to participating in the MRC program, our building collected fluency based screening measures for all students K-3 at least 3 times per year. Due (at least in part) to our participation in the MRC program, our building has begun or will begin collecting fluency based screening measures for all students K-3 at least 3 times per year. Prior to participating in the MRC program, teachers in our building used screening data to assist in identifying students for supplemental interventions. Due (at least in part) to our participation in the MRC program, teachers in our building now use screening data to assist in identifying students for supplemental interventions. Prior to participating in the MRC program, teachers regularly reviewed progress-monitoring data (weekly graphs) of students receiving supplemental interventions. Strongly Agree Agree Disagree Strongly Disagree 55.3% 22.6% 15.1% 7.0% 21.5% 31.3% 27.7% 19.5% 41.4% 41.4% 15.2% 2.0% 27.7% 41.5% 17.9% 12.8% 13.6% 36.9% 40.9% 8.6% MINNESOTA READING CORPS STATE-WIDE EVALUATION 75

76 Due (at least in part) to our participation in the MRC program, teachers now regularly review progressmonitoring data (weekly graphs) of students receiving supplemental interventions. Prior to participating in the MRC program, teachers viewed progress monitoring (weekly data) as an important method to evaluate the impact of instruction on students. Due (at least in part) to our participation in the MRC program, teachers now view progress monitoring (weekly data) as an important method to evaluate the impact of instruction on students. Prior to participating in the MRC program, our school used aggregated data as one way to evaluate the instructional practices of the site. Due (at least in part) to our participation in the MRC program, our school now uses aggregated data as one way to evaluate the instructional practices of the site. Prior to participating in the MRC program, the building principal shared data on student performance with the superintendent or school board. Due (at least in part) to our participation in the MRC program, the building principal now shares data on student performance with the superintendent or school board. Prior to participating in the MRC program, teachers shared progress monitor graphs with parents. Due (at least in part) to our participation in the MRC program, teachers now share progress monitor graphs with parents. Due (at least in part) to our participation in the MRC program, greater emphasis has been placed on selecting reading interventions for students that have a scientific research base. Prior to participating in the MRC program, instruction was modified if student performance was not improving based on the progress monitoring data collected. Due (at least in part) to our participation in the MRC program, instruction is now modified if student performance is not improving based on the progress monitoring data collected. Prior to participating in the MRC program, the district had adopted its own data warehouse system for efficiently storing and accessing data. 19.4% 54.1% 16.8% 8.7% 13.5% 40.5% 38.5% 7.5% 21.2% 52.0% 18.7% 8.1% 25.6% 53.3% 19.1% 2.0% 17.8% 49.2% 23.4% 9.6% 35.0% 48.2% 12.7% 4.1% 16.5% 41.2% 28.4% 13.9% 19.1% 32.2% 39.2% 6.5% 16.4% 55.4% 21.0% 7.2% 29.1% 54.3% 11.6% 5.0% 22.6% 52.8% 22.1% 2.5% 20.0% 59.5% 13.3% 7.2% 38.7% 39.7% 17.6% 4.0% MINNESOTA READING CORPS STATE-WIDE EVALUATION 76

77 Due (at least in part) to our participation in the MRC program, the district has now adopted its own data warehouse system for efficiently storing and accessing data. Due (at least in part) to our participation in the MRC program, the district has taken concrete steps I am aware of to formally link Pre-K with K-3 literacy instruction. 38.7% 39.7% 17.6% 4.0% 13.4% 38.7% 34.5% 13.4% 6. Impact on AmeriCorps Members What is the impact of the MRC experience on the AmeriCorps Members? In the spring of the program year, an electronic survey was sent to all members, requesting their feedback on the types of impact they perceived the MRC program had on them as individuals. The following results summarize feedback received from 465 members. Table 46: MRC Member Impact Survey Results Please indicate your agreement with each of the following statements based on your experience with Minnesota Reading Corps. Answer Options Strongly Agree Agree Disagree Strongly Disagree As a result of my participation in the MRC program, I am considering a career involving children As a result of my participation in the MRC program, I am considering a career in teaching or education As a result of my participation in the MRC program, I am committed to continued volunteering in schools As a result of my participation in the MRC program, I am committed to ongoing promotion of childhood literacy. As a result of my participation in the MRC program, I am committed to continued community service. 42.3% 39.6% 15.6% 2.4% 38.4% 33.7% 23.5% 4.4% 34.3% 52.7% 11.9% 1.1% 57.8% 38.0% 3.7% 0.4% 41.4% 47.6% 9.9% 1.1% MINNESOTA READING CORPS STATE-WIDE EVALUATION 77

78 As a result of my participation in the MRC program, if a job I hold in the future does not have community service as part of it's mission, I will encourage the organization to include opportunities for community service. 30.8% 53.5% 14.1% 1.5% During the school year, a sub-group of MRC members served in a specialized capacity within a kindergarten focus program. These members worked primarily or exclusively with kindergarten age students, meeting with students for 40 minutes per day rather than 20 minutes per day. Half of this intervention time involved the same literacy interventions as those available outside of this K-Focus model. The other half of the time involved the member implementing a shared book reading intervention with groups of 4 students. Survey results for K-Focus members specifically were disaggregated to determine whether their unique role within the MRC program was related to different perceptions of outcome and future plans for members. In all, 43 K- Focus members responded to the survey. Overall while the pattern of responses for K-Focus members was not significantly different from the responses of all members, K-Focus members appeared somewhat less strong in their intention to continue in careers with children or education, but somewhat more strong in their commitment to ongoing advocacy around childhood literacy and community service. Table 47: K-Focus MRC Member Impact Survey Results Please indicate your agreement with each of the following statements based on your experience with Minnesota Reading Corps. Answer Options Strongly Agree Agree Disagree Strongly Disagree As a result of my participation in the MRC program, I am considering a career involving children 38.1% 42.9% 14.3% 4.8% As a result of my participation in the MRC program, I am considering a career in teaching or education As a result of my participation in the MRC program, I am committed to continued volunteering in schools As a result of my participation in the MRC program, I am committed to ongoing promotion of childhood literacy. 31.0% 35.7% 28.6% 4.8% 33.3% 47.6% 16.7% 2.4% 59.5% 28.6% 9.5% 2.4% MINNESOTA READING CORPS STATE-WIDE EVALUATION 78

79 As a result of my participation in the MRC program, I am committed to continued community service. As a result of my participation in the MRC program, if a job I hold in the future does not have community service as part of it's mission, I will encourage the organization to include opportunities for community service. 47.6% 45.2% 7.1% 0.0% 52.4% 40.5% 7.1% 0.0% 7. Action Research: Results of Pilot Studies Study of Kindergarten Focus Model During the school year, MRC began a new pilot study for students enrolled in kindergarten. Students participating in this pilot study were given two 20-minute tutoring sessions per day. One session consisted of one or more of the standard pre-literacy MRC interventions, delivered to a pair of students together. The other daily session consisted of a shared book reading intervention that incorporated elements of phonemic awareness, phonics, and vocabulary instruction. Students participated in the shared book reading intervention in groups of four. Members were chosen to participate in this pilot study based on district of service and master coach recommendation, and were given additional training to perform the shared book reading intervention with fidelity. For purposes of evaluation, performance of students participating in the K-Focus program was compared to the performance of kindergarten students participating in the standard MRC interventions within schools in which K-Focus was also available. Table 48: Comparison of Percent of Students Meeting Individual Growth Rates by Participation in Kindergarten Focus Model Grade K (LSF) K-Focus Total % Above Target 85.52% Total Number of Students 1651 Non K-Focus Total % Above Target 74.76% Total Number of Students 416 Total % Above Target 83.36% Total Number of Students 2067 To further analyze the result of participation in the kindergarten focus model intervention, a comparison of average weekly growth for these students was compared to the average weekly growth of kindergarten students in schools in which K-Focus was available, but who received standard 1:1 tutoring sessions MINNESOTA READING CORPS STATE-WIDE EVALUATION 79

80 instead. Results of this analysis, as shown below, indicate that average weekly growth was significantly greater for students involved in the kindergarten focus model relative to those involved in standard 1:1 tutoring sessions. Table 49: Average Weekly Growth Rate of All MRC Participating Students in Kindergarten Focus Model * Grade K (LSF) K-Focus Growth Rate 2.07 Non K-Focus Growth Rate 1.62** Target Growth Rate 1.15 Number of K-Focus Students 2067 * - Only students with 3 or more data points on the given measure were included in growth rate calculations ** - Difference is statistically significant p<.001 The following chart provides visual description of the growth curve for kindergarten students in the K-Focus and Non K-Focus programs Figure 15: Growth Curve Estimates, by K-Focus Participation One additional measure of growth for the K-Focus program is the number of kindergarten students served. By providing a targeted program designed for kindergarten students, the MRC program hoped to increase the number of students in this grade level served. The table below presents the number of kindergarten students served each school year, for both years with the K-Focus model, and prior years when the K-Focus model was not available. In each year MINNESOTA READING CORPS STATE-WIDE EVALUATION 80

81 that a site began implementing K-Focus, the number of students increased by 5-7 times over the previous year. For example, sites beginning K-Focus in served a total of 144 students in and 117 students in (both years in which K-Focus was not implemented), and then served a total of 888 students in (the first year implementing K-Focus). Table 50: Three-Year History of Total Numbers of Kindergarten Students Served in Sites Implementing K- Focus by Year Site Began Implementing K-Focus Total Number of Students Served Year Site Began K- Focus Active Exited Moved Referred to other services TOTAL Re- Enrolled TOTAL Total Number of Students Served Year Site Began K- Re- Referred to Focus Active Exited Moved Enrolled other services TOTAL TOTAL Total Number of Students Served Year Site Began K- Re- Referred to Focus Active Exited Moved Enrolled other services TOTAL TOTAL Study of Word Construction Intervention In December 2011, a small cohort of members was trained to provide a new intervention to students primarily during the second semester of first grade. This intervention was designed from the research of Bruce McCandliss and Isabel Beck, who demonstrated an effective method of supporting students development of accurate and fluent word decoding skills. Students use letter tiles to build words, write words, and practice reading the words in isolation and in silly sentences. This protocol was selected for inclusion in the Minnesota Reading Corps program to support students who had mastered other basic early word decoding skills covered in MRC early literacy interventions, but who MINNESOTA READING CORPS STATE-WIDE EVALUATION 81

82 were not fully ready to read connected text independently. The interested reader is referred to the primary research publication describing the intervention method: McCandliss, B., Beck, I.L., Sandak, R., & Perfetti, C. (2003). Focusing attention on decoding for children with poor reading skills: Design and preliminary tests of the word building intervention. Scientific Studies of Reading, 7, An initial examination of the effect of the Word Construction intervention was conducted, exploring the effect of Word Construction on the slope of growth on ORF in the second half of first grade. Students participating in Word Construction were matched to a comparison group of first graders who did not receive the Word Construction intervention. The comparison group was matched on the following variables: Performance on Winter NWF Benchmark Assessment (within 7 words read correct / minute); Performance on Winter ORF Benchmark Assessment (within 5 words read correct / minute); School district (same site, when possible). A series of matched pairs t-tests (with Bonferroni correction for family-wise error) were conducted to establish whether any differences between the treatment and comparison group existed on the Winter ORF Benchmark, and the differences between the treatment and comparison group on the ORF slope in the second half of first grade. Tests were conducted first on all students receiving Word Construction, and next on only students with strong Winter Benchmark performance on the NWF (above target of 52), but weak Winter Benchmark performance on the ORF (below target of 22) labeled strong decoders, non-fluent readers in the table below. Table 51: Matched pairs t-tests of Word Construction treatment vs. comparison groups Answer Options Treatment Mean Comparison Mean P(difference) Number of Pairs All Students in Word Construction ORF Winter Benchmark Performance ORF Growth Slope (words/minute/week) Strong Decoders, Non-fluent Readers Only ORF Winter Benchmark Performance MINNESOTA READING CORPS STATE-WIDE EVALUATION 82

83 ORF Growth Slope (words/minute/week) No significant differences were found on initial ORF performance, however, differences approached significance and comparison students were generally slightly higher than the students who participated in the Word Construction intervention. For winter first grade benchmark, even small differences in performance can be important differences, because scores are typically much lower overall. No significant differences were found on ORF slope, indicating that students in the treatment group did not grow at a faster rate in the second half of first grade than the comparison students. Interestingly, the difference was slightly positive (though non-significant) for the strong decoder, non-fluent reader subset of participants, despite a bias of slightly lower (though non-significant) initial performance on the winter ORF benchmark. Further examination of Word Construction performance should be conducted with a larger sample size, especially for the strong decoder, poor reader group, where samples were too small to draw meaningful conclusions. Family Literacy Pilot Projects During the school year, Minnesota Reading Corps members serving both pre-k and k-3 students participated in a pilot program designed to engage family participation in student literacy. For both age groups, this pilot program involved sending home books or stories for the students to read, together with instructions and suggestions for family members about engaging in shared reading. Family members were asked to sign a response form indicating their level of participation. In the following analysis, results were included if responses fell in valid ranges. Overall, between 32-45% of participating MRC students received at least one book / story at home, and between 71-93% of these families responded with at least one signature indicating participation. MINNESOTA READING CORPS STATE-WIDE EVALUATION 83

84 Table 52: Family Literacy Pilot Program Participation Rates N (Valid N) Pre K (6578) K (8643) Number Books/Stories Sent Home Number of Signatures Returned Number (Percent) Families that Received at Least One Book/Story Number (Percent) Families With At Least One Signature Of Those Who Received At Least One Book/Story Mean (SD) Books/Stories Sent Home for Those Participating Mean (SD) Signatures Received for Those Receiving at Least One Book/Story 20,258 12, (32%) 1499 (71%) 9.53 (6.25) 6.04 (6.21) 85, , (45%) 3626 (93%) (20.03) (52.43) MINNESOTA READING CORPS STATE-WIDE EVALUATION 84

85 Appendix 1 The following table provides a count of unique Pre-K and K-3 Minnesota Reading Corps participating students by grade level as listed in OnCorps regardless of data presence or tutoring time. Student status is also reported. Grade Active Exited Moved Re-Enrolled Referred to other services Grand Total PreK PreK PreK PreK PreK Unknown PreK Total TOTAL MINNESOTA READING CORPS STATE-WIDE EVALUATION 85