Minnesota Reading Corps State-Wide Evaluation

Transcription

1 Minnesota Reading Corps State-Wide Evaluation KERRY BOLLMAN, SSP, NCSP Instructional Services Coordinator, Reading Center Director Saint Croix River Education District BENJAMIN SILBERGLITT, PhD Director, Software Applications Technology and Information Educational Services DAVID PARKER, PhD School Psychologist, Master Coach South Washington County Schools, Minnesota Reading Corps MINNESOTA READING CORPS STATE-WIDE EVALUATION 1

2 Table of Contents BACKGROUND OF THE MINNESOTA READING CORPS 5 EVALUATION DESIGN 6 ASSESSMENT DATA COLLECTION 7 EVALUATION REPORT 9 1. CURRENT IMPACT 9 GROWTH IN DATA REPORTED 12 DEMOGRAPHICS FIDELITY OF ASSESSMENT DATA COLLECTION FIDELITY OF INTERVENTION IMPLEMENTATION 28 GROWTH IN FIDELITY REPORTING STUDENT OUTCOMES 40 PRE-KINDERGARTEN STUDENT PERFORMANCE 40 SUPPLEMENTAL INTERVENTION IN MRC PRE-K PROGRAM 48 CLASSROOM OUTCOMES FOR PRE-K SITES 51 KINDERGARTEN-GRADE 3 STUDENT PERFORMANCE 52 PERFORMANCE ON THE STATEWIDE READING ASSESSMENT SYSTEMS CHANGE IMPACT ON AMERICORPS MEMBERS ACTION RESEARCH: RESULTS OF PILOT STUDIES Appendix 85 MINNESOTA READING CORPS STATE-WIDE EVALUATION 2

3 Tables and Figures TABLE 1: OVERALL MEMBER COUNTS AND SERVICE HOURS, BY REGION AND TOTAL... 9 TABLE 2: PRE-KINDERGARTEN PARTICIPATION TABLE 3: NUMBER OF K-3 STUDENTS BENCHMARKED AS FOLLOW-UP TO PRIOR YEARS' TUTORING.. 11 TABLE 4: NUMBER OF STUDENTS WITH AT LEAST 1 DATA POINT, ANY MEASURE TABLE 5: NUMBER OF STUDENTS WITH AT LEAST 5 WEEKS OF DATA (R-CBM, NWF, OR LSF) TABLE 6: NUMBER OF STUDENTS WITH AT LEAST 10 WEEKS OF DATA (R-CBM, NWF, OR LSF) TABLE 7: NUMBER OF STUDENTS WITH AT LEAST 20 WEEKS OF DATA (R-CBM, NWF, OR LSF) FIGURE 1: GROWTH IN STUDENT DATA REPORTED TABLE 8: MINNESOTA READING CORPS PARTICIPATION: ALL STUDENTS TABLE 9: PARTICIPATION LEVELS IN MRC TUTORING TABLE 10: PARTICIPATION LEVELS IN K-3 MRC TUTORING BY REGION TABLE 11: PRE-KINDERGARTEN - GRADE 3 PARTICIPANT DEMOGRAPHIC DATA TABLE 12: FIDELITY OF ASSESSMENT DATA COLLECTION PROCEDURES TABLE 13: FIDELITY OF INTERVENTION IMPLEMENTATION PROCEDURES FIGURE 2: GROWTH IN FIDELITY REPORTING TABLE 14: KINDERGARTEN PARTICIPANT PERFORMANCE ON IGDIS: FALL BENCHMARK TABLE 15: PRE-KINDERGARTEN PARTICIPANT PERFORMANCE ON IGDIS: WINTER BENCHMARK TABLE 16: PRE-KINDERGARTEN PARTICIPANT PERFORMANCE ON IGDIS: SPRING BENCHMARK FIGURE 3: PERCENTAGE OF STUDENTS ON OR ABOVE, NEAR, AND FAR FROM TARGET: FALL, WINTER, AND SPRING FIGURE 4: NORMATIVE PERFORMANCE OF 4-YEAR-OLDS ON IGDI MEASURES: FALL, WINTER, AND SPRING TABLE 17: NORMATIVE PERFORMANCE OF 4-YEAR-OLDS ON IGDI MEASURES: FALL, WINTER, AND SPRING TABLE 18: FLOOR EFFECT ISSUES WITH PRE-KINDERGARTEN ASSESSMENTS - PERCENTAGE OF STUDENTS NOT COMPLETING SAMPLE IGDIS ITEMS OR SCORING ZERO ON LETTER NAMING/SOUND FLUENCY BY MEASURE AND SEASON (ALL PREK STUDENTS) TABLE 19: PRE-KINDERGARTEN STUDENT GROWTH TABLE 20: IGDI FALL-SPRING GROWTH BY HOURS PER WEEK IN CORE INSTRUCTION: 4-YEAR-OLDS TABLE 21: IGDI FALL-SPRING GROWTH BY HOURS PER WEEK IN CORE INSTRUCTION: ALL PRE- KINDERGARTEN STUDENTS TABLE 22: PARTICIPATION IN PRE-K SUPPLEMENTAL INTERVENTIONS: NUMBER OF STUDENTS AND NUMBER OF MINUTES PER WEEK TABLE 23: IGDI FALL-SPRING GROWTH BY PARTICIPATION IN SUPPLEMENTAL INSTRUCTION: 4-YEAR- OLDS TABLE 24: IGDI FALL-SPRING GROWTH BY PARTICIPATION IN SUPPLEMENTAL INSTRUCTION: ALL PRE-KINDERGARTEN STUDENTS FIGURE 5: CROSS-COHORT PERCENT ABOVE TARGET ON EARLY LITERACY MEASURES TABLE 25: PERCENT OF PRE-K STUDENTS MOVING FROM BELOW TO AT OR ABOVE TARGET TABLE 26: ELLCO PERFORMANCE IN THE FALL AND SPRING TABLE 27: KINDERGARTEN - GRADE 3 PARTICIPANT GROWTH TABLE 28: AVERAGE LINEAR GROWTH RATES, BY REGION FIGURE 7: GRADE 1 NONSENSE WORD FLUENCY GROWTH CURVE ESTIMATES BY REGION FIGURE 8: GRADE 1: ORAL READING FLUENCY GROWTH CURVE ESTIMATES BY REGION MINNESOTA READING CORPS STATE-WIDE EVALUATION 3

4 FIGURE 9: GRADE 2: ORAL READING FLUENCY GROWTH CURVE ESTIMATES BY REGION FIGURE 10: GRADE 3: ORAL READING FLUENCY GROWTH CURVE ESTIMATES BY REGION TABLE 29: KINDERGARTEN - GRADE 3 PERCENTAGE OF STUDENTS ABOVE GROWTH TARGETS BY REGION TABLE 30: AVERAGE TUTORING PARTICIPATION BY GRADE LEVEL TABLE 31: AVERAGE TUTORING PARTICIPATION FOR K-3 BY GRADE LEVEL AND REGION TABLE 32: PERCENTAGE OF STUDENTS WITH 3 WEEKS OF PROGRESS MONITORING DATA WHO EXIT TABLE 33: PERCENTAGE OF STUDENTS WITH 3 WEEKS OF PROGRESS MONITORING DATA WHO EXIT BY REGION FIGURE 11: PERCENTAGE OF STUDENTS SUCCESSFULLY EXITED: TO COMPARISON TABLE 34: PERCENTAGE WHO EXIT: FALL BENCHMARK TIER 2 VS. TIER TABLE 35: PERCENTAGE WHO EXIT: FALL BENCHMARK TIER 2 VS. TIER 3 BY REGION FIGURE 12: PERCENT OF STUDENTS IN EACH TIER WHO EXITED SUCCESSFULLY TABLE 36: PERCENTAGE WHO EXIT WHO ALSO MEET OR EXCEED SPRING BENCHMARK TABLE 37: PERCENTAGE WHO EXIT WHO ALSO MEET OR EXCEED SPRING BENCHMARK BY REGION FIGURE 13: CHANGE IN PERCENTAGE OF STUDENTS EXITING AND MEETING SPRING BENCHMARK TARGETS FIGURE 14: PERCENT OF STUDENTS WITH SUCCESSFUL EXIT OR BENCHMARK SCORE AND SUCCESS ON STATEWIDE READING ASSESSMENTS GRADE 3: COMPARISON OF TO TABLE 39: OUTCOMES OF SECOND GRADE MRC PARTICIPATING STUDENTS ON SPRING GRADE 3 STATEWIDE READING ASSESSMENT TABLE 40: OUTCOMES OF FIRST GRADE MRC PARTICIPATING STUDENTS ON SPRING GRADE 3 STATEWIDE READING ASSESSMENT TABLE 41: OUTCOMES OF KINDERGARTEN MRC PARTICIPATING STUDENTS ON SPRING GRADE 3 STATEWIDE READING ASSESSMENT TABLE 42: TOTAL NUMBER OF STUDENTS MONITORED IN MRC PROGRAMS, WITH THIRD GRADE STATEWIDE READING ASSESSMENT IN TABLE 45: INTERNAL COACH SYSTEMS CHANGE SURVEY RESULTS TABLE 46: MRC MEMBER IMPACT SURVEY RESULTS TABLE 47: K-FOCUS MRC MEMBER IMPACT SURVEY RESULTS TABLE 48: COMPARISON OF PERCENT OF STUDENTS MEETING INDIVIDUAL GROWTH RATES BY PARTICIPATION IN KINDERGARTEN FOCUS MODEL TABLE 49: AVERAGE WEEKLY GROWTH RATE OF ALL MRC PARTICIPATING STUDENTS IN KINDERGARTEN FOCUS MODEL * FIGURE 15: GROWTH CURVE ESTIMATES, BY K-FOCUS PARTICIPATION TABLE 50: THREE-YEAR HISTORY OF TOTAL NUMBERS OF KINDERGARTEN STUDENTS SERVED IN SITES IMPLEMENTING K-FOCUS BY YEAR SITE BEGAN IMPLEMENTING K-FOCUS TABLE 51: MATCHED PAIRS T-TESTS OF WORD CONSTRUCTION TREATMENT VS. COMPARISON GROUPS TABLE 52: FAMILY LITERACY PILOT PROGRAM PARTICIPATION RATES MINNESOTA READING CORPS STATE-WIDE EVALUATION 4

5 BACKGROUND OF THE MINNESOTA READING CORPS Minnesota Reading Corps (MRC) is an AmeriCorps program that provides trained literacy tutors (Members) for children age three to grade three. Some MRC Members work with preschoolers and focus on integrating talking, reading, and writing into all classroom activities. Other Members provide supplemental literacy skills tutoring for children in kindergarten to third grade. Still others recruit, train, and manage community volunteers to expand the capacity of the program. MRC Members and community volunteers are trained in specific researchbased literacy instructional protocols, and supported by expert coaches. Members use reliable, valid assessment tools to monitor student progress on a regular basis, and with help from their expert coaches, use data from assessments to inform tutoring strategies for each student. Use of specific research based instructional techniques and technically adequate assessment tools for decision making make the MRC program both highly unique and highly coveted across the literacy landscape. The vision of the Minnesota Reading Corps is to impact literacy in the state of MN through children, AmeriCorps members and communities as follows: All children in MN, ages 3 to grade 3, who qualify for MRC, will have access to MRC and will meet reading standards by third grade. AmeriCorps members, through the training, development and service opportunity provided by MRC, will pursue education related careers and/or continue to be ambassadors for children's literacy throughout their lives. Schools and community institutions/organizations, through their experiences with MRC, will understand and adopt the MRC methods for increasing literacy; those institutions will, in turn, promote MRC literacy methods to their colleagues. MINNESOTA READING CORPS STATE-WIDE EVALUATION 5

6 EVALUATION DESIGN The evaluation for the Minnesota Reading Corps program year includes six broad questions as listed below. Additions to the current year evaluation include the evaluation of a new intervention designed to support early word reading, and a family literacy pilot program. 1. What is the current impact of the MRC on the state of Minnesota in terms of numbers of students and programs receiving support? 2. Are the data collection tools used with children being implemented with fidelity? 3. Are the interventions used with children being implemented with fidelity? 4. Is the performance of MRC-participating students in terms of their literacy improvement consistent with expectations? 5. Are the organizations with which the MRC is working changing to adopt the practices of the MRC? 6. What is the impact of the MRC experience on the AmeriCorps Members? MINNESOTA READING CORPS STATE-WIDE EVALUATION 6

7 Assessment Data Collection The assessment data listed below was collected during the school year. Minnesota Reading Corps members collect all student assessment data. Internal and master coaches collect fidelity data for assessments and interventions. Internal coaches, site supervisors, and members complete survey data regarding perceptions of systems change and impact. Internal and master coaches collect Pre-K environmental observation data. Student Data for Pre-school Programs: Age 3 on or before Sept 1 st Fall Winter Spring IGDI Rhyming IGDI Picture Naming IGDI Alliteration IGDI Rhyming IGDI Picture Naming IGDI Alliteration IGDI Rhyming IGDI Picture Naming IGDI Alliteration Age 4 on or before Sept 1 st Age 5 on or before Sept 1 st but not enrolled in K IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency MINNESOTA READING CORPS STATE-WIDE EVALUATION 7

8 Student Data for K-3 Programs: Kindergarten Letter Naming Fluency Letter Sound Fluency Grade 1 Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Grade 2 Oral Reading Fluency (3 passages) Grade 3 Oral Reading Fluency (3 passages) Fall Winter Spring Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Fluency Nonsense Word Fluency Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Letter Naming Fluency Letter Sound Fluency Nonsense Word Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Additional Data: Observations of assessment fidelity three times per year Observations of intervention fidelity between three and nine times per year in K-3 programs, monthly in Pre-K programs. Early Language and Literacy Classroom Observation two times per year in Pre-K programs only End of year MRC Member surveys End of year MRC site surveys MINNESOTA READING CORPS STATE-WIDE EVALUATION 8

9 EVALUATION REPORT 1. Current Impact What is the current impact of the MRC on the state of Minnesota in terms of numbers of students and programs receiving support? In the tables below, the number of Minnesota Reading Corps members, by position type, serving during the school year, who collected data for students during one or more benchmark assessment windows and submitted the data for evaluation is recorded. In addition, the number of students receiving MRC services for whom data are recorded is reported. Numbers of participating students are compiled according to the following criteria: Number of preschool students with data from 1 benchmark window Number of preschool students with data from 3 benchmark windows Number of K-3 students benchmarked as a follow up from previous year s tutoring Number of K-3 students with at least 1 week of data on at least 1 measure Number of K-3 students with at least 5 weeks of data on at least 1 measure Number of K-3 students with at least 10 weeks of data on at least 1 measure Number of K-3 students with at least 20 weeks of data on at least 1 measure The following table provides a count of Minnesota Reading Corps participating Members who served during the school year in Pre-K and K-3 programs. The total number of hours served is also displayed. Overall counts, regional counts, and counts-by-position are provided. Table 1: Overall Member Counts and Service Hours, by Region and Total Region Position Number of Members Number of Hours Served Central K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps MINNESOTA READING CORPS STATE-WIDE EVALUATION 9

10 Volunteer Coordinator Metro K-3 Literacy Tutor Family Literacy Member Kindergarten-Focused Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Northcentral K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Northeast K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Northwest K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Southeast K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Southwest K-3 Literacy Tutor Preschool Literacy Tutor-Community Corps Preschool Literacy Tutor-Professional Corps Volunteer Coordinator Grand Total MINNESOTA READING CORPS STATE-WIDE EVALUATION 10

11 Table 2: Pre-Kindergarten Participation Region Number of Members Number of Students with 1 or More Assessments in 1 Window Number of Students with 1 or more Assessments in 3 Windows Central Metro 118 3, North Central North East North West South East South West Total 207 5,953 4,750 Table 3: Number of K-3 Students benchmarked as follow-up to prior years' tutoring Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Table 4: Number of Students with at least 1 data point, any measure Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL MINNESOTA READING CORPS STATE-WIDE EVALUATION 11

12 Table 5: Number of Students with at least 5 weeks of data (R-CBM, NWF, or LSF) Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Table 6: Number of Students with at least 10 weeks of data (R-CBM, NWF, or LSF) Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Table 7: Number of Students with at least 20 weeks of data (R-CBM, NWF, or LSF) Region Gr K Gr 1 Gr 2 Gr 3 TOTAL North East North Central Metro North West South East South West Central TOTAL Growth in Data Reported The figure below compares total numbers of students served with data recorded for the MRC program during the , , , and school years. Significant growth in total amount of student data recorded is noted across pre-k and k-3 programs. MINNESOTA READING CORPS STATE-WIDE EVALUATION 12

13 Figure 1: Growth in Student Data Reported 14,000 12,000 10,000 8,000 6,000 4, ,000 0 # Pre-K Students with Data in 1 or More Benchmark Windows # Pre-K Students with Data in 3 Benchmark Windows # K-3 Students with 5 or More Data Points # K-3 Students with 20 or More Data Points # K-3 Students Benchmarked as Follow up to Last Year's Tutoring In order to more fully acknowledge the impact on student participation in the Minnesota Reading Corps, the following tables summarize the total number of students identified as having participated in the MRC program, regardless of data presence or tutoring time: Table 8: Minnesota Reading Corps Participation: All Students Grade Active Exited Moved Re-Enrolled Referred to other services Grand Total PreK PreK PreK PreK PreK Age Unknown MINNESOTA READING CORPS STATE-WIDE EVALUATION 13

14 PreK Total K TOTAL Further analysis was completed to identify the average participation dosage for students in both the Pre-K and the K-3 MRC programs. Average number of weeks, sessions per week, and minutes per week of tutoring service provided is summarized below, first in total across the state, and then broken down for the K-3 program by region. Increases in both number of sessions per week and minutes per week is noted in K-3 for the school year relative to Table 9: Participation Levels in MRC Tutoring Number of Students Average Sessions per Week Average Total Minutes Average Minutes per Week Average Average Total Grade Weeks Sessions Minutes PreK PreK PreK PreK PreK Unknown 81 N/A N/A N/A N/A N/A N/A PreK Total K TOTAL Table 10: Participation Levels in K-3 MRC Tutoring by Region Grade North East Number of Students Average Weeks Average Sessions Average Sessions per Week Average Total Minutes Average Minutes per Week Total Minutes K MINNESOTA READING CORPS STATE-WIDE EVALUATION 14

15 TOTAL North Central K TOTAL Metro K TOTAL North West K TOTAL South East K TOTAL South West K TOTAL Central K TOTAL MINNESOTA READING CORPS STATE-WIDE EVALUATION 15

16 Demographics In order to more fully describe the population of children served by the Minnesota Reading Corps program, data regarding gender, ethnicity, and English Learner Status were collected. These demographic data for participating students were entered into the MRC OnCorps Database by members. Table 11: Pre-Kindergarten - Grade 3 Participant Demographic Data Gender Ethnicity English Learner Status Pre-K 51.37% Male 48.57% Female 0.07% Not Reported 2.35% American Indian / Alaska Native 19.57% African American 8.45% Asian 12.54% Hispanic / Latino 0.50% Pacific Islander 2.65% Unknown / No Response 6.43% Multiple 46.68% White 0.83% Not Reported 75.24% English as Primary Language 24.11% ELL 0.65% Not Reported K % Male 46.02% Female 4.65% Not Reported 2.55% American Indian / Alaska Native 18.51% African American 8.24% Asian 0.02% Filipino 10.10% Hispanic / Latino 0.29% Pacific Islander 2.27% Unknown/No Response 2.58% Multiple 48.85% White 6.59% Not Reported 66.71% English as Primary Language 14.88% ELL 18.41% Not Reported MINNESOTA READING CORPS STATE-WIDE EVALUATION 16

17 2. Fidelity of Assessment Data Collection What is the Research Base for MRC Program Assessments? As listed in the above, the assessment tools used to determine literacy progress of MRC-participating students include the following measures: Picture Naming Fluency Alliteration Fluency Rhyming Fluency Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Oral Reading Fluency These tools were selected for use in the MRC because of their well-established statistical reliability and validity for screening and progress monitoring purposes. Picture Naming, Alliteration, and Rhyming measures were developed through the University of Minnesota, and are commonly referred to as Individual Growth and Development Indicators (IGDIs) of literacy. Letter Naming, Letter Sounds, and Nonsense Words are measures of early literacy skills thoroughly researched by many groups, but most famously packaged by two assessment programs: DIBELS and AIMSweb. Oral Reading Fluency provides an assessment of connected text reading. Early and ongoing research on this measure has also been conducted at the University of Minnesota. All these measures fit under the umbrella of Curriculum-Based Measurement (CBM), and are fluency based assessments, meaning that students respond to an unlimited number of items within a fixed amount of time, and the number of correct responses is counted. The information that follows summarizes empirical findings related to the statistical reliability and validity of the measures used in the Minnesota Reading Corps. Picture Naming Fluency: r=.44 to.78 1 month alternate form reliability r=.67 test-retest 3-week reliability r=.47 to.75 with PPVT-3 and.63 to.81 with PLS-3 MINNESOTA READING CORPS STATE-WIDE EVALUATION 17

18 r=.32 to.37 with DIBELS Letter Naming Fluency and.44 to.49 with DIBELS Initial Sound Fluency r=.41 (longitudinal) and.60 (cross sectional) between scores and chronological age, with correlations of.63,.32, and.48 for typically developing, HeadStart, and ECSE populations respectively Sources: McConnell, S.R., Priest, J.S., Davis, S.D., & McEvoy, M.A. (2002). Best Practices in Measuring Growth and Development for Preschool Children, In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology (4 th ed., Vol. 2, ). Washington DC: National Association of School Psychologists. Missall, K.N., & McConnell, S.R. (April, 2004). Psychometric Characteristics for Individual Growth and Development Indicators: Picture Naming, Rhyming, and Alliteration (Technical Report). Minneapolis, MN: University of Minnesota. Accessed online at July 27, Missall, K.N. et. al. (2007). Examination of Predictive Validity of Preschool Early Literacy Skills. School Psychology Review 36 (3) Missall, K. N., McConnell, S. R., & Cadigan, K. (2006). Early literacy development: Skill growth and relations between classroom variables for preschool children. Journal of Early Intervention, 29, Phaneuf, R. L., & Silberglitt, B. (2003). Tracking preschoolers' language and preliteracy development using a general outcome measurement system. Topics in Early Childhood Special Education, 23, Priest, J. S., McConnell, S. R., Walker, D., Carta, J. J., Kaminski, R. A., McEvoy, M. A., Good, R., Greenwood, C. R., & Shinn, M. R. (2001). General growth outcomes for young children: Developing a foundation for continuous progress measurement. Journal of Early Intervention, 24, Wackerle, Alisha K. (2007). Test review: Selection of general growth outcomes for children between birth and age eight. Assessment for Effective Intervention. 33(1), MINNESOTA READING CORPS STATE-WIDE EVALUATION 18

19 Alliteration: r=.46 to.80 test-retest reliability over 3 weeks r=.40 to.57 with PPVT-3 r=.34 to.55 with Clay s Concepts about Print r=.75 to.79 with TOPA r=.39 to.71 with DIBELS Letter Naming Fluency r=.61 with chronological age Sources: McConnell, S.R., Priest, J.S., Davis, S.D., & McEvoy, M.A. (2002). Best Practices in Measuring Growth and Development for Preschool Children, In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology (4 th ed., Vol. 2, ). Washington DC: National Association of School Psychologists. Missall, K.N., & McConnell, S.R. (April, 2004). Psychometric Characteristics for Individual Growth and Development Indicators: Picture Naming, Rhyming, and Alliteration (Technical Report). Minneapolis, MN: University of Minnesota. Accessed online at July 27, Missall, K.N. et. al. (2007). Examination of Predictive Validity of Preschool Early Literacy Skills. School Psychology Review 36 (3) Missall, K. N., McConnell, S. R., & Cadigan, K. (2006). Early literacy development: Skill growth and relations between classroom variables for preschool children. Journal of Early Intervention, 29, Phaneuf, R. L., & Silberglitt, B. (2003). Tracking preschoolers' language and preliteracy development using a general outcome measurement system. Topics in Early Childhood Special Education, 23, Priest, J. S., McConnell, S. R., Walker, D., Carta, J. J., Kaminski, R. A., McEvoy, M. A., Good, R., Greenwood, C. R., & Shinn, M. R. (2001). General growth outcomes for young children: Developing a foundation for continuous progress measurement. Journal of Early Intervention, 24, VanDerHeyden, A.M., Snyder, P.A., Broussard, C., & Ramsdell, K. (2007). Measuring Response to Early Literacy Intervention with Preschoolers at Risk. Topics in Early Childhood Special Education 27(4), MINNESOTA READING CORPS STATE-WIDE EVALUATION 19

20 Wackerle, Alisha K. (2007). Test review: Selection of general growth outcomes for children between birth and age eight. Assessment for Effective Intervention. 33(1), Rhyming: r=.83 to.89 test-retest reliability over 3 weeks r=.56 to.62 with PPVT-3 r=.54 to.64 with Clay s Concepts about Print r=.44 to.62 with TOPA r=.44 to.63 with IGDI Picture Naming and.43 with IGDI Alliteration r=.48 to.59 with DIBELS Letter Naming Fluency r=.44 to.68 with DIBELS Initial Sound Fluency r=.46 with chronological age Sources: McConnell, S.R., Priest, J.S., Davis, S.D., & McEvoy, M.A. (2002). Best Practices in Measuring Growth and Development for Preschool Children, In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology (4 th ed., Vol. 2, ). Washington DC: National Association of School Psychologists. Missall, K.N., & McConnell, S.R. (April, 2004). Psychometric Characteristics for Individual Growth and Development Indicators: Picture Naming, Rhyming, and Alliteration (Technical Report). Minneapolis, MN: University of Minnesota. Accessed online at July 27, Missall, K.N. et. al. (2007). Examination of Predictive Validity of Preschool Early Literacy Skills. School Psychology Review 36 (3) Missall, K. N., McConnell, S. R., & Cadigan, K. (2006). Early literacy development: Skill growth and relations between classroom variables for preschool children. Journal of Early Intervention, 29, Phaneuf, R. L., & Silberglitt, B. (2003). Tracking preschoolers' language and preliteracy development using a general outcome measurement system. Topics in Early Childhood Special Education, 23, MINNESOTA READING CORPS STATE-WIDE EVALUATION 20

21 Priest, J. S., McConnell, S. R., Walker, D., Carta, J. J., Kaminski, R. A., McEvoy, M. A., Good, R., Greenwood, C. R., & Shinn, M. R. (2001). General growth outcomes for young children: Developing a foundation for continuous progress measurement. Journal of Early Intervention, 24, VanDerHeyden, A.M., Snyder, P.A., Broussard, C., & Ramsdell, K. (2007). Measuring Response to Early Literacy Intervention with Preschoolers at Risk. Topics in Early Childhood Special Education 27(4), Wackerle, Alisha K. (2007). Test review: Selection of general growth outcomes for children between birth and age eight. Assessment for Effective Intervention. 33(1), Letter Naming Fluency: r=.94 inter rater reliability r=.90 2 week test retest reliability r=.88 1 month alternate reliability r=.93 alternate forms reliability r=.70 with WJ-R Readiness Cluster r=.70 with WJ Psychoeducational Battery r=.53 to.58 with CTOPP Composite Predictive r=.65 with WJ Total Reading Cluster Predictive r=.71 with R-CBM ELL Predictive r =.67 with a composite of DIBELS NWF and R-CBM Sources: Assessment Committee Report for Reading First. (2002). Analysis of Reading Assessment Measures. Retrieved February 21, 2007, from Good, R.H., Kaminski, R.A., Shinn, M. Bratten, J., Shinn, M., & Laimon, L. (in preparation). Technical Adequacy and Decision Making Utility of DIBELS (Technical Report). Eugene, OR: University of Oregon Good, R.H. III., Kaminski, R.A., Simmons, D., Kame enui, E.J. (2001). Using Dynamic Indicators of Basic Early Literacy Skills (DIBELS) in an outcomes-driven model: Steps to reading outcomes. Unpublished manuscript, University of Oregon at Eugene. MINNESOTA READING CORPS STATE-WIDE EVALUATION 21

22 Elliot, J., Lee, S.W., Tolefson, N. (2001). A Reliability and Validity Study of the Dynamic Indicators of Basic Early Literacy Skills Modified. School Psychology Review, 30 (1), Haager, D. & Gersten, R (April, 2004). Predictive Validity of DIBELS for English Learners in Urban Schools. DIBELS Summit conference presentation, Albuquerque, NM Hintz, J.M., Ryan, A.L., & Stoner, G. (2003). Concurrent Validity and Diagnostic Accuracy of DIBELS and the CTOPP. School Psychology Review Kaminski, R.A. & Good, R.H. (1996). Toward a Technology for Assessment Basic Early Literacy Skills. School Psychology Review, 25, Rouse, H., Fantauzzo, J.W. (2006). Validity of the Dynamic Indicators of Basic Early Literacy Skills as an Indicator of Early Literacy for Urban Kindergarten Children. School Psychology Review 35 (3) Letter Sound Fluency: r=.83 2-week test-retest reliability r=.80 alternate form reliability r=.79 with Letter Naming Fluency Predictive r=.72 with R-CBM Sources: Elliott, J., Lee, S.W., & Tollefson, N. (2001). A Reliability and Validity Study of the Dynamic Indicators of Basic Early Literacy Skills Modified. School Psychology Review, 30 (1), Fuchs, L., Fuchs D. (2004). Determining Adequate Yearly Progress from Kindergarten through Grade 6 with Curriculum Based Measurement. Assessment for Effective Intervention 29 (4) Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School-Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, MINNESOTA READING CORPS STATE-WIDE EVALUATION 22

23 Scott, S.A., Sheppard, J., Davidson, M.M., & Browning, M.M. (2001). Prediction of First Graders Growth in Oral Reading Fluency Using Kindergarten Letter Naming Fluency. Journal of School Psychology, 39(3), Ritchey, K.D (2008). Assessing Letter Sound Knowledge: A Comparison of Letter Sound Fluency and Nonsense Word Fluency. Exceptional Children 74 (4) Nonsense Word Fluency: r=.83 one month alternate form reliability r=.36 to.59 with WJ-R Readiness Cluster Predictive r=.82 with Spring R-CBM in Spring of grade 1 Predictive r =.65 with oral reading and.54 with maze in grade 3 Ell Predictive r=.63 with a composite of DIBELS NWF and R-CBM Sources: Burke, M. D., Hagan-Burke, S. (2007). Concurrent criterion-related validity of early literacy indicators for middle of first grade. Assessment for Effective Intervention. 32(2), Good, R.H., Kaminski, R.A., Shinn, M. Bratten, J., Shinn, M., & Laimon, L. (in preparation). Technical Adequacy and Decision Making Utility of DIBELS (Technical Report). Eugene, OR: University of Oregon Good, R.H., Kaminski, R.A., Simmons, D., & Kame-enui, E.J. (2001). Using DIBELS in an Outcomes Driven Model: Steps to Reading Outcomes. Unpublished manuscript, University of Oregon, Eugene. Haager, D. & Gersten, R (April, 2004). Predictive Validity of DIBELS for English Learners in Urban Schools. DIBELS Summit conference presentation, Albuquerque, NM Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School-Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, MINNESOTA READING CORPS STATE-WIDE EVALUATION 23

24 Kaminski, R.A. & God, R.H. (1996). Toward a Technology for Assessment Basic Early Literacy Skills. School Psychology Review, 25, Ritchey, K.D (2008). Assessing Letter Sound Knowledge: A Comparison of Letter Sound Fluency and Nonsense Word Fluency. Exceptional Children 74 (4) Rouse, H., Fantauzzo, J.W. (2006). Validity of the Dynamic Indicators of Basic Early Literacy Skills as an Indicator of Early Literacy for Urban Kindergarten Children. School Psychology Review 35 (3) Vanderwood, M.., Linklater, D., Healy, K. (2008). Predictive Accuracy of Nonsense Word Fluency for English Language Learners. School Psychology Review 37 (1) Oral Reading Fluency: r=.92 to.97 test retest reliability r=.89 to.94 alternate form reliability r=.82 to.86 with Gates-MacGinite Reading Test r=.83 to Iowa Test of Basic Skills r =.88 to Stanford Achievement Test r=.73 to.80 to Colorado Student Assessment Program r=.67 to Michigan Student Assessment Program r=.73 to North Carolina Student Assessment Program r=74 to Arizona Student Assessment Program r=.61 to.65 to Ohio Proficiency Test, Reading Portion r=.58 to.82 with Oregon Student Assessment Program (SAT 10) Sources: Barger, J. (2003). Comparing the DIBELS Oral Reading Fluency indicator and the North Carolina end of grade reading assessment (Technical Report). Ashville, NC: North Carolina Teacher Academy. Baker S. et. al,. (2008). Reading Fluency as a Predictor of Reading Proficiency in Low-Performing, High-Poverty Schools. School Psychology Review 37 (1) MINNESOTA READING CORPS STATE-WIDE EVALUATION 24

25 Burke, M. D., Hagan-Burke, S. (2007). Concurrent criterion-related validity of early literacy indicators for middle of first grade. Assessment for Effective Intervention. 32(2), Deno, S. L., Mirkin, P. K., & Chiang, B. (1982). Identifying valid measures of reading. Exceptional Children, Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School-Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, Hintze, J.M, et al (2002). Oral Reading Fluency and Prediction of Reading Comprehension in African American and Caucasian Elementary School Children. School Psychology Review, 31 (4) Hintze, J. M. & Silberglitt, B. (in press). A Longitudinal Examination of the Diagnostic Accuracy and Predictive Validity of R-CBM and High-Stakes Testing. School Psychology Review. Marston, D., Fuchs, L., & Deno, S. (1987). Measuring pupil progress: a comparison of standardized achievement tests and curriculum-related measures. Diagnostique, 11, Marston, D. (1989). Curriculum-based measurement: What is it and why do it? In M. R. Shinn (Ed.), Curriculum-based measurement: Assessing special children (pp ). New York: Guilford Press. McGlinchey, M. T., & Hixson, M. D. (2004). Contemporary research on curriculumbased measurement: Using curriculum-based measurement to predict performance on state assessments in reading. School Psychology Review, 33(2), Schilling, S. G., Carlisle, J. F., Scott, S. E., & Zeng, J. (2007). Are fluency measures accurate predictors of reading achievement? The Elementary School Journal, 107(5), Silberglitt, B. & Hintze, J. M. (in press). Formative Assessment Using Oral Reading Fluency Cut Scores to Track Progress Toward Success on State-Mandated MINNESOTA READING CORPS STATE-WIDE EVALUATION 25

26 Achievement Tests: A Comparison of Methods. Journal of Psychoeducational Assessment. Shaw, R., & Shaw, D. (2002). DIBELS Oral Reading Fluency-Based Indicators of the third-grade reading skills for Colorado State Assessment Program (CSAP) (Technical Report). Eugene, OR: University of Oregon. Shinn, M., Good, R., Knutson, N., Tilly, W., & Collins, A. (1992). Curriculum-based measurement of oral reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, Stage, S. A., & Jacobsen, M. D. (2001). Predicting student success on a statemandated performance-based assessment using oral reading fluency. School Psychology Review, 30(3), Tindal, G., Germann, G., & Deno, S. (1983). Descriptive research on the Pine County Norms: A compilation of findings (Research Report No. 132). Minneapolis, MN: University of Minnesota Institute for Research on Learning Disabilities. Vander Meer, C. D., Lentz, F. E., & Stollar, S. (2005). The relationship between oral reading fluency and Ohio proficiency testing in reading (Technical Report). Eugene, OR: University of Oregon. Wilson, J. (2005). The relationship of Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Oral Reading Fluency to performance on Arizona Instrument to Measure Standards (AIMS). Tempe, AZ: Tempe School District No. 3. Are the data collection tools used with children being implemented with fidelity? Analysis of fidelity with which student assessment data are conducted is a critical initial aspect to the evaluation of the MRC program so that results from evaluation of these data may be reported with confidence. In order to accomplish this, a series of Accuracy of Implementation Rating Scales (AIRS) have been compiled from each Minnesota Reading Corps (MRC) site. MRC Internal Coaches and Master Coaches were trained in August 2011 to administer and score assessment measures, and to conduct observations of Reading Corps members as they administer and score these measures. The AIRS MINNESOTA READING CORPS STATE-WIDE EVALUATION 26

27 are structured observational protocols that provide an opportunity for observers to certify that each aspect of a standardized administration for each assessment measure has been fully conducted. Master and/or Internal Coaches completed a minimum of 1 AIRS for each Reading Corps Member for each type of assessment the member conducted at least 3 times each year, near the benchmark data collection periods. The table below documents the total number of AIRS assessments compiled, the total number of AIRS assessments completely filled out on forms, and percent fidelity documented of those completely filled out on forms for each measure. In addition to reporting these data in aggregate format to document high fidelity of assessment procedures across the state, this observation system also provided Members with immediate feedback regarding the quality of their own assessment skills, and an opportunity to receive clarification or re-training as needed in a timely manner. MINNESOTA READING CORPS STATE-WIDE EVALUATION 27

28 Table 12: Fidelity of Assessment Data Collection Procedures Measure Rhyming Alliteration Picture Naming PK Letter Naming PK Letter Sounds K-3 Letter Naming K-3 Letter Sounds Nonsense Words Oral Reading Fluency Total AIRS Collected Total Complete AIRS Fidelity Range Reported 0% - 100% 11% - 100% 0% - 100% 36%- 100% 0% - 100% 46% - 100% 38% - 100% 0% - 100% 15% - 100% Median % Fidelity Reported Mean % Fidelity Reported Standard Deviation 100% 100% 100% 100% 100% 100% 100% 100% 100% 99% 99% 99% 99% 99% 98% 98% 98% 98% Fidelity of Intervention Implementation Are the interventions used with children being implemented with fidelity? As with the assessment tools, analysis of the level of fidelity with which the student intervention protocols are followed is a critical initial aspect to the evaluation of the MRC program so that results of student growth analysis may be attributed to accurate implementation of intervention scripts. In order to accomplish this, a series of intervention integrity observations have been MINNESOTA READING CORPS STATE-WIDE EVALUATION 28

29 compiled from each MRC site. MRC internal and master coaches were trained in August 2011 to evaluate implementation integrity for each of the MRC interventions. The integrity checklists provide an opportunity for observers to certify that each aspect of a standardized administration for each intervention has been fully conducted. Master and/ or internal coaches completed a minimum of 1 intervention integrity checklist for each MRC member during each monthly visit, for a possible total of 9 checklists per member per year. The interested reader is referred to Ehrhardt, Barnett, Lentz, Stollar, & Reifin, (1996) for a description of how to use scripts to improve intervention integrity. The first 10 intervention protocols listed are available to all students involved in the K-3 MRC program. The next 7 are offered in a sub-set of K-3 MRC sites whose Members have been nominated and provided additional training on implementation of small group intervention protocols. The final group of intervention protocols are those offered in the Pre-K MRC program. The table below documents the number of integrity checklists compiled and percent fidelity documented for each intervention: Table 13: Fidelity of Intervention Implementation Procedures Intervention Total Fidelity Median Mean Standard Complete Range Percent Percent Deviation Fidelity Reported Fidelity Fidelity Checks Reported Reported Duet Reading % - 100% 100% 96% 0.08 Newscaster Reading % - 100% 100% 96% 0.08 Repeated Reading with % - 100% 94% 87% 0.15 Comprehension Strategy Practice Stop / Go % - 100% 100% 97% 0.66 Pencil Tap % - 100% 100% 94% 0.11 Great Leaps % - 100% 96% 92% 0.13 Letter Sound Identification % - 100% 100% 95% 0.11 Word Blending % - 100% 100% 96% 0.10 Phoneme Blending % - 100% 100% 96% 0.09 Phoneme Segmenting % - 100% 100% 97% 0.07 Word Construction %-100% 95% 91% 0.15 Partner Reading (Paired) 44 78% - 100% 100% 97% 0.06 Phoneme Blending (Paired) 88 36% - 100% 100% 95% 0.11 Phoneme Segmenting (Paired) 62 46% - 100% 100% 94% 0.12 Letter Sound Identification % - 100% 100% 94% 0.11 (Paired) Blending Words (Paired) 64 64% - 100% 100% 95% 0.09 Repeated Read Aloud (K Focus) % - 100% 89% 85% 0.15 Pre-K Oral Language % - 100% 100% 97% 0.07 MINNESOTA READING CORPS STATE-WIDE EVALUATION 29

30 Pre-K Phonological Awareness % - 100% 100% 96% 0.06 Pre-K Phonological Awareness %-100% 100% 97% 0.08 Pre-K Phonological Awareness %-100% 100% 97% 0.05 Pre-K Phonological Awareness %-100% 100% 90% 0.13 Pre-K Repeated Read Aloud % - 100% 91% 83% 0.19 Pre-K Repeated Read Aloud %-100% 91% 86% 0.19 Pre-K Repeated Read Aloud %-100% 91% 74% 0.34 Pre-K Repeated Read Aloud %-100% 100% 95% 0.10 Pre-K Sign In % - 100% 100% 95% 0.10 Pre-K Visual Discrimination % - 100% 100% 94% 0.09 Pre-K Visual Discrimination %-100% 100% 94% 0.08 Pre-K Visual Discrimination %-100% 100% 96% 0.07 Pre-K Visual Discrimination %-100% 100% 97% 0.04 Pre-K Visual Discrimination %-100% 100% 98% 0.07 Pre-K Visual Discrimination %-100% 100% 100% 0.13 Growth in Fidelity Reporting The chart below compares total numbers of fidelity observations recorded for the MRC program during the , , , and school years for interventions offered across more than one year. Significant growth in total complete observations recorded is noted, mirroring growth in the program. MINNESOTA READING CORPS STATE-WIDE EVALUATION 30

31 Figure 2: Growth in Fidelity Reporting What is the Research Base for MRC Program Interventions? The interventions identified for use in the MRC program are each designed to provide additional practice that is supplemental to the core reading instructional program offered by the local school site. This practice is provided with the intention of building automaticity and fluency of important reading skills that have already been introduced by local classroom teachers. It is important to note that MRC participation is in addition to, not in replacement of, a comprehensive core reading instructional program, and that the MRC program should in no way be viewed as a substitute for high quality core instruction. MRC provides important additional guided practice time in reading for students who need this support. For further discussion regarding the benefit of supplemental support to students at risk for reading failure, see Harn (2008). For a discussion of benefit of well-matched interventions, see Wagner et al (2006). The chosen interventions share a common theme in focus on building fluency for basic reading skills such as phonemic awareness, letter sound knowledge, MINNESOTA READING CORPS STATE-WIDE EVALUATION 31

32 decoding skill, and sight word recognition. Fluency is interpreted in this program as incorporating rate, accuracy, and prosody, or expression. Richard Allington, Former president of the International Reading Association writes: "There are a substantial number of rigorously designed research studies demonstrating (1) that fluency can be developed, most readily through a variety of techniques that involve rereading texts and (2) that fostering fluency has reliable positive impacts on comprehension and performance. Thus when fluency is an instructional goal, as it should be for struggling readers, we have a wealth of research to guide our instructional planning. (Allington, 2001) For further discussion on the relationship between oral reading fluency and comprehension skills, the interested reader is referred to Tenenbaum & Wolking (1989) and Sindelar, Monda, & O Shea, (1990). A unique feature of MRC is the consistent use of research-based intervention protocols with participating students to provide this additional support. MRC members select from a menu of research-based supplemental reading interventions for use with participating MRC students as listed below. For each intervention protocol, a description of the research base, and/or sources of empirical evidence of intervention effectiveness are listed. Repeated Reading with Comprehension Strategy Practice Nelson, J. S., Alber, S. R., & Grody, A. (2004). Effects of systematic error correction and repeated readings on reading accuracy and proficiency of second graders with disabilities. Education and Treatment of Children, 27, Staubitz, J. E., Cartledge, G., Yurick, A., & Lo, Y. (2004). Repeated reading for students with emotional or behavioral disorders: Peerand trainermediated instruction. Behavior Disorders, 31, Therrien, W. J. (2004). Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education, 25, MINNESOTA READING CORPS STATE-WIDE EVALUATION 32