Colorado Reading Corps Evaluation 2012-2013

Evaluation 2012-2013

TABLE OF CONTENTS DESCRIPTION OF COLORADO READING CORPS... 3 INTRODUCTION TO THE EVALUATION... 6 EVALUATION RESULTS... 7 What is the scope of the Reading Corps program in the state of Colorado?...7 What are the primary literacy outcomes of the Colorado Reading Corps?...10 How well is the Reading Corps program implemented in Colorado?...19 What do key stakeholders and implementers report regarding how participation in Reading Corps has affected their beliefs, opinions, and/or professional practices?...21 APPENDIX A ADDITIONAL ASSESSMENT INFORMATION...26 APPENDIX B ADDITIONAL INTERVENTION INFORMATION...33 Prepared by ServeMinnesota staff. Send questions to david@serveminnesota.org.

DESCRIPTION OF COLORADO READING CORPS Colorado Reading Corps (CRC) is an AmeriCorps program that provides schools with literacy tutors to support reading development for students in kindergarten through grade 3. Reading Corps tutors are trained to implement evidencebased literacy instruction and assessment protocols. Tutors are also supported by a multi-level coaching model that includes site-based ( Internal ) and external ( Master ) coaches. Effective Service-Delivery The CRC model aligns with response-to-intervention (RTI), which is a framework for delivering educational services effectively and efficiently (Burns, Deno, & Jimerson, 2007 1 ). The key aspects of that alignment include the following: Clear literacy targets at each age level from K through grade 3 Benchmark assessment three times a year to identify students eligible for oneon-one interventions Scientifically based interventions Frequent progress monitoring during intervention delivery High quality training in program procedures, coaching, and observations to support fidelity of implementation In the RtI framework, data play the key roles of screening student eligibility for additional services and monitoring student progress towards achieving academic goals (i.e., benchmarks). The CRC screens students for program eligibility three times a year (i.e., fall, winter, spring) using empirically-derived grade- and content-specific performance benchmarks. Eligible students (defined as students below target scores) are determined potential candidates 1 Burns, K.M., Deno, S.L., & Jimerson, S.R. (2007). Toward a unified Response-to-Intervention model. In S.R. Jimerson, M.K. Burns, & A. VanDerHeyden (Eds.), Handbook of Response to Intervention (pp. 428-440). New York: Springer. COLORADO READING CORPS EVALUATION 2012-2013 3

to receive Reading Corps support, which is often referred to as Tier 2 within an RTI framework. The CRC program is focused on the Big Five Ideas in Literacy as identified by the National Reading Panel, including phonological awareness, phonics, fluency, vocabulary, and comprehension. Full-time tutors work with approximately 15-18 K-3 students daily for 20 minutes each. The tutoring interventions align with the Big Five literacy targets and are supplemental to the core reading instruction provided at each school. The goal of the tutoring is to raise individual students literacy levels so that they are on track to meet or exceed the next program-specified literacy benchmark. Coaching and Support As mentioned above the CRC provides multiple layers of supervision to ensure integrity of program implementation. Site-specific Internal Coaches, who are typically staff literacy specialists, teachers, or curriculum directors, serve as immediate on-site supervisors, mentors, and advocates for members. The Internal Coach s role is to monitor members and provide guidance in the implementation of CRC s literacy rich schedule (PreK only), assessments and interventions. As the front-line supervisor, the Internal Coach is a critical component of the supervisory structure. The external, or Master Coach, is a contracted literacy experts who provides site staff (i.e., Internal Coaches and AmeriCorps members) with expert consultation on literacy instruction and ensures implementation integrity of CRC program elements. In addition to these two coaching layers, a third layer consisting of AmeriCorps program support helps ensure a successful year of AmeriCorps service. Program staff are CRC employees who provide administrative oversight for program implementation to sites participating in CRC. Training Prior to each school year beginning, CRC hosts a four day Summer Institute to train returning and new Master Coaches, Internal Coaches, and AmeriCorps members. This intensive, information-filled conference provides foundational training in the research-based literacy interventions employed by CRC. During several sessions at the Summer Institute, members learn the skills, knowledge, and tools needed to serve as literacy interventionists. Members are provided with detailed Literacy Handbooks as well as online resources that mirror and COLORADO READING CORPS EVALUATION 2012-2013 4

supplement the contents of the Handbooks (e.g., videos of model interventions and best practices). Both the Handbooks and website are intended to provide members with just-in-time support, and opportunities for continued professional development and skill refinement. COLORADO READING CORPS EVALUATION 2012-2013 5

INTRODUCTION TO THE EVALUATION The Colorado Reading Corps evaluation addresses four broad questions with data collected during the 2012-2013 school year. The evaluation report is organized around each of these questions. Prior to presenting the results for each, a brief description of the data sources and collection methods is included. Following the results is an interpretation. 1. What is the scope of the Reading Corps program in the state of Colorado? 2. What are the primary literacy outcomes of the Colorado Reading Corps? 3. How well is the Reading Corps program implemented in Colorado? 4. What do key stakeholders and implementers report regarding how participation in Reading Corps has affected their beliefs, opinions, and/or professional practices? COLORADO READING CORPS EVALUATION 2012-2013 6

EVALUATION RESULTS 1. What is the scope of the Reading Corps program in the state of Colorado? Data Source and Collection Methods: Data for evaluating the scope of the Reading Corps program in Colorado come from records kept and maintained by the Reading Corps tutors. Each tutor recorded basic student information and data on their tutoring activities. Table 1: Presents summary of the people and schools involved Table 2: Presents summary of student participation Table 3: Presents average tutoring participation levels for students Table 4: Presents demographic information for students Table 1: Summary of Tutors, Internal Coaches, Master Coaches, and Schools Schools (Districts) Internal Coaches Master Coaches Number of Tutors* 12 (1) 12 1 19 *NOTE: Includes only tutors who entered student literacy data. COLORADO READING CORPS EVALUATION 2012-2013 7

Table 2: Kindergarten through Grade 3 Student Participation Total Students* Served by Colorado Reading Corps Grade Level Total Number Served* Kindergarten 77 1 st Grade 128 2 nd Grade 133 3 rd Grade 134 TOTAL 472 *NOTE: Includes only students with at least one data point entered progress monitoring system. Table 3: Participation Levels in Reading Corps Tutoring Grade Mean Weeks per Student Mean Sessions per Student Mean Minutes per Week K 11.74 39.86 63.53 1 20.30 75.70 70.51 2 23.17 88.19 74.17 3 21.46 78.65 70.37 TOTAL 20.05 74.25 70.38 Table 4: Kindergarten - Grade 3 Participant Demographic Data (Reported by Reading Corps Tutors) Gender Ethnicity Language 52.54% Male 47.46% Female 1.48% American Indian or Alaskan Native 2.33% Asian or Pacific Islander 25.85% Hispanic 72.46% English 2.12% Spanish 1.48% Other 1.91% Black, Not of Hispanic Origin 23.94% No Report COLORADO READING CORPS EVALUATION 2012-2013 8

55.90% White, Not of Hispanic Origin 4.66% Multiple Reports 7.80%% No Report or unknown Interpretation: Results in Tables 1 and 2 indicate tutors were able to provide services to nearly 500 students in 12 different elementary schools. Grades 1-3 had similar numbers of students served while kindergarten had relatively fewer students served. Internal Coaches make decisions about which grades will be provided Reading Corps services, and lower numbers in kindergarten may reflect school staff interest in providing students exposure to quality core instruction in order to identify students needing Reading Corps, but may also reflect other factors (e.g., greater availability of other school-based resources at the kindergarten level). Table 3 shows relatively consistent tutoring participation across grades 1-3, and lower levels in kindergarten. Differences between kindergarten and older grades may be due to similar factors as discussed above. Results show that the Reading Corps program was able to provide an average of nearly 20 weeks of support for students in grades 1-3 (over 11 weeks for kindergarten students), with over 70 minutes of tutoring provided each week (over 63 minutes for kindergarten students). Table 4 shows slightly more boys than girls received Reading Corps, and that the two primary ethnicities served were white (>55%) and Hispanic (>25%). COLORADO READING CORPS EVALUATION 2012-2013 9

2. What are the primary literacy outcomes of the Colorado Reading Corps? Data Source and Collection Methods: Data for literacy outcomes come from two sources that are maintained by the Reading Corps tutors. Each tutor recorded the status of students who were served by Reading Corps, noting if (and when) students met criteria for exiting the program. In addition, tutors also recorded student literacy data from triannual benchmark screenings, which were used to determine if students were at or above year-end (i.e., Spring) benchmark targets. Tutors also collected weekly literacy data that were stored and graphed using an internet-based progress monitoring system. The assessments used at each grade are listed in Appendix A (and in the parentheses of the Table 7 column headers). Table 5: Presents conservative exit statistics for students (see NOTE for details about exiting students) Table 6: Presents percentage of students who exited and reached spring literacy targets Table 7: Presents average growth statistics using weekly literacy data from progress monitoring measures at each grade Figures 1-5: Show linear growth curve estimates for weekly data from progress monitoring measures at each grade

Table 5: Percentage of Students Who Exited Reading Corps* Grade Number of Students Number Exited* Percent Exited** K 77 34 44.16% 1 128 56 43.75% 2 133 48 36.09% 3 134 70 52.23% K-3 TOTAL 472 208 44.07% * Exited indicates student progress was at or above grade-level trajectories, which is the criterion for exiting Reading Corps. Practically, Reading Corps determines at or above gradelevel trajectory as having 3-5 consecutive weekly points above a target growth line (as depicted on the internet-based progress monitoring system) plus 2 points above an upcoming seasonal benchmark target score. **NOTE: Tutors are required to maintain a full caseload at all times, including at the end of the school year. Students in Reading Corps exit, and therefore enter, on a rolling basis, and each tutor has a full caseload (>15 students) at the end of the school year. Thus, it is reasonable to assume that many of the students at the end of the year have not had sufficient tutoring to meet the exit criteria described above. Given that Percent Exited is computed based on all students who were provided services, including students on a tutor s caseload at the end of the school year, it is considered a conservative estimate. Table 6: Percentage Who Exit Who Also Meet or Exceed Spring Benchmark Number Exited with Spring Benchmark* Number Above Spring Benchmark Percent Above Spring Benchmark Number Grade Exited K 34 32 28 87.50% 1 56 53 49 92.45% 2 48 38 34 89.47% 3 70 66 44 66.67% K-3 Total 208 189 155 82.01% *NOTE: Accounts for students who may have not have a score for the spring benchmark assessments for various reasons (e.g., moved, extended absence). COLORADO READING CORPS EVALUATION 2012-2013 11

Table 7: Kindergarten - Grade 3 Participant Growth* Grade K (Letter Sound Fluency) Grade 1 (Nonsense Word Fluency) Grade 1 (R-CBM)** Grade 2 (R-CBM) Grade 3 (R-CBM) Mean Growth Rate 1.58 1.63 1.55 1.49 1.38 Target Growth Rate 1.15 1.11 1.67 1.31 1.08 Number of Students 76 117 83 88 104 Percentage of Students Above Target Growth Rate 65.8% 70.1% 43.4% 66.2% 75.4% * Only students with 3 or more data points on the given measure were included in growth rate calculations ** Students in this group may have also participated in Grade 1 NWF COLORADO READING CORPS EVALUATION 2012-2013 12

Letter Sounds Correct per Minute Figure 1: Grade K: Letter Sound Fluency Growth Curve Estimate 50 Grade K: Letter Sound Fluency Growth Curve Estimate 40 30 20 10 0 TOTAL TARGET -10-20

Correct Sounds per Minute Colorado Reading Corps Figure 2: Grade 1 Nonsense Word Fluency Growth Curve Estimate 80 Grade 1: Nonsense Word Fluency Growth Curve Estimate 70 60 50 40 TOTAL TARGET 30 20 10 0 COLORADO READING CORPS EVALUATION 2012-2013 14

Words Correct per Minute Colorado Reading Corps Figure 3: Grade 1: Oral Reading Fluency Growth Curve Estimate 60 Grade 1: Oral Reading Fluency Growth Curve Estimate 50 40 TOTAL 30 TARGET 20 10 0 COLORADO READING CORPS EVALUATION 2012-2013 15

Words Correct per Minute Colorado Reading Corps Figure 4: Grade 2: Oral Reading Fluency Growth Curve Estimate Grade 2: Oral Reading Fluency Growth Curve Estimate 100 90 80 70 60 50 40 TOTAL TARGET 30 20 10 0 COLORADO READING CORPS EVALUATION 2012-2013 16

Words Correct per Minute Colorado Reading Corps Figure 5: Grade 3: Oral Reading Fluency Growth Curve Estimate Grade 3: Oral Reading Fluency Growth Curve Estimate 160 140 120 100 80 TOTAL 60 TARGET 40 20 0 COLORADO READING CORPS EVALUATION 2012-2013 17

Interpretation: Results from Table 5 indicate that, overall, approximately 44% of the 472 students served met criteria for exiting Reading Corps, which indicates these students had demonstrated sufficiently strong reading growth to warrant no longer receiving support. It is important to note that the overall percentage (44%) is based on total students served, which is a conservative estimate of student performance (see Note below table). The percentage of students who did not exit (approximately 56%) likely includes many students who would benefit from CRC services in the following year. Table 6 shows that, overall, more than 80% of students who exited subsequently met the spring benchmark target for their grade. Students in kindergarten through grade 2 had similar performance, with 87% to 92% of students meeting spring benchmarks, whereas the proportion of students in grade 3 (66%) who met spring benchmark targets was lower. Reading Corps works exclusively with students who are initially below grade-level benchmarks, and these data suggest performance consistent with gradelevel expectations for a relatively high proportion of students who meet exit criteria. The growth data presented in Table 7 and Figures 1-5 demonstrate that for most grades average growth rates on the progress monitoring measures exceeded target growth rates. For example, Table 7 shows the average growth rate for kindergarten students was 1.58 correct letter sounds (CLS) per week, compared to a target growth rate of 1.15 CLS per week. This is further reflected in Figure 1, which illustrates mean growth (red line labeled Total ) compared to the seasonal targets (short black lines labeled Target ) for kindergarten students. It shows that in the fall, average student performance (approximately 1-3 CLS) was well below the target (10 CLS), but that by spring average student performance (approximately 43 CLS) exceeded the target (41 CLS). Table 7 further reflects average student performance by showing approximately 66% of kindergarten students had individual growth rates that exceeded the target growth rate. Although a similar trend is found in other grades, Figure 4 shows that the average performance across grade 2 did not exceed the spring target (although it did narrow the gap that existed at fall and winter benchmarks), and Figure 3 shows performance in grade 1 on the curriculum-based measure for reading (R-CBM) was not as strong. COLORADO READING CORPS EVALUATION 2012-2013 18

Letter Naming Letter Sounds Nonsense Words Oral Reading Fluency Colorado Reading Corps 3. How well is the Reading Corps program implemented in Colorado? Data Source and Collection Methods: Implementation data are collected by Reading Corps Master and Internal Coaches. Coaches observe Members administer each assessment at least 3 times each year, prior to collecting benchmark data. Coaches also observe Members administer interventions at least monthly. During both types of observations, coaches complete fidelity checklists that produce data on implementation quality (see tables below). Assessment fidelity data are collected using the Accuracy of Implementation Rating Scales (AIRS), and intervention fidelity data are collected using structured checklist protocols that correspond to key aspects of implementation for each intervention. Table 8: Presents fidelity data for assessment observations. Table 9: Presents fidelity data for intervention observations. Table 8: Fidelity of Assessment Data Collection Procedures Measure Total AIRS Collected Fidelity Range Reported 104 104 104 112 69% - 100% 83% - 100% 67% - 100% 75% - 100% Median % Fidelity Reported 100% 100% 100% 100% COLORADO READING CORPS EVALUATION 2012-2013 19

Mean % Fidelity Reported 95.38% 97.00% 95.73% 97.33% Standard Deviation 8.32% 4.92% 7.00% 4.59% Table 9: Fidelity of Intervention Implementation Procedures Intervention Total Complete Fidelity Checks Fidelity Range Reported Median Percent Fidelity Reported Mean Percent Fidelity Reported Standard Deviation Blending Words 77 57% - 100% 100% 93.61% 10.53% Duet Reading 144 56% - 100% 100% 94.23% 8.16% Great Leaps 133 50% - 100% 100% 94.68% 8.49% Letter / Sound Correspondence 39 25% - 100% 100% 90.38% 17.74% Newscaster Reading 6 91% - 100% 100% 98.48% 3.71% Pencil Tap 7 60% - 88% 79% 79.27% 10.06% Phoneme Blending 14 75% - 100% 100% 96.30% 7.81% Phoneme Segmenting 13 43% - 100% 75% 74.88% 19.52% Repeated Reading with Comprehension 89 33% - 100% 92% 87.85% 12.28% Interpretation: Results from Table 8 show that assessment data were collected with very high levels of fidelity, which suggests that the student outcome data are accurate. Results from Table 9 show that most interventions were conducted with high degrees of fidelity, which means they were implemented as intended. Given that each intervention has an extensive evidence-based (see Appendix B), this indicates the interventions were implemented in accordance with their established evidence-base. Two interventions had relatively low fidelity for implementation: pencil tap and phoneme segmenting. These were two of the three least-used interventions, but data suggest a need for more tailored coaching and training for these interventions. COLORADO READING CORPS EVALUATION 2012-2013 20

4. What do key stakeholders and implementers report regarding how participation in Reading Corps has affected their beliefs, opinions, and/or professional practices? Data Source and Collection Methods: Data for stakeholder and implementer beliefs, opinions, and professional practices come from participant surveys. Reading Corps Internal Coaches and tutors completed surveys in the spring of the 2012-2013 school year. A total of 12 Internal Coaches and 17 tutors completed surveys. Bolded survey items indicate items reflecting the impact of Reading Corps. Select unbolded items can be compared with corresponding bolded items to further understand the Reading Corps impact on certain educational practices. Table 10: Presents survey results from Internal Coaches Table 11: Presents survey results from tutors Table 10: Internal Coach Survey Results The following questions capture the overall opinions of Internal Coaches regarding the Reading Corps Program. Answer Options Overall, the Reading Corps program has influenced systems change at my site. Reading Corps is a good fit with my school or center. Reading Corps adds value to the instructional program at this school or center Strongly Agree Agree Disagree Strongly Disagree 41.7% 50.0% 8.30% 0.00% 100.00% 0.00% 0.00% 0.00% 100.00% 0.00% 0.00% 0.00% COLORADO READING CORPS EVALUATION 2012-2013 21

The following questions are designed to capture Internal Coach perceptions regarding changes to their local system resulting from participation in the Reading Corps program. Answer Options Prior to participating in the CRC program, our building collected fluency based screening measures for all students K-3 at least 3 times per year. Due (at least in part) to our participation in the CRC program, our building has begun or will begin collecting fluency based screening measures for all students K-3 at least 3 times per year. Prior to participating in the CRC program, teachers in our building used screening data to assist in identifying students for supplemental interventions. Due (at least in part) to our participation in the CRC program, teachers in our building now use screening data to assist in identifying students for supplemental interventions. Prior to participating in the CRC program, teachers regularly reviewed progress-monitoring data (weekly graphs) of students receiving supplemental interventions. Due (at least in part) to our participation in the CRC program, teachers now regularly review progressmonitoring data (weekly graphs) of students receiving supplemental interventions. Prior to participating in the CRC program, teachers viewed progress monitoring (weekly data) as an important method to evaluate the impact of instruction on students. Due (at least in part) to our participation in the CRC program, teachers now view progress monitoring (weekly data) as an important method to evaluate the impact of instruction on students. Prior to participating in the CRC program, our school used aggregated data as one way to evaluate the instructional practices of the site. Due (at least in part) to our participation in the CRC program, our school now uses aggregated data as one way to evaluate the instructional practices of the site. Prior to participating in the CRC program, the building principal shared data on student performance with the superintendent or school board. Strongly Agree Agree Disagree Strongly Disagree 83.33% 16.67% 0.00% 0.00% 0.00% 41.67% 50.00% 8.33% 16.67% 75.00% 8.33% 0.00% 16.67% 75.00% 8.33% 0.00% 8.33% 41.67% 33.33% 16.67% 25.00% 66.67% 8.33% 0.00% 16.67% 58.33% 16.67% 8.33% 16.67% 75.00% 8.33% 0.00% 33.33% 33.33% 33.33% 0.00% 8.33% 83.33% 8.33% 0.00% 27.27% 27.27% 36.36% 9.09% COLORADO READING CORPS EVALUATION 2012-2013 22

Due (at least in part) to our participation in the CRC program, the building principal now shares data on student performance with the superintendent or school board. Prior to participating in the CRC program, teachers shared progress monitor graphs with parents. Due (at least in part) to our participation in the CRC program, teachers now share progress monitor graphs with parents. Due (at least in part) to our participation in the CRC program, greater emphasis has been placed on selecting reading interventions for students that have a scientific research base. Prior to participating in the CRC program, instruction was modified if student performance was not improving based on the progress monitoring data collected. Due (at least in part) to our participation in the CRC program, instruction is now modified if student performance is not improving based on the progress monitoring data collected. Prior to participating in the CRC program, the district had adopted its own data warehouse system for efficiently storing and accessing data. Due (at least in part) to our participation in the CRC program, the district has now adopted its own data warehouse system for efficiently storing and accessing data. Due (at least in part) to our participation in the CRC program, the district has taken concrete steps I am aware of to formally link Pre-K with K-3 literacy instruction. 0.00% 36.36% 63.64% 0.00% 8.33% 66.67% 16.67% 8.33% 16.67% 58.33% 25.00% 0.00% 16.67% 75.00% 8.33% 0.00% 16.67% 58.33% 16.67% 8.33% 45.45% 45.45% 9.09% 0.00% 50.00% 50.00% 0.00% 0.00% 0.00% 33.33% 50.00% 16.67% 8.33% 25.00% 66.67% 0.00% COLORADO READING CORPS EVALUATION 2012-2013 23

Table 11: CRC Tutor Impact Survey Results The following items capture the overall opinions of Tutors regarding the Reading Corps Program. Answer Options Participation in Reading Corps had a positive impact on me this school year. Participation in Reading Corps had a positive impact on the site I served this school year. Participation in Reading Corps had a positive impact on the students I served this year. Strongly Agree Agree Disagree Strongly Disagree 70.59% 29.41% 0.00% 0.00% 76.47% 23.53% 0.00% 0.00% 76.47% 23.53% 0.00% 0.00% The following items asked Reading Corps Tutors to rate their agreement was based on their experience in 2012-2013. Answer Options Strongly Agree Agree Disagree Strongly Disagree As a result of my participation in the CRC program, I am considering a career involving children As a result of my participation in the CRC program, I am considering a career in teaching or education As a result of my participation in the CRC program, I am committed to continued volunteering in schools As a result of my participation in the CRC program, I am committed to ongoing promotion of childhood literacy. As a result of my participation in the CRC program, I am committed to continued community service. As a result of my participation in the CRC program, if a job I hold in the future does not have community service as part of its mission, I will encourage the organization to include opportunities for community service. 35.29% 52.94% 11.76% 0.00% 35.29% 47.06% 17.65% 0.00% 29.41% 47.06% 23.53% 0.00% 64.71% 35.29% 0.00% 0.00% 29.41% 52.94% 17.65% 0.00% 23.53% 70.59% 5.88% 0.00% COLORADO READING CORPS EVALUATION 2012-2013 24

Interpretation: Results shown in Table 10 indicate Internal Coaches strongly agreed that Reading Corps had a positive overall impact on their school. Results for non-bolded items show that the schools in which Reading Corps operated had already begun systems change related to the procedures used by Reading Corps. For example, 100% of Internal Coaches agreed that their building used fluency-based screening measures, and approximately 85% agreed that teachers at their schools viewed progress monitoring data as an important tool for evaluating the impact of instruction on students. However, bolded items indicate Reading Corps was considered at least part of the reason for positive system change toward integrating these procedures within the school. For example, nearly 93% of Internal Coaches reported that Reading Corps contributed to teachers viewing progress monitoring data to evaluate instruction. Table 11 displays results for Reading Corps tutors. It shows that the experience of Reading Corps tutors was unequivocally positive and that tutors agreed Reading Corps had a positive impact at the schools in which they worked. Additional survey items note that tutors consider their experience as contributing to ongoing commitments to promote childhood literacy and engage in continued community service. Further, over 80% of tutors indicated they are considering a career in education due to their participation in Reading Corps. COLORADO READING CORPS EVALUATION 2012-2013 25

APPENDIX A ADDITIONAL ASSESSMENT INFORMATION Assessment Measures and Procedures The assessment tools used to determine literacy progress of CRC-participating students include the following measures: Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Oral Reading Fluency The following table depicts which measures are used at each grade across the school year. BOLDED measures are used to progress monitor (1 st grade students are monitored for progress with two measures during part of the winter). Kindergarten Letter Naming Fluency Letter Sound Fluency Grade 1 Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Grade 2 Oral Reading Fluency (3 passages) Grade 3 Oral Reading Fluency (3 passages) Fall Winter Spring Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency Fluency Nonsense Word Fluency Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Letter Naming Fluency Letter Sound Fluency Nonsense Word Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) Oral Reading Fluency (3 passages) COLORADO READING CORPS EVALUATION 2012-2013 26

Assessment Research Base Assessment tools were selected for use in the Reading Corps because of their well-established statistical reliability and validity for screening and progress monitoring purposes. Letter Naming, Letter Sounds, and Nonsense Words are measures of early literacy skills thoroughly researched by many groups, but most famously packaged by two assessment programs: DIBELS and AIMSweb. Oral Reading Fluency provides an assessment of connected text reading. Early and ongoing research on this measure has also been conducted at the University of Minnesota. All these measures fit under the umbrella of Curriculum-Based Measurement (CBM), and are fluency based assessments, meaning that students respond to an unlimited number of items within a fixed amount of time, and the number of correct responses is counted. The information that follows summarizes empirical findings related to the statistical reliability and validity of the measures used in the Reading Corps. Letter Naming Fluency: r=.94 inter rater reliability r=.90 2 week test retest reliability r=.88 1 month alternate reliability r=.93 alternate forms reliability r=.70 with WJ-R Readiness Cluster r=.70 with WJ Psychoeducational Battery r=.53 to.58 with CTOPP Composite Predictive r=.65 with WJ Total Reading Cluster Predictive r=.71 with R-CBM ELL Predictive r =.67 with a composite of DIBELS NWF and R-CBM Sources: Assessment Committee Report for Reading First. (2002). Analysis of Reading Assessment Measures. Retrieved February 21, 2007, from http://dibels/uoregon.edu/techreports/dibels_5th_ed.pdf Good, R.H., Kaminski, R.A., Shinn, M. Bratten, J., Shinn, M., & Laimon, L. (in preparation). Technical Adequacy and Decision Making Utility of DIBELS (Technical Report). Eugene, OR: University of Oregon COLORADO READING CORPS EVALUATION 2012-2013 27

Good, R.H. III., Kaminski, R.A., Simmons, D., Kame enui, E.J. (2001). Using Dynamic Indicators of Basic Early Literacy Skills (DIBELS) in an outcomesdriven model: Steps to reading outcomes. Unpublished manuscript, University of Oregon at Eugene. Elliot, J., Lee, S.W., Tolefson, N. (2001). A Reliability and Validity Study of the Dynamic Indicators of Basic Early Literacy Skills Modified. School Psychology Review, 30 (1), 33-49. Haager, D. & Gersten, R (April, 2004). Predictive Validity of DIBELS for English Learners in Urban Schools. DIBELS Summit conference presentation, Albuquerque, NM Hintz, J.M., Ryan, A.L., & Stoner, G. (2003). Concurrent Validity and Diagnostic Accuracy of DIBELS and the CTOPP. School Psychology Review Kaminski, R.A. & Good, R.H. (1996). Toward a Technology for Assessment Basic Early Literacy Skills. School Psychology Review, 25, 215-227. Rouse, H., Fantauzzo, J.W. (2006). Validity of the Dynamic Indicators of Basic Early Literacy Skills as an Indicator of Early Literacy for Urban Kindergarten Children. School Psychology Review 35 (3)3 341-355. Letter Sound Fluency: r=.83 2-week test-retest reliability r=.80 alternate form reliability r=.79 with Letter Naming Fluency Predictive r=.72 with R-CBM Sources: Elliott, J., Lee, S.W., & Tollefson, N. (2001). A Reliability and Validity Study of the Dynamic Indicators of Basic Early Literacy Skills Modified. School Psychology Review, 30 (1), 33-49. Fuchs, L., Fuchs D. (2004). Determining Adequate Yearly Progress from Kindergarten through Grade 6 with Curriculum Based Measurement. Assessment for Effective Intervention 29 (4) 25-37. COLORADO READING CORPS EVALUATION 2012-2013 28

Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School- Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, 59-72. Scott, S.A., Sheppard, J., Davidson, M.M., & Browning, M.M. (2001). Prediction of First Graders Growth in Oral Reading Fluency Using Kindergarten Letter Naming Fluency. Journal of School Psychology, 39(3), 225-237. Ritchey, K.D (2008). Assessing Letter Sound Knowledge: A Comparison of Letter Sound Fluency and Nonsense Word Fluency. Exceptional Children 74 (4) 487-506. Nonsense Word Fluency: r=.83 one month alternate form reliability r=.36 to.59 with WJ-R Readiness Cluster Predictive r=.82 with Spring R-CBM in Spring of grade 1 Predictive r =.65 with oral reading and.54 with maze in grade 3 Ell Predictive r=.63 with a composite of DIBELS NWF and R-CBM Sources: Burke, M. D., Hagan-Burke, S. (2007). Concurrent criterion-related validity of early literacy indicators for middle of first grade. Assessment for Effective Intervention. 32(2), 66-77. Good, R.H., Kaminski, R.A., Shinn, M. Bratten, J., Shinn, M., & Laimon, L. (in preparation). Technical Adequacy and Decision Making Utility of DIBELS (Technical Report). Eugene, OR: University of Oregon Good, R.H., Kaminski, R.A., Simmons, D., & Kame-enui, E.J. (2001). Using DIBELS in an Outcomes Driven Model: Steps to Reading Outcomes. Unpublished manuscript, University of Oregon, Eugene. Haager, D. & Gersten, R (April, 2004). Predictive Validity of DIBELS for English Learners in Urban Schools. DIBELS Summit conference presentation, Albuquerque, NM COLORADO READING CORPS EVALUATION 2012-2013 29

Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School- Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, 59-72 Kaminski, R.A. & God, R.H. (1996). Toward a Technology for Assessment Basic Early Literacy Skills. School Psychology Review, 25, 215-227. Ritchey, K.D (2008). Assessing Letter Sound Knowledge: A Comparison of Letter Sound Fluency and Nonsense Word Fluency. Exceptional Children 74 (4) 487-506. Rouse, H., Fantauzzo, J.W. (2006). Validity of the Dynamic Indicators of Basic Early Literacy Skills as an Indicator of Early Literacy for Urban Kindergarten Children. School Psychology Review 35 (3)3 341-355. Vanderwood, M.., Linklater, D., Healy, K. (2008). Predictive Accuracy of Nonsense Word Fluency for English Language Learners. School Psychology Review 37 (1) 5-17. Oral Reading Fluency: r=.92 to.97 test retest reliability r=.89 to.94 alternate form reliability r=.82 to.86 with Gates-MacGinite Reading Test r=.83 to Iowa Test of Basic Skills r =.88 to Stanford Achievement Test r=.73 to.80 to Colorado Student Assessment Program r=.67 to Michigan Student Assessment Program r=.73 to North Carolina Student Assessment Program r=74 to Arizona Student Assessment Program r=.61 to.65 to Ohio Proficiency Test, Reading Portion r=.58 to.82 with Oregon Student Assessment Program (SAT 10) Sources: Barger, J. (2003). Comparing the DIBELS Oral Reading Fluency indicator and the North Carolina end of grade reading assessment (Technical Report). Ashville, NC: North Carolina Teacher Academy. COLORADO READING CORPS EVALUATION 2012-2013 30

Baker S. et. al,. (2008). Reading Fluency as a Predictor of Reading Proficiency in Low-Performing, High-Poverty Schools. School Psychology Review 37 (1) 18-37. Burke, M. D., Hagan-Burke, S. (2007). Concurrent criterion-related validity of early literacy indicators for middle of first grade. Assessment for Effective Intervention. 32(2), 66-77. Deno, S. L., Mirkin, P. K., & Chiang, B. (1982). Identifying valid measures of reading. Exceptional Children, 49. 36-45. Howe, K. B., Scierka, B. J., Gibbons, K. A., & Silberglitt, B. (2003). A School- Wide Organization System for Raising Reading Achievement Using General Outcome Measures and Evidence-Based Instruction: One Education District s Experience. Assessment for Effective Intervention, 28, 59-72 Hintze, J.M, et al (2002). Oral Reading Fluency and Prediction of Reading Comprehension in African American and Caucasian Elementary School Children. School Psychology Review, 31 (4) 540-553 Hintze, J. M. & Silberglitt, B. (in press). A Longitudinal Examination of the Diagnostic Accuracy and Predictive Validity of R-CBM and High-Stakes Testing. School Psychology Review. Marston, D., Fuchs, L., & Deno, S. (1987). Measuring pupil progress: a comparison of standardized achievement tests and curriculum-related measures. Diagnostique, 11, 77-90. Marston, D. (1989). Curriculum-based measurement: What is it and why do it? In M. R. Shinn (Ed.), Curriculum-based measurement: Assessing special children (pp. 18-78). New York: Guilford Press. McGlinchey, M. T., & Hixson, M. D. (2004). Contemporary research on curriculum-based measurement: Using curriculum-based measurement to predict performance on state assessments in reading. School Psychology Review, 33(2), 193-204. COLORADO READING CORPS EVALUATION 2012-2013 31

Schilling, S. G., Carlisle, J. F., Scott, S. E., & Zeng, J. (2007). Are fluency measures accurate predictors of reading achievement? The Elementary School Journal, 107(5), 429-448. Silberglitt, B. & Hintze, J. M. (in press). Formative Assessment Using Oral Reading Fluency Cut Scores to Track Progress Toward Success on State- Mandated Achievement Tests: A Comparison of Methods. Journal of Psychoeducational Assessment. Shaw, R., & Shaw, D. (2002). DIBELS Oral Reading Fluency-Based Indicators of the third-grade reading skills for Colorado State Assessment Program (CSAP) (Technical Report). Eugene, OR: University of Oregon. Shinn, M., Good, R., Knutson, N., Tilly, W., & Collins, A. (1992). Curriculumbased measurement of oral reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 459-479. Stage, S. A., & Jacobsen, M. D. (2001). Predicting student success on a state-mandated performance-based assessment using oral reading fluency. School Psychology Review, 30(3), 407-420. Tindal, G., Germann, G., & Deno, S. (1983). Descriptive research on the Pine County Norms: A compilation of findings (Research Report No. 132). Minneapolis, MN: University of Minnesota Institute for Research on Learning Disabilities. Vander Meer, C. D., Lentz, F. E., & Stollar, S. (2005). The relationship between oral reading fluency and Ohio proficiency testing in reading (Technical Report). Eugene, OR: University of Oregon. Wilson, J. (2005). The relationship of Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Oral Reading Fluency to performance on Arizona Instrument to Measure Standards (AIMS). Tempe, AZ: Tempe School District No. 3. COLORADO READING CORPS EVALUATION 2012-2013 32

APPENDIX B ADDITIONAL INTERVENTION INFORMATION Intervention Research Base The interventions used in the Reading Corps program are designed to provide additional practice that is supplemental to the core reading instructional program offered by the local school site. The interventions target automaticity and fluency of important reading skills that have been introduced by local classroom teachers. It is important to note that Reading Corps participation is in addition to, not in replacement of, a comprehensive core reading instructional program, and that the Reading Corps program should in no way be viewed as a substitute for high quality core instruction. Reading Corps (for more information, see: Harn, B.A., Linan-Thomson, S., & Roberts G. (2008). Intensifying Instruction: Does Additional Instructional Time Make a Difference for the Most At-Risk First Graders? Journal of Learning Disabilities 41(2).) A unique feature of Reading Corps is the consistent use of research-based intervention protocols with participating students to provide this additional support. School-based Internal Coaches select from a menu of research-based supplemental reading interventions for use with participating students as listed below. For each intervention protocol, a description of the research base, and/or sources of empirical evidence of intervention effectiveness are listed. Repeated Reading with Comprehension Strategy Practice Nelson, J. S., Alber, S. R., & Grody, A. (2004). Effects of systematic error correction and repeated readings on reading accuracy and proficiency of second graders with disabilities. Education and Treatment of Children, 27, 186 198. Staubitz, J. E., Cartledge, G., Yurick, A., & Lo, Y. (2004). Repeated reading for students with emotional or behavioral disorders: Peerand trainermediated instruction. Behavior Disorders, 31, 51 64 COLORADO READING CORPS EVALUATION 2012-2013 33

Therrien, W. J. (2004). Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education, 25, 252 261. Moyer, S.B. (1982). Repeated reading. Journal of Learning Disabilities, 45, 619-623 Rashotte, C.A., & Torgeson, J.K. (1985). Repeated reading and reading fluency in learning disabled children. Reading Research Quarterly. 20, 180-188 Samuels, S. J. (1979). The method of repeated reading. The Reading Teacher, 32, 403-408. Samuels, S.J., (1987). Information processing abilities and reading. Journal of Learning Disabilities, 20(1), 18-22. Sindelar, P.T., Monda, L.E., & O Shea, L.J. (1990). Effects of repeated reading on instructional and mastery level readers. Journal of Educational Research, 83, 220-226. Therrien, W.J. (2004). Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education. 25(4) 252-261 Morrow, L. M. (1985). Retelling stories: A strategy for improving young children s comprehension, concept of story structure, and oral language complexity. The Elementary School Journal, 85, 646 661. Duet Reading Aulls, M.W., (1982). Developing Readers in Today s Elementary Schools. Allyn & Bacon: Boston. Blevins, W. (2001). Building Fluency: Lessons and Strategies for Reading Success. New York: Scholastic Professional Books. Dowhower, S.L. (1991). Speaking of prosody: Fluency s unattended bedfellow. Theory into Practice, 30 (3), 165-175. COLORADO READING CORPS EVALUATION 2012-2013 34

Mathes, P.G., Simmons, D.C., & Davis, B.I. (1992). Assisted reading techniques for developing reading fluency. Reading Research and Instruction, 31, 70-77. Weinstein, G., & Cooke, N. L. (1992). The effects of two repeated reading interventions on generalization of fluency. Learning Disability Quarterly, 15, 21 27. Newscaster Reading Armbruster, B.B., Lehr, F., & Osborn, J. (2001). Put reading first: The research building blocks for teaching children to read. Washington, DC: US Department of Education, National Institute for Literacy. Dowhower. S.L. (1987). Effects of repeated reading on second-grade transitional readers fluency and comprehension. Reading Research Quarterly. 22, 389-406. (listening to a tape) Heckelman, R.G. (1969). A neurological-impress method of remedial reading instruction. Academic Therapy, 4, 277-282. Daly, E. J., III, & Martens, B. (1994). A comparison of three interventions for increasing oral reading performance: Application of the instructional hierarchy. Journal of Applied Behavior Analysis, 29, 507 518. Skinner, C. H., Adamson, K. L., Woodward, J. R., Jackson, R. R., Atchison, L. A., & Mims, J. W. (1993). The effects of models rates of reading on students reading during listening previewing. Journal of Learning Disabilities, 26, 674 681. Rasinski, T.V. (2003). The fluent reader: Reading strategies for building word recognition, fluency, and comprehension. New York, NY: Scholastic Professional Books. Searfoss, L. (1975). Radio Reading. The Reading Teacher, 29, 295-296. Stahl S. (2004). What do we Know About Fluency?: Findings of the National Reading Panel. In McCardle, P., & Chhabra, V. (Eds) The Voice of Evidence in Reading Research. Brookes: AU. COLORADO READING CORPS EVALUATION 2012-2013 35

Stop Go Blevins, W. (2001). Building Fluency: Lessons and Strategies for Reading Success. New York: Scholastic Professional Books. Rasinski, T., & Padak, N. (1994). Effects of fluency development on urban second-graders. Jorunal of Education Research, 87. Rasinski, T.V. (2003). The fluent reader: Reading strategies for building word recognition, fluency, and comprehension. New York, NY: Scholastic Professional Books. Pencil Tap Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Education Research. 77(1), 81-112. Howell, K., W., & Nolet. V., (2000). Curriculum-Based Evaluation: Teaching and Decision Making 3 rd Ed. Belmont, CA: Wadsworth. Lysakowski, R.S., & Walberg, H.J. (1982). Instructional effects of cues, participation, and corrective feedback: A quantitative synthesis. American Educational Research Journal Vol 19(4), 559-578 Tenenbaum, G., & Goldring, E. (1989). A meta-analysis fo the effecta of enhanced instruction: Cues, participation, reinforcement and feedback and correctives on motor skill learning. Journal of Research & Development in Education. Vol 22(3) 53-64. Letter Sound Identification Adams, M.J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press. Adams, M.J. (2001).Alphabetic anxiety and explicit, systematic phonics instruction: A cognitive science perspective. In S.B. Neuman & D.K. Dickinson (eds.), Handbook of Early Literacy Research (pp. 66-80). New York: Guilford Press. COLORADO READING CORPS EVALUATION 2012-2013 36

Chard, D.J., & Osborn, J. (1999). Word Recognition: Paving the road to successful reading. Intervention in school and clinic, 34(5), 271-277. Word Blending Adams, M.J. (2001).Alphabetic anxiety and explicit, systematic phonics instruction: A cognitive science perspective. In S.B. Neuman & D.K. Dickinson (eds.), Handbook of Early Literacy Research (pp. 66-80). New York: Guilford Press. Goswami, U. (2000). Causal connections in beginning reading: The importance of rhyme. Journal or Research in Reading, 22(3) 217-240. Greaney, K.T., Tunmer, W.E., & Chapman, J.W., (1997). Journal of Educational Psychology, 89(4) 645-651. Phoneme Blending Adams, M.J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press. Bos, C.D., & Vaughn, S. (2002). Strategies for teaching students with learning and behavioral problems (5 th Ed.). Boston: Allyn & Bacon. Ehri, L.C., Nunees, S.R., & Willows, D.M. (2001). Phonemic awareness instruction helps children learn to read: Evidence from the National Reading Panel s meta-analysis. Reading Research Quarterly, 36(3). 250-287. Elkonin, D.B. (1973). U.S.S.R. In J. Downing (Ed.), Comparative Reading (pp.551-579). New York: MacMillan. National Reading Panel. (2000). Teaching children to read: An evidencebased assessment of the scientific research literature on reading and its implications for reading instruction. Bethesda, MA: National Institutes of Health. Santi, K.L., Menchetti, B.M., & Edwards, B.J. (2004). A comparison of eight kindergarten phonemic awareness programs based on empirically COLORADO READING CORPS EVALUATION 2012-2013 37

validated instructional principals. Remedial and Special Education, Vol 25(3) 189-196. Smith, C.R. (1998). From gibberish to phonemic awareness: Effective decoding instruction. Exceptional Children, Vol 30(6) 20-25 Smith, S.B., Simmons, D.C., & Kame enui, E, J. (1998). Phonological Awareness: Research bases. In D.C. Simmons & E.J. Kame enui (Eds.), What Reading research tells us about children with diverse learning needs: Bases and basics. Mahwah, NJ: Lawrence Erlbaum Associates. Snider, V. E. (1995). A primer on phonemic awareness: What it is, why it is important, and how to teach it. School Psychology Review, 24, 443 455. Phoneme Segmentation Adams, M.J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press. Blachman, B. A. (1991). Early intervention for children s reading problems: Clinical applications of the research on phonological awareness. Topics in Language Disorders, 12, 51 65. Bos, C.D., & Vaughn, S. (2002). Strategies for teaching students with learning and behavioral problems (5 th Ed.). Boston: Allyn & Bacon. Ehri, L.C., Nunees, S.R., & Willows, D.M. (2001). Phonemic awareness instruction helps children learn to read: Evidence from the National Reading Panel s meta-analysis. Reading Research Quarterly, 36(3). 250-287. National Reading Panel. (2000). Teaching children to read: Anevidencebased assessment of the scientific research literature on reading and its implications for reading instruction. Bethesda, MA: National Institutes of Health. Santi, K.L., Menchetti, B.M., & Edwards, B.J. (2004). A comparison of eight kindergarten phonemic awareness programs based on empirically validated instructional principals. Remedial and Special Education, Vol 25(3) 189-196. COLORADO READING CORPS EVALUATION 2012-2013 38

Smith, C.R. (1998). From gibberish to phonemic awareness: Effective decoding instruction. Exceptional Children Vol 30(6) 20-25. Smith, S.B., Simmons, D.C., & Kame enui, E, J. (1998). Phonological Awareness: Research bases. In D.C. Simmons & E.J. Kame enui (Eds.), What Reading research tells us about children with diverse learning needs: Bases and basics. Mahwah, NJ: Lawrence Erlbaum Associates. Snider, V. E. (1995). A primer on phonemic awareness: What it is, why it is important, and how to teach it. School Psychology Review, 24, 443 455. COLORADO READING CORPS EVALUATION 2012-2013 39