Special School District Early Childhood Special Education Program Evaluation Martha Disbennett, Chair Board Approved: September 22, 2009
Executive Summary As required by the Missouri School Improvement Plan (MSIP) standards, school districts must evaluate Early Childhood Special Education services biennially. The question approved by the Board of Education for the present program evaluation was as follows: What percentage of preschool age children with Individual Education Plans in Special School District s Early Childhood Special Education Program demonstrated growth in literacy skills in the areas of Interactions with Story Books, Emergent Story Book Reading Skills, Early Writing Skills, Knowledge of Print and Alphabet, and Phonological Awareness? The present evaluation includes a review of achievement and survey data. Based on stakeholder review of the data, strengths, concerns and recommendations are noted. Results Strengths The ECSE Department is pleased with the gains in acquisition of literacy skills as a result of implementing the Animated Literacy Curriculum, a more focused, research-based curriculum. The gains our students have made in crucial emergent literacy skills will be a benefit as they enter kindergarten. Preparing them to have the building blocks necessary for reading in their later school years is part of our mission. In addition, the parents of children in the ECSE program are seeing the benefits of intentional literacy instruction. Concerns Despite positive results, the data indicate that not all students demonstrated growth in literacy skill domains as indicated on the LSC. In addition, a concern was noted with regard to the proportion of parents who indicated they rarely observed their child making sound-letter connections. Recommendations Additional analysis and exploration of the data via data teams are planned to assess characteristics of those students who did not demonstrate literacy skill growth. ECSE will also provide parents with information regarding literacy development in preschool children as it relates to the importance of phonemic awareness. To continually improve instruction for all of our students, ECSE will also (a) track students who participated in the Animated Literacy and Breakthrough to Literacy program to see how they performed on MAP/MAP-A in 3 rd grade in the area of Communication Arts, (b) continue to review the most current research on literacy skill development in children during the preschool years and inform staff of best practice in the area and (c) continue to measure student progress in the area of literacy as well as other developmental domains and work with staff on data gathering, analysis, and instructional decision making using data. Board Approved: 9/22/2009 1
Program Evaluation Question(s) What percentage of preschool age children with Individual Education Plans in Special School District s Early Childhood Special Education Program demonstrated growth in literacy skills in the areas of Interactions with Story Books, Emergent Story Book Reading Skills, Early Writing Skills, Knowledge of Print and Alphabet, and Phonological Awareness? I. Program/Service Information 1. Name of Program or Services: Early Childhood Special Education 2. Personnel Responsible for Evaluation and Program (list): Martha Disbennett 3. Demographic Description of Program: Location(s): Affton, Bayless, Brentwood, Jennings, Hancock, Ladue, Lindbergh, Maplewood/Richmond Heights, Normandy, Parkway, Ritenour, Riverview Gardens, Valley Park, Webster Groves, Wellston. Number of staff: 309 Participants: Preschool students ages 3-5 Length of program/service: Maximum 3 years 4. Date of Evaluation (Year/Duration): September 2008-June 2009 5. Goal/Objective of Program/Services: The goal of the Early Childhood Special Education Program is to identify preschool age children with disabilities and provide special education in the least restrictive environment. 6. Brief description of relationship between program goals, CSIP and MSIP Standards: Program goals are related to CSIP and MSIP standards. CSIP relationship- Goal 1: Develop and Enhance quality educational/instructional programs to improve performance and enable students to meet their personal, academic and career goals. Objective 1: Annually meet state criteria for performance of students for all subgroups for Adequate Yearly Progress in all content areas. MSIP- State Performance Plan Indicator #7- Percent of preschool children with IEPs who demonstrate improved acquisition and use of knowledge and skills (including early language/communication and early literacy). II. Evaluation Criteria for Programs/Services Offered For the purpose of evaluation, the ECSE Department used a Literacy Skills Checklist (LSC) developed in conjunction with St. Louis University s Department of Communication, as well as a parent perception survey related to skills assessed by the LSC. Board Approved: 9/22/2009 2
III. Description of Stakeholders Engagement in Program Evaluation: The ECSE stakeholders met on April 2 and April 24, 2009 to determine the evaluation question and what data would need to be collected to answer the question. A follow up meeting to review the data was held on September 14, 2009. The committee members and their roles are listed below. Carol Ackerson - Adjunct Professor, Department of Communications, St. Louis University Tracy Brangle - ECSE Staff Development Facilitator Traci Broeker - ECSE Speech Pathologist Kay Castello - ECSE Classroom Teacher, Ritneour Jennifer Corcoran - ECSE Speech Pathologist, Ladue Laurie Corners - ECSE Area Coordinator Brenda Deakin - ECSE EPS - OT/PT Amy DeGuire Parent, Lindbergh Mary Dunn - ECSE Speech Pathologist, Riverview Gardens Mary Elfrink - ECSE Diagnostic Teacher, Head Start Monica Fanning - Area Coordinator for County-Wide Deaf/Hard of Hearing Karen Flavin - Early Childhood Director, Ritenour Chris Gardiner - ECSE Area Coordinator Susan Harrman - ECSE Diagnostic Teacher, Central Area Karen Jones - ECSE Itinerant Teacher, South Area Mary Beth Krull - ECSE Diagnostic Teacher, South Jeanne Longo - ECSE Speech Pathologist, MRH/Hancock Dawn Hogan - Lopez-Early Childhood Director, Ladue Julie McClard - ECSE Area Coordinator Lisa MacKenzie - ECSE Itinerant Teacher, Riverview Gardens Laura Moore - ECSE Itinerant Teacher, Parkway Stephanie Opela - EC Teacher, Ritenour Nancy Sexton - ECSE Area Coordinator Nancy Tumbrink - ECSE Area Coordinator Martha Disbennett - ECSE Director IV. Results As a result of the ECSE Program Evaluation conducted in 2007 it was recommended that the Animated Literacy Curriculum, which the department piloted in 2006, be implemented department wide. In the fall of 2007 all teachers were trained on the curriculum and it was implemented in the majority of ECSE classrooms. Riverview Gardens was the exception as they requested to utilize the Breakthrough to Literacy curriculum and all ECSE teachers implemented that program in their classrooms. For this program evaluation the progress/achievement data utilized was taken from the 2008-2009 LSC. In addition, a parent survey was conducted to assess the degree to which parents witnessed their child demonstrating early literacy skills. Board Approved: 9/22/2009 3
Literacy Skills Checklist (LSC) The LSC (Appendix A) was originally developed by the ECSE Program Evaluation Committee in 2005. The items were identified from the Missouri Pre-K literacy standards. In 2006 the ECSE Department collaborated with faculty from the Communication Sciences and Disorders department at St. Louis University to conduct research focusing on language/literacy outcomes of children with developmental disabilities in the presence of classroom-based literacy interventions. The study was designed to examine child outcomes resulting from participation in three different emergent literacy curricula approaches utilized by ECSE and/or early childhood teachers and speech pathologists within a developmentally appropriate classroom. As part of this collaboration, the LSC was revised. The LSC is an on-going, curriculum-based observational checklist tool that is used to measure the critical components of emergent literacy (i.e., those skills which are best predictors of future success in reading and writing). Research indicates that the ability to read and write requires systematic, intentional teaching as it does not occur naturally (Neuman, Copple & Bredekamp, 2000; Chandler et. al., 2008). The LSC is research-based and was field-tested during a 12 week pilot study completed during the 2007-2008 school year. Analysis of the data by the SSD Office of Program Evaluation indicated strong internal consistency for the overall scale (Chronbach s alpha =.97), as well as for the individual subscales (Chronbach s alpha range =.87 to.90). In addition, results suggested a slight advantage for students participating in the Animated Literacy curriculum with regard to literacy skill growth during the 12 week pilot. Subsequent analysis conducted by collaborating faculty at St. Louis University also indicated a positive trend in favor of the efficacy of the Animated Literacy curriculum. ECSE and St. Louis University staff collaborated and presented the results of the pilot study in 2008 to the American Speech-Language-Hearing Association (ASHA), as well as the Missouri Speech-Language-Hearing Association. Results of the full study will be presented at the fall 2009 ASHA convention. During the 2008-2009 school year the LSC was administered to all students in ECSE classrooms, including IEP and Non-IEP students. IEP students served by itinerant ECSE teachers and speech pathologists were assessed with the exception of students with a diagnosis of speech only. These students are seen in a therapeutic setting which does not allow for observation in a functional educational setting as is needed to utilize the LSC. The checklist was administered at the beginning of the school year for returning students and within 30 days for new students entering during the school year. It was administered again at the end of the year to those students who received at least 6 months of intervention. Results of the pre and post assessments are presented in Table 1. Board Approved: 9/22/2009 4
Table 1. Literacy Skill Checklist Data (2008-2009) School Year LSC Scale Total LSC Interactions with Storybooks Emergent Book Reading Skills 08-09 Early Writing Skills Knowledge of Print/Alphabet Phonological Awareness Note: * p <.001 Student Average n Pre-Test Post-Test Group Increase t IEP 424 36.86 51.08 14.87 23.88* No IEP 212 56.05 70.92 14.22 17.12* IEP 424 12.70 16.02 2.63 18.22* No IEP 212 17.70 20.33 3.31 10.99* IEP 424 3.56 5.61 2.52 15.80* No IEP 211 6.91 9.43 2.05 12.67* IEP 422 6.24 8.60 1.97 15.82* No IEP 211 9.19 11.16 2.36 9.61* IEP 422 6.88 10.08 2.90 19.65* No IEP 211 10.87 13.77 3.20 13.80* IEP 424 7.53 10.76 4.89 15.67* No IEP 212 11.44 16.33 3.23 15.00* As indicated in Table 1, dependent t-test analysis of the LSC data indicated statistically significant growth for students with and without disabilities on the Total LSC scale, as well as individual LSC subscales. To more directly address the evaluation question regarding the percentage of students who demonstrated growth in literacy skills, additional analyses were conducted and are summarized in Table 2 below. Table 2. Percentage of Students Demonstrating Growth School Year 08-09 LSC Scale Total LSC Interactions with Storybooks Emergent Book Reading Skills Early Writing Skills Knowledge of Print/Alphabet Phonological Awareness Growth Demonstrated Student Group Total n n % IEP 424 386 91.0 No IEP 212 187 88.2 IEP 424 332 78.3 No IEP 212 144 67.9 IEP 424 268 63.2 No IEP 211 145 68.7 IEP 422 285 67.5 No IEP 211 105 49.8 IEP 422 326 77.3 No IEP 211 151 71.6 IEP 424 312 73.6 No IEP 212 169 79.7 As noted in Table 2, analysis of individual student scores indicated that 91% of students with an Board Approved: 9/22/2009 5
IEP increased their Total LSC score from pretest to posttest, compared to 88.2% of non-iep students. The percentage of students demonstrating growth (i.e., posttest gain) on individual LSC subscales was generally commensurate between groups in the areas of Emergent Book Reading Skills, Knowledge of Print/Alphabet and Phonological Awareness. A somewhat higher proportion of students with IEPs demonstrated growth on the Early Writing Skills and Interactions with Storybooks scales. What is generally the most promising data is the increase in knowledge of phonological awareness and knowledge of print/alphabet as those are the best predictors of future success in reading and writing (Neuman et. al., 2000). Despite these positive results, the data indicate that not all students demonstrated growth. Additional analysis and exploration of the data via data teams are planned to assess characteristics of those students who did not demonstrate literacy skill growth. Parent Survey: Student Literacy Skills A parent survey (Appendix B) was developed and distributed in the spring of 2009 to assess the degree to which parents witness their child demonstrating early literacy skills. Survey items mirrored the components and items that comprise the LSC. Surveys were sent to all the parents of children who participated in the LSC. The intent of the survey was to determine the degree to which parents saw their child demonstrating various literacy skills. Approximately 630 surveys were sent to parents and 483 were returned resulting in a 76.7 % return rate. Results of the survey are reported below in Table 3. Table 3. Parent Survey: Student Literacy Skills Percent of Responses (n=483) Question No Response Rarely Sometimes Frequently Handles books properly 0 3.1 14.1 82.8 Participates in storybook reading 0 17.0 30.6 52.4 Tells title/name of favorite/familiar book 0.6 20.3 26.1 53.0 Likes to scribble, draw or write 0 8.1 22.6 69.4 Copies/traces/writes letter of name 0.2 21.7 21.9 56.1 Recognizes words/logos in environment 0.2 6.4 12.0 81.4 Recognizes/names any letter in alphabet 0.4 12.4 21.1 66.0 Attends to/participates in songs 0.4 6.4 17.8 75.4 Makes sound:letter / letter:sound connections 1.4 27.7 31.7 39.1 Question No Response Yes No Has shown increased interest in literacy skills since beginning early childhood services 2.7 88.6 8.7 What is particularly promising from this information is the parents awareness of their child s ability to recognize words/logos in the environment as well as their book handling skills. These are functional skills that are easily observed day to day. In addition, 88.6% of parents indicated that their child had demonstrated increased interest in literacy skills since beginning the ECSE Board Approved: 9/22/2009 6
program. A concern was noted with regard to the proportion of parents who indicated they rarely observed their child making sound-letter connections (i.e., 27.7%). As this is an important building block to reading, ECSE will provide parents with information regarding literacy development in preschool children as it relates to phonemic awareness. V. Summary Strengths The ECSE Department is pleased with the gains in acquisition of literacy skills as a result of implementing the Animated Literacy Curriculum, a more focused, research-based curriculum. The gains our students have made in crucial emergent literacy skills will be a benefit as they enter kindergarten. Preparing them to have the building blocks necessary for reading in their later school years is part of our mission. In addition, the parents of children in the ECSE program are seeing the benefits of intentional literacy instruction. Concerns Despite positive results, the data indicate that not all students demonstrated growth in literacy skill domains as indicated on the LSC. In addition, a concern was noted with regard to the proportion of parents (27.7%) who indicated they rarely observed their child making sound-letter connections. Recommendations Additional analysis and exploration of the data via data teams are planned to assess characteristics of those students who did not demonstrate literacy skill growth. ECSE will also provide parents with information regarding literacy development in preschool children as it relates to the importance of phonemic awareness. To continually improve instruction for all of our students, ECSE will also (a) track students who participated in the Animated Literacy and Breakthrough to Literacy program to see how they performed on MAP/MAP-A in 3 rd grade in the area of Communication Arts, (b) continue to review the most current research on literacy skill development in children during the preschool years and inform staff of best practice in the area and (c) continue to measure student progress in the area of literacy as well as other developmental domains and work with staff on data gathering, analysis, and instructional decision making using data. VI. Action Plan for Recommendations as a Result of Program Evaluation Person responsible to champion action plan: Martha Disbennett Timeframe for reporting updates to Board of Education: Biannually Date: Signature of Administrator Responsible for Chairing Evaluation Board Approved: 9/22/2009 7
References Neuman, S., Copple, C. & Bredekamp, S. (2000). Learning to read and write. Washington, DC: National Association for the Education of Young Children. Chandler, L. et. al. (2008). Promoting early literacy skills within daily activities and routines in the preschool classroom. Young Exceptional Children, 11. Board Approved: 9/22/2009 8
Appendix A Literacy Skills Checklist Board Approved: 9/22/2009 9
Please use UPPER CASE letters only. Early Literacy Skills Checklist A 4 1. Encore Student I. D. Number: (leave blank for non-iep students) xxxxxxxxxx 2. Student Name: (Format: FIRST NAME LAST NAME) xxxxxxxxxxxxxxxxxxxxxxxxxxxx 3. Date of Birth: xx/xx/xxxx 4. Staff Name: (Format: FIRST NAME LAST NAME) xxxxxxxxxxxxxxxxxxxxxxxxxxxx 5. Date of Test: xx/xx/xxxx 6. District xxxxxxxxxxxxxxxxxxxxxxxxxxxx 7. School xxxxxxxxxxxxxxxxxxxxxxxxxxxx 8. Student's age: (Mark one) $ 3 years old $ 4 years old $ 5 years old 9. Mark one: $ Typical Peer $ IEP Student $ Control Subject 10. For IEP students, mark all areas in which the student qualified for special education services: $ Communication $ Cognition $ Motor $ Adaptive Behavior $ Social/Emotional Behavior 11. For IEP students with School Age Diagnosis, mark all areas that apply. $ Not Applicable $ Autism $ Deaf/Blind $ Emotional Disturbance $ Hearing Impairment/Deafness $ Mental Retardation/Intellectual Disability $ Multiple Disabilities $ Orthopedic Impairment $ Other Health Impairment $ Learning Disability $ Language Impairment $ Sound System Disorder $ Speech/Fluency $ Speech/Voice $ Traumatic Brain Injury $ Vision Impairment $ Young Child with Developmental Delay 8 0 1 A48 Page 1 Copyright 2007 by Ackerson, C., Saint Louis University and Brangle, T., Special School District of St. Louis County, ECSE.
Early Literacy Skills Checklist A 4 12. Mark the item that best describes the curriculum/literacy instruction that this child is receiving. $ Project Construct with Animated Literacy $ Project Construct with Breakthrough to Literacy $ Project Construct with no formal literacy instruction $ Creative Curriculum with Animated Literacy $ Curriculum/literacy instruction other than those described above $ No preschool curriculum or formal literacy instruction 8 0 2 A48 Page 2 Copyright 2007 by Ackerson, C., Saint Louis University and Brangle, T., Special School District of St. Louis County, ECSE.
KEY: Early Literacy Skills Checklist A 4 (0) = Does not exhibit; not observed (1) = Exhibits inconsistently; the skill is emerging in the presence of support / cues / prompts (2) = Exhibits consistently and independently (without support / spontaneously) at least 80% of time *NOTE: Receptive or Expressive skills may be demonstrated via oral or augmentative communication. Behavior / Skill = (A1 thru E10) Allow child to look at books independently A1. Shows interest in books... 0 1 2 A2. Exhibits book handling skills... A3. Points to / shows pictures in books... Child Interactions -- 1:1 or group A4. "Listens attentively" while being read to (1:1)... A5. Recognizes a familiar or favorite book by cover... A6. Points to a WORD... A7. Participates in interactive readings... A8. Labels/describes or responds to who, what, what doing and/or where questions... A9. Makes comments or responds to questions related to prediction, character's emotions/plans and/or cause-effect relations... A10. Connects information and events in books to self or life experiences; adds personal information... Observation during group readings A11. Attends to group shared storybook readings... 8 0 3 A48 Page 3 Copyright 2007 by Ackerson, C., Saint Louis University and Brangle, T., Special School District of St. Louis County, ECSE.
KEY: Early Literacy Skills Checklist A 4 (0) = Does not exhibit; not observed (1) = Exhibits inconsistently; the skill is emerging in the presence of support / cues / prompts (2) = Exhibits consistently and independently (without support / spontaneously) at least 80% of time *NOTE: Receptive or Expressive skills may be demonstrated via oral or augmentative communication. Familiar/favorite story book "reading". Direct child to pick her/his favorite book or the one s/he wants to read from a choice of 3-4 books. B1. Tells the title/name of a favorite OR familiar book... 0 1 2 B2. Tells who the book is about / characters... B3. Identifies setting of familiar story in response to "Where/When did the story happen?"... B4. "Reads" familiar/favorite story in response to "Read your book to me" - with the book present... Retelling a story B5. Mimics or repeats phrases or actions from a familiar story - without the book present - during play or extension activities... B6. Retells familiar story to include 2-3 simple events - without the use of the book - but using puppets, flannel board, art materials, or other visual aids.... 8 0 4 A48 Page 4 Copyright 2007 by Ackerson, C., Saint Louis University and Brangle, T., Special School District of St. Louis County, ECSE.
KEY: Early Literacy Skills Checklist A 4 (0) = Does not exhibit; not observed (1) = Exhibits inconsistently; the skill is emerging in the presence of support / cues / prompts (2) = Exhibits consistently and independently (without support / spontaneously) at least 80% of time *NOTE: Receptive or Expressive skills may be demonstrated via oral or augmentative communication. 0 1 2 C1. Watches when others write... C2. Intentionally draws simple pictures to represent something; need not be recognizable by an adult... C3. Make random markings/scribbles with writing tool... C4. Uses scribble writing AND some letter-like forms... C5. Copies / traces the first letter of own name using writing tool... C6. Writes 2 or more recognizable letters... D1. RECOGNIZES / POINTS to familiar logos... 0 1 2 D2. "Reads" print in community/classroom environment... D3. Recognizes that print runs left to right... D4. Use print vocabulary such as "READ", "WORD", "LETTER", or "WRITE"... D5. Recognizes and/or labels beginning/first alphabet letter in own name... D6. RECOGNIZES own name in print... D7. LABELS / NAMES at least 5 lower and/or upper-case alphabet letters... D8. RECOGNIZES 12+ lower and/or upper-case alphabet letters... 8 0 5 A48 Page 5 Copyright 2007 by Ackerson, C., Saint Louis University and Brangle, T., Special School District of St. Louis County, ECSE.
KEY: Early Literacy Skills Checklist A 4 (0) = Does not exhibit; not observed (1) = Exhibits inconsistently; the skill is emerging in the presence of support / cues / prompts (2) = Exhibits consistently and independently (without support / spontaneously) at least 80% of time *NOTE: Receptive or Expressive skills may be demonstrated via oral or augmentative communication. *Up to 3 practice items may be administered for each task (E1-E11). During practice, maximal cueing, support, and explanation may be used; however, practice stimuli must be different than test stimuli. 0 1 2 E1. Babbles and/or plays with the sounds of language... E2. Attends to rhymes, songs, and nonsense word play... E3. Participates vocally OR verbally in songs... E4. Segments syllables in word... E5. Rhyme awareness - completes the rhyming word in nursery rhyme line, poem, song, or book... E6. Rhyme matching - detects the word from a series of 3 pictures that rhymes with the target word... E7. Rhyme generation - adds word/non-word to rhyme pattern... E8. Alliteration matching - detects the word from a series of 3 pictures that starts with the same sound as the target word... E9. Alliteration generation - produces a word beginning with the same sound as a target word... E10. Makes sound-letter connections... E11. Makes letter-sound connections... 8 0 6 A48 Page 6 Copyright 2007 by Ackerson, C., Saint Louis University and Brangle, T., Special School District of St. Louis County, ECSE.
Appendix B Parent Survey: Student Literacy Skills Board Approved: 9/22/2009 10
D Early Childhood Parent Survey: Literacy B 1. Level of service: $ Early Childhood Special Education Classroom $ Preschool / Daycare Classroom $ Itinerant Special Education Services 2. District: $ Affton $ Bayless $ Brentwood $ Hancock Place $ Jennings $ Lindbergh $ Maplewood-Richmond Heights $ Ladue $ Normandy $ Parkway $ Ritenour $ Riverview Gardens $ Valley Park $ Webster Groves $ Wellston 3. Mark One: $ Student with IEP $ Student without IEP Families: Please answer the questions below about your child. Return the completed survey to your child's teacher as soon as possible. Response Definition: R=Rarely S=Sometimes F=Frequently 4. Handles books properly (e.g., holds book upright, turns pages from front to back)... 5. Participates in interactive readings (e.g., fills in words or lines from a story, joins in with predictable phrases)... 6. Tells the title/name of a favorite or familiar book... 7. Likes to scribble, draw or write.... 8. Copies/traces or writes any letter in his/her name with a writing tool (e.g., pen, marker, crayon).... 9. Recognizes words or logos in the environment (e.g., identifies Cheerios vs. Fruit Loops, shouts "McDonald's!" when driving by in the car)... 10. Recognizes or names any letter(s) in the alphabet (e.g., points to the letter "A" when you ask him/her to or says the name of letters in his/her name).... 11. Attends to or participates in songs (e.g., smiles when you sing, fills in the word "boat" when you start singing "Row, Row, Row your ")... 12. Makes 'sound:letter' or 'letter:sound' connections (e.g., names the appropriate letter when you say "What letter says mmm?" or makes the correct sound when you say "What does the letter 'B' say")?... R S F R S F R S F R S F R S F R S F R S F R S F R S F 13. Has your child shown an increased interest in pre-reading and writing skills (such as the skills addressed in this survey) since he/she began receiving early childhood services? $ Yes $ No 14. Please add any information you wish regarding your child's interest in reading, writing or school: DB3 Page 1 3 0 1