Examining the Validity and UHlity of an Assessment of Preschool Early Literacy Skills

Transcription

1 Examining the Validity and UHlity of an Assessment of Preschool Early Literacy Skills Preschool Early Literacy Indicators (PELI) DIBELS Literacy Summit Santa Ana Pueblo March 7, 2013 Ruth Kaminski, Ph.D. Mary AbboD, Ph.D. Katherine Bravo- Aguayo Rachael LaHmer Roland Good Dynamic Measurement Group

2 Overview Overview of PELI Overview of research on PELI Research results Preliminary benchmark goals and next steps

3 Acknowledgements: PELI Team Project Co- Director: Mary AbboD Project Coordinator: Katherine Bravo- Aguayo Data Analysts: Roland Good, Rachael LaHmer Research Assistants: Sara Gomez Annie Hommel Debby Laimon Rose McMahon Analisa Madrid Doug Rice Amy Warnock

4 The Issue Up to 40% of children enter kindergarten behind peers in emergent literacy skills that are foundahons to school success (Bailet et al., 2009). Reading difficulhes are easier to prevent than to remediate (Berninger et al., 2002; Coyne et al., 2004; Torgesen, 2002). Emergence of RTI model in early childhood shows promise for meehng needs of all children (Ball & Trammell, 2011; Greenwood et al., 2011) Need for reliable, valid measures of early literacy skills that are appropriate for preschool- age children that can be used for screening and progress monitoring.

5 Background What is the PELI? Screening and progress monitoring assessment of foundahonal early literacy skills for preschool children ages 3 6. Skills embedded in story- book format Alphabet knowledge Phonological awareness Vocabulary and oral language Comprehension

6 Book format Assessor sits beside child and looks at book with child. Assessor gives direchons and asks queshons following standard prompts. Skills assessed through series of book- related achvihes. PELI Overview

7 Key Features of Stories Stories were wriden and illustrated to represent a broad range of scenarios and achvihes familiar to preschool students Characters have diverse idenhhes (family structure, ethnicity, disability status) Average administrahon Hme about 11 minutes

8 SecHons Alphabet knowledge Name upper- and lower- case leders Comprehension Answer queshons about story Make predichons and inferences Recall details from story (CLOZE task) Phonological awareness IdenHfy first syllable in two- syllable words IdenHfy first sound in single- syllable words Vocabulary/oral language Names pictures Tells about pictures/words

9 Current PELI Forms What s for Dinner? On the Farm Off to the Grocery Store A Day at the Beach Grandma s Birthday A New Pet A Trip to Outer Space Time for Bed A Day at the Playground A Special Day at School

10 3 Sample Book 2 7 & * - 74,7* , 472

11 Research on PELI Study 1: n = DescripHve stahshcs Concurrent Validity Study 2 n = 32 Reliability Inter- rater Alternate form Test- retest Study 3 n = 165 DescripHve stahshcs Concurrent and predichve validity Study 4 n = 44 Comparison of versions Study 5 n = 136 Vocabulary study Study 6 n = 258 Preliminary benchmark goals

12 Study 1: DescripHve StaHsHcs

13 Study 1: Concurrent CorrelaHons with SNRTs

14 Study 1: Concurrent CorrelaHons with IGDIs and DIBELS LNF

15 Study 2: Alternate Form Reliability Form Note: CorrelaHons are based on subjects with pair- wise complete data. All correlahons are stahshcally significant at the p <.01 level. AK range = , PA range = ; VOL range = ; Comp range = N = 32.

16 Study 2: Inter- rater Reliability PELI Subtest Form 1 Form 2 Form 3 Form 4 Form 5 Overall AK PA VOL Comp Total Note: All correlahons are stahshcally significant at the p <.001 level.

17 PELI Research, : MI Site DescripHve data on PELI - BOY, MOY, EOY Concurrent correlahons with CELF ELI, DIBELS LNF, FSF, DCS PredicHve correlahons with CELF ELI, DIBELS LNF, FSF, DCS

18 Study 3: Concurrent and PredicHve CorrelaHons with CELF and DIBELS PELI subtest - time of year Expressive Vocabulary CELF - end of year DIBELS measures - end of year Recalling Sentences ELI LNF FSF DCS Alphabet Knowledge - beg..35* **.44*.54** Alphabet Knowledge - mid **.77 Alphabet Knowledge - end **.83.49**.72 Comprehension total - beg **.46* Comprehension total - mid *.47** Comprehension total - end.43** *.32 Phonemic awareness total - beg..29* *.39* Phonemic awareness total - mid..35* * Phonemic awareness total - end.42** * Vocabulary/Oral Language total - beg..29**.37*.43**.41*.44*.48** Vocabulary/Oral Language total - mid..45** **.40* Vocabulary/Oral Language total - end.29*.36**.36**.14.44*.33 PELI Composite Score - beg ** PELI Composite Score - mid PELI Composite Score - end * Note.Data was collected from one district during the school year. Unless otherwise noted, correlations are statistically significant at the p <.001 level. ** = p <.01; * = p <.05; = not significant. Sample sizes for correlations with CELF measures range from 43 to 51. Sample sizes for correlations with DIBELS measures are 29.

19 Studies 4 and 5 Version Study Embedded phonemic awareness vs. phonological awareness game at end of story Embedded comprehension queshons versus queshons at end of story Vocabulary Study To enhance form equivalence and minimize floor/ ceiling effects 15 words per story selected based on word pool values and expert review Examined child responses to 15 words for each story to select 10 best words

20 Study 6: Preliminary Benchmark Goals CondiHonal probability of meehng goal on external criterion measure(s) CELF PreK Expressive Language Index (40 th %ile) DIBELS Next Kindergarten BOY Composite Score Primary specificahon: Establish goals whereby odds are favorable (80-90%) of achieving outcome Secondary specificahon: Keep marginal percents consistent (e.g., 40%)

21 Study 6: Preliminary Benchmark Goals Other considerahons: LogisHc regression predichng odds PaDern of student performance in scaderplot Receiver Operator CharacterisHc (ROC) curve analysis Metrics for decision uhlity: sensihvity, specificity, negahve predichve power, posihve predichve power, percent accurate classificahon, Kappa

22 Study 6: Preliminary Benchmark Goals Role Variable Goal Cut Point Description Screening Decision Predictor PCSe PELI Composite Score, End of Year Outcome Criterion celfeli CELF Short Form Expressive Language Index %! ",-- ",-- %# $!,/-,/-!"#$"#%& $#,.-,.- "!,#-,#- "#! Correlation =.68 "$% "&% $$% $&% '()*+,$-,-- "&" "!" $"" $%" $0",$- ",-- Scatterplot with benchmark goals (solid lines) and cut points for risk (dashed lines). Logistic regression with goal (solid dot) and cut point (open dot). Receiver Operator Characteristic (ROC) curves. celfeli Outcome: At or Above Benchmark % True Negative Below Benchmark % Fale Negative Well Below Benchmark % True Positive Marginal Total False Positive Marginal Percent 14% 27% 59% Sensitivity Odds (conditional percent) of students with screening decision achieving goal (At or Above Benchmark) PCSe Screening Decision: Likely to need intensive Likely to need strategic Likely to need core Marginal total Marginal percent At or Above Benchmark outcome Core decision Intensive decision Well Below Benchmark outcome Core decision Intensive decision specificity % 36% 83% Negative Predictive Power Positive Predictive Power Accurate Classification Kappa

23 Study 6: Preliminary Benchmark Goals Role Variable Goal Cut Point Description Screening Decision Predictor PCSe PELI Composite Score, End of Year Outcome Criterion DCSPe DIBELS Composite Score, Pre-K End of Year!"#$%& *! )! (! '! &! %! $! #! "!! Correlation =.69 "# '# ""# "'# #"# #'# +,-./ Scatterplot with benchmark goals (solid lines) and cut points for risk (dashed lines). "0!! 0)! 0'! 0%! 0#! 0!! "!* "&* #!* Logistic regression with goal (solid dot) and cut point (open dot). "0!! 0)! 0'! 0%! 0#! 0!! 0!! 0#! 0%! 0'! 0)! "0!! Receiver Operator Characteristic (ROC) curves. DCSPe Outcome: At or Above Benchmark % True Negative Below Benchmark % Fale Negative Well Below Benchmark % True Positive Marginal Total False Positive Marginal Percent 30% 26% 45% Sensitivity Odds (conditional percent) of students with screening decision achieving goal (At or Above Benchmark) PCSe Screening Decision: Likely to need intensive Likely to need strategic Likely to need core Marginal total Marginal percent At or Above Benchmark outcome Core decision Intensive decision Well Below Benchmark outcome Core decision Intensive decision specificity % 83% 81% Negative Predictive Power Positive Predictive Power Accurate Classification Kappa

24 Preliminary Benchmark Goals Composite Score BOY MOY EOY Alphabet Knowledge Comprehension Phonological Awareness Vocabulary

25 Benchmark Goal Study Students yr olds yr olds (addihonal 835 students in supplemental study) Sites Approximately 70 classrooms 5 states: IA, Ca, CT, MI, OR (supplemental sites in MI, PA, IN, IA, LA, CA)

26 For Further InformaHon: Ruth Kaminski Mary AbboD