Literacy Assessment Field Test Data Analysis: Evaluation Report

Transcription

1 Getting on Track Early for School Success Literacy Assessment Field Test Data Analysis: Evaluation Report NOVEMBER 2013 Literacy Assessment Field Test Data Analysis: Evaluation Report. (November 2013). Getting on Track Early for School Success:

2 Team Lead Analyst: Michele Zimowski, PhD, Senior Survey Methodologist, Department of Statistics and Methodology, NORC, University of Chicago Committee on Education, University of Chicago 1. Stephen Raudenbush, EdD, PI, Getting on Track Early for School Success, Professor, Department of Sociology and Chairman of the Committee on Education, University of Chicago 2. Terese Schwartzman, PhD, Project Director, Getting on Track for School Success, Committee on Education, University of Chicago 3. Jennifer Adams, PhD, Senior Assessment Development Associate Literacy; Committee on Education, University of Chicago Academic Departments, University of Chicago 1. Susan Goldin-Meadow, PhD, Professor, Departments of Comparative Human Development and Psychology and Committee on Education, University of Chicago 2. Susan C. Levine, PhD, Professor, Departments of Comparative Human Development and Psychology and Committee on Education, University of Chicago Urban Education Institute (UEI), University of Chicago 1. Molly Branson Thayer, PhD, Director for Strategic Initiatives, Urban Education Institute, University of Chicago 2. Margaret Walsh, MAT, Director of STEP Literacy, Urban Education Institute, University of Chicago NORC, University of Chicago 1. Marc Hernandez, PhD, Senior Research Scientist, Academic Research Centers, NORC, University of Chicago 2. Michael Reynolds, PhD, Senior Deputy Director for Research, Academic Research Centers, NORC, University of Chicago LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT I

3 Table of Contents Summary... 1 Classical Item Analysis Literacy Field Test... 2 IRT (Item-Response-Theoretic) Analysis Literacy Field Test... 3 For Further Information Related Papers Appendix A: Pre-K Literacy Classical Item Analysis LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT II

4 List of Exhibits Exhibit 1: Exhibit 2: Exhibit 3: ICCs for Word Morphology Form A...4 ICCs for Rhyming Form A...5 ICCs for Upper-Case Letter Identification...5 Exhibit 4: Typical Item Characteristic Curves for a Two-Parameter IRT Model...6 Exhibit 5: Probability of a Correct Response to Each Word Morphology Item as a Function of Age...7 Exhibit 6: Probability of a Correct Response to Each Rhyming Item as a Function of Age...7 Exhibit 7: Probability of a Correct Response to Each Upper-Case Letter Identification Item as a Function of Age...8 Exhibit 8: Proficiency as a Function of Age for Three Measures in the Assessment Instrument...9 Exhibit 9: Correlations Among the Proficiency Scores...9 Item Statistics for Concepts about Print...11 Item Statistics for Receptive and Expressive Rhyming Form A...11 Item Statistics for Receptive and Expressive Rhyming Form B...12 Item Statistics for the Uppercase Letter Identification...12 Item Statistics for Lowercase Letter Identification...13 Item Statistics for Letter-Sound Correspondence...13 Item Statistics for Listening Comprehension Form A...14 Item Statistics for Listening Comprehension Form B...15 Item Statistics for Story Comprehension Form A...15 Item Statistics for Story Comprehension Form B...16 Item Statistics for Wordless Book Comprehension Form A...16 Item Statistics for Wordless Book Comprehension Form B...16 Item Statistics for Vocabulary and Concepts Form A...17 Item Statistics for Vocabulary and Concepts Form B...17 Item Statistics for Word Morphology Form A...18 Item Statistics for Word Morphology Form B...18 LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT III

5 Summary Promoting the development of oral language among preschool children is foundational to our work at the University of Chicago. We are developing a coherent system of literacy instruction that begins in pre-k and reliably enables children of all social, ethnic, and linguistic backgrounds to attain high levels of reading comprehension and academic achievement by grade 3. At the core of this system are statistically reliable assessments that integrate research and practice and provide information to preschool teachers that is highly relevant to individual, group, and whole class instruction. Our approach is grounded in research demonstrating that focusing on early oral language development positively influences children s later proficiency in reading comprehension, writing and numerical reasoning (Dickinson, McCabe, Anastasopoulos, Peisner-Feinberg, Poe, 2003; Dickinson & Neuman, 2006; Dickinson & Tabors, 2001; Griffin, Hemphill, Camp and Wolf, 2004; Scarborough, 2001; Gunderson & Levine, 2011). Recent preschool evaluations find that these oral language skills are not sufficiently emphasized in preschool instruction (Castro, Espinosa, & Paez, 2011). We seek to change this state of affairs by clarifying instructional goals for the development of critical early language and print related skills in preschool settings and providing teachers with frequent objective, accurate and valid assessments of these. This paper reports on the Classical Item and Item-Response-Theoretic (IRT) analyses of the Pre-K Oral Language and Literacy Assessment field test data as of August The field test occurred between October and December The assessment was administered to 417 students ranging in age from 37 to 67 months. The racial and ethnic composition of the field test sample was: 59.6% African American; 20.8% White; 8% Hispanic; and 11.6% Other. The validation of the assessment included administration of the following gold-standard assessments of preschool literacy skills: the Woodcock-Johnson-III (WJ- III) and the Receptive One-Word Picture Vocabulary Test 4 (ROWPVT-4). Data analysis is scheduled to be complete by the winter of For more details on the Pre-K Literacy Assessment field test, please see Getting on Track Early for School Success: Project Overview. LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 1

6 Classical Item Analysis Literacy Field Test As the first step in evaluating the operating characteristics of the literacy assessment instrument in the field-test sample of preschool aged children, we computed classical item and test statistics for most measures in the instrument. The results from this initial classical item analysis are included in Appendix A. They summarize information at the level of individual measures for all print-based and nearly all oral language measures in the instrument and classical statistics for the individual items in the instrument. For each measure, the results include the: number and type of items in the measure number of children who completed it estimate of the internal consistency of the items (alpha coefficient of reliability) average correlation among the items range of item p-values1 age range and median age in months of the preschoolers at the time they completed the measure It also presents classical statistics for the individual items in the instrument. For each item in each measure, it includes: the number of children who attempted the item (#tried) the number who answered the item correctly (#right) the percent or p-value of children who answered the item correctly (#right/ #tried) a logit transform of the item s p-value2 point-biserial (Pearson) and biserial item-test correlations, which indicate the degree to which the responses to the item correlate with the sum of correct responses to the other items in the measure The results from the classical analysis indicate that our measures of print-based literacy and oral language performed quite well in the field test sample of preschool children. Of the 278 multiple-choice and freeresponse items in the instrument, all but five items fell within the desired range of item difficulty (i.e., p- value 0.07 and 0.93). The difficulty level of the insdividual items within most measures varied widely, covering the range of proficiencies within our large sample of children. With few exceptions, each measure included some items that proved to be relatively difficult for the sample of preschool children. These results suggest that the difficulty range of our pool of items is well suited for measuring LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 2

7 the range and mix of proficiencies in the target population of low-income preschoolers as well as growth in the children s skills as they benefit from classroom instruction and attain higher levels of proficiency. The responses to the individual items also displayed desirable Pearson item-test correlations. These correlations serve as an index of each item s power to discriminate among individuals at higher and lower levels of proficiency. Of the 278 multiple-choice and free-response items in the instrument, only nine items exhibited Pearson item-test correlations below 0.20, a cutoff commonly used in samples of this size to identify items potentially in need of revision (e.g., Schmeiser and Welch, 2006). Close to 90 percent of the items displayed relatively strong (0.30 to 0.49) to very strong (above 0.50) item-test correlations, with over 40 percent of items in the very strong range, indicating that the items are capable of discriminating among preschool children of varying levels of proficiency. With the exception of Story Comprehension, all measures yielded alpha coefficients above Reliability estimates of this magnitude are generally considered to be good to excellent. They are unusually high for measures with 30 or fewer items, especially when obtained with samples of preschool-aged children, and speak favorably for the measures we have developed. IRT (Item-Response-Theoretic) Analysis Literacy Field Test Preliminary results from fitting IRT models to the field-test data also reveal that the overwhelming majority of items in the literacy assessment instrument exhibit favorable measurement properties. To illustrate, the results from fitting a two-parameter IRT model to the item responses from three of the literacy measures in the instrument, Word Morphology, Rhyming, and Upper-Letter Identification, are displayed in Exhibits 1 through 3 in the form of item characteristic curves (ICCs). The ICC for each item describes in probabilistic terms what happens when children with varying levels of proficiency encounter the item on the assessment instrument. The ICC gives the probability of correct response to the item as a function of the characteristics of the item and a child s level of proficiency on the underlying construct, commonly referred to as θ (theta). The shape of each ICC and its placement along the proficiency scale depend on the estimates of the item s parameters (location and slope) obtained in the IRT analysis. The location (or difficulty) parameter of an item refers to the point along the proficiency where a child has a 0.5 probability of answering the item correctly. The slope parameter refers to the steepness of the ICC in the middle section of the curve. It measures how well the item discriminates between children with proficiencies above and below the item s location on the proficiency scale. Items with steeper curves have more discriminating power than items with flatter curves. Exhibit 4 displays typical ICCs found in LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 3

8 practice with the two-parameter model. The dashed lines in the exhibit show that items 1 and 3 have a difficulty level of -1.0 on the theta scale, while Item 2 has a difficulty level of 0.0 on the scale. Item 2 displays a rather high discrimination of 2.0, Item 3 an average discrimination of 1.0, and Item 1, a lower than average discrimination of 0.5. Examining Exhibits 1 through 3, we see that the Word Morphology items, which cover a broad range of word structures, vary widely in their difficulty levels ranging in value from to 2.88 on the theta scale. As expected, the Rhyming items, which focus on a specific skill, display considerably less variation in their item difficulties. The same holds true for the Upper-Case Letter Identification items. The slopes for most items in these measures fall within the average to high range. As expected, the slopes for the Word Morphology items tend to be a bit lower than the slopes for items in the other two measures, which focus on specific skills. With the exception of the slope for item WMA11, the items as a whole exhibit desirable measurement properties suitable for developing booklets adapted to the proficiency levels of individual children in a pre-school classroom. Exhibit 1: ICCs for Word Morphology Form A LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 4

9 Exhibit 2: ICCs for Rhyming Form A Exhibit 3: ICCs for Upper-Case Letter Identification LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 5

10 Exhibit 4: Typical Item Characteristic Curves for a Two-Parameter IRT Model To examine the feasibility of using the age of a child as a proxy for proficiency in the selection of items for the baseline booklets for each child, we fit a logistic model to the responses to each individual item in the Word Morphology, Rhyming, and Upper-Letter Identification measures of the assessment instrument. Age in months at the time of testing served as the predictor variable in each of these analyses, with the item responses, scored as right and wrong, as the dependent variable. From the parameter estimates obtained in these analyses, we generated the curves displayed in Exhibits 5 through 7. The curves show the probability of a correct response to each item in a measure as a function of a child s age in months. Although the relationship between age in months and the probability of a correct response was statistically significant for most of the items examined here, the values of the slope parameters tended to be low, especially in the case of Word Morphology. The results of these analyses suggest that age in months, especially in the case of Word Morphology, provides only a limited amount of information on how a given child will respond to any particular item in these measures. Nonetheless, the results demonstrate that older children tend to have more developed skills in these areas than their younger counterparts. LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 6

11 Exhibit 5: Age Probability of a Correct Response to Each Word Morphology Item as a Function of Exhibit 6: Probability of a Correct Response to Each Rhyming Item as a Function of Age LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 7

12 Exhibit 7: Probability of a Correct Response to Each Upper-Case Letter Identification Item as a Function of Age We also examined the linear relationship between the estimates of proficiency obtained in the IRT analysis and the age of the children at the time of testing. Exhibit 8 shows the results from this regression analysis. It reveals that age in months at the time of testing displays a weak, but positive linear relationship with proficiency, accounting for about 10 percent of the variance in the proficiency scores on each measure. In interpreting these results, it is important to note that the proficiency scores, as well as the underlying distributions, for Rhyming and Upper-Case Letter Identification are not normally distributed. The rhyming scores displayed bimodality, suggesting that the ability to rhyme may be an all or none phenomenon. Inspection of the proficiency scores for Upper-Case letter Identification, on the other hand, revealed that nearly 30 percent of the children in the field-test sample were able to correctly identify all 26 upper-case letters. LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 8

13 Exhibit 8: Proficiency as a Function of Age for Three Measures in the Assessment Instrument In addition, we looked at the correlations among the proficiency scores on the three measures. They are presented in Exhibit 9. Again, it is important to consider that the proficiency scores for Rhyming and Upper-Case Letter Identification are not normally distributed. Exhibit 9: Correlations Among the Proficiency Scores Measure Word Morphology Rhyming Upper-case Letter Identification Word Morphology Rhyming Upper-Case Letter Identification The IRT analyses presented above represent three of the seven literacy domains. LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 9

14 For Further Information Stephen Raudenbush, Project PI, Professor, Department of Sociology, the College, and the Harris School of Public Policy Studies; Chair, Committee on Education, University of Chicago, Jennifer Adams, Senior Assessment Development Associate for Literacy, Committee on Education, University of Chicago, Michele Zimowski, Senior Survey Methodologist, NORC at the University of Chicago, Zimowski- Marc Hernandez, Senior Research Scientist, NORC at the University of Chicago, Hernandez- Related Papers 1. Getting on Track Early for School Success: Project Overview. (November 2013). Getting on Track Early for School Success: 2. Research and Practice in the Field of Early Literacy Learning. (November 2013). Getting on Track Early for School Success: 3. Research and Practice in the Field of Early Mathematics Learning. (November 2013). Getting on Track Early for School Success: 4. From the Classroom to the Lab and Back: Instructional Strategies to Improve Children s Early Math Skills. (November 2013). Getting on Track Early for School Success: LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 10

15 Appendix A: Pre-K Literacy Classical Item Analysis Item Statistics for Concepts about Print 1 CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP CAP Item Statistics for Receptive and Expressive Rhyming Form A 1 RIA RIA RIA RIA RIA RIA RIA RIA RPA RPA RPA RPA RPA RPA RPA LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 11

16 Item Statistics for Receptive and Expressive Rhyming Form B 1 RIB RIB RIB RIB RIB RIB RIB RIB RPB RPB RPB RPB RPB RPB RPB Item Statistics for the Uppercase Letter Identification 1 LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LUC LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 12

17 Item Statistics for Lowercase Letter Identification 1 LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC LLC Item Statistics for Letter-Sound Correspondence 1 LSND LSND LSND LSND LSND LSND LSND LSND LSND LSND LSND LSND LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 13

18 Item Statistics for Listening Comprehension Form A 1 LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LCA LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 14

19 Item Statistics for Listening Comprehension Form B 1 LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB LCB Item Statistics for Story Comprehension Form A 1 SCA SCA SCA SCA SCA SCA SCA SCA LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 15

20 Item Statistics for Story Comprehension Form B 1 SCB SCB SCB SCB SCB SCB SCB SCB Item Statistics for Wordless Book Comprehension Form A 1 WPA WPA WPA WPA WPA WPA WPA WPA WPA Item Statistics for Wordless Book Comprehension Form B 1 WPB WPB WPB WPB WPB WPB WPB WPB WPB LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 16

21 Item Statistics for Vocabulary and Concepts Form A 1 VMA VMA VMA VMA VMA VMA VMA VMA VMA VMA VCA VCA VCA VCA VCA VCA Item Statistics for Vocabulary and Concepts Form B 1 VMB VMB VMB VMB VMB VMB VMB VMB VMB VMB VCB VCB VCB VCB VCB LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 17

22 Item Statistics for Word Morphology Form A 1 WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA WMA Item Statistics for Word Morphology Form B 1 WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB WMB LITERACY ASSESSMENT FIELD TEST DATA ANALYSIS: EVALUATION REPORT 18