Can reading accuracy and comprehension be separated in the Neale Analysis of Reading Ability?

Transcription

1 187 British Journal of Educational Psychology (2004), 74, The British Psychological Society Can reading accuracy and comprehension be separated in the Neale Analysis of Reading Ability? Alice L. R. Spooner 1 *, Alan D. Baddeley 2 and Susan E. Gathercole 3 1 Department of Psychology, University of Liverpool, UK 2 Department of Experimental Psychology, University of Bristol, UK 3 Department of Psychology, University of Durham, UK Background. The Neale Analysis of Reading Ability (NARA) (Neale, 1997) is widely used in education and research. It provides measures of reading accuracy (decoding) and comprehension, which are frequently interpreted separately. Aims. Three studies were conducted to investigate the degree to which the NARA measures could be separated. Samples. British 7- and 8-year-olds participated in Study 1 (N=114) and Study 2 (N=212). In Study 3, 16 skilled and less-skilled comprehenders were identified from the Study 2 sample. Methods. Study 1: By investigating their contribution to silent reading comprehension, the independence of NARA decoding and comprehension scores was determined. Study 2: Decoding groups matched for listening comprehension were compared on the NARA comprehension measure, and population performance was compared across listening comprehension and NARA reading comprehension. Study 3: Comprehension groups were compared on ability to answer open-ended and forcedchoice questions. Results. Firstly, NARA comprehension performance depended on decoding, to the extent that children with high listening comprehension ability but low decoding ability attained low NARA comprehension scores. Secondly, 32% of children who attained low NARA comprehension scores exhibited high listening comprehension. Thirdly, comprehension groups differed when assessed with open-ended questions but not when assessed with forced-choice questions. Conclusions. The NARA can underestimate the comprehension ability of children with weak decoding skills and children who have some difficulty with open-ended questions. The decoding and comprehension measures of the NARA cannot be separated. These findings have important implications for the interpretation of the measures provided by the NARA, in education and research. * Correspondence should be addressed to Alice Spooner, Department of Psychology, University of Central Lancashire, Preston, Lancashire PR1 2HE, UK ( [email protected]).

2 188 Alice L. R. Spooner et al. The Neale Analysis of Reading Ability (Neale, 1958, 1997; Neale, Christophers, & Whetton, 1989) provides standardized scores for reading accuracy and reading comprehension, and is widely used in education and research in English-speaking countries. Children read aloud stories, and the administrator counts and corrects any reading errors. When no more than 16 errors are made on a story, comprehension questions are asked. Children progress through stories of increasing difficulty until they make 16 or more reading errors on any one story. The total number of reading errors made throughout the test yields a reading accuracy score, and the number of questions answered correctly yields a comprehension score. The Neale Analysis of Reading Ability (NARA) is a useful tool of assessment in education. It provides a measure of children s ability to explain their understanding of texts that they can read. This set of skills closely resembles those required for successful reading for comprehension at school. This is particularly true of interactive situations, where children can ask for assistance, where questions can be rephrased when necessary, and where children s expressive speech is an important aspect of communication in the classroom. As a standardized and widely used test, it allows an individual child to be compared to the population. The two parallel versions of the test enable it to be used twice, to assess children s progress over time. Of particular interest to the authors was the use of the Neale Analysis of Reading Ability in research to identify children with particular profiles across reading accuracy (decoding) and reading comprehension. Such groups can then be compared with skilled groups to investigate cognitive differences that might lie at the root of the specific difficulty (Cain, 1999; Cain & Oakhill, 1999; Cain, Oakhill, & Bryant, 2000; Cain, Oakhill, & Sharp, 1998; Nation, Adams, Bowyer-Crane, & Snowling, 1999; Nation & Snowling, 1998a, 1998b; Oakhill, 1982; Oakhill, Yuill, & Parkin, 1986; Stothard & Hulme, 1992, 1995; Yuill, Oakhill, & Parkin, 1989). For example, less skilled comprehenders exhibit weak comprehension skills despite age-appropriate decoding. Frequently, comparisons are made between less skilled comprehenders and skilled comprehenders children who exhibit age-appropriate skills in both comprehension and decoding. A finding that the skilled and less skilled groups differ on some component process of comprehension may suggest that impairment on that process underlies the comprehension difficulties of the less skilled group. Although Marie Neale has never (to our knowledge) proposed that the decoding and comprehension components of the NARA are independent, researchers who use both measures to identify specific profiles make the assumption that they are. On truly independent measures, performance can vary on one measure without affecting performance on the other. However, employment of two dependent measures risks inaccurate assessment of individual ability across the two skills. For example, a child might focus effort into the reading accuracy component of the test at the cost of his/her comprehension, thus presenting the profile of a less skilled comprehender. The work presented here was conducted to ascertain the independence of the reading comprehension and the reading accuracy measures provided by the NARA. One might expect general comprehension ability (not reading comprehension in particular) and reading accuracy (decoding) to be largely independent. Comprehension refers to the construction of a mental representation of the information described. Decoding, on the other hand, is the retrieval of the auditory code of written words being able to read a word aloud. These are very different processes, and largely independent in terms of information processing. Consequently, apart from general

3 Reading accuracy and comprehension ability such as general intelligence or attention, general comprehension ability and decoding ability should be statistically independent. This viewpoint is supported by the Structure Building Framework of comprehension proposed by Gernsbacher (1990), which proposes that comprehension is not restricted to linguistic materials. The same processes are used in comprehension of any material, whether written text, spoken discourse, or a series of pictures. The majority of individuals have little difficulty accessing information in different modalities, for example, identifying objects in a picture or the meaning of single written words. In these circumstances reading comprehension, listening comprehension and picture comprehension will all be highly statistically related (Carr, Brown, Vavrus, & Evans, 1990; Gernsbacher, Varner, & Faust, 1990). However, in samples where some mode is problematic one would expect comprehension in that modality to be impaired. So, for example, in a group of individuals who exhibit particular difficulty interpreting speech sounds, listening comprehension would be expected to be lower than reading and picture comprehension. The Structure Building Framework can be used to make predictions about performance on the NARA. An individual who has good decoding skills would be expected to obtain a NARA comprehension score that is representative of his/her general comprehension ability. Such individuals would exhibit strong statistical dependence between NARA comprehension and listening comprehension (provided that their listening skills are also unimpaired). In contrast, an individual who has weak decoding skills would be expected to obtain a lower NARA comprehension score than is indicative of his/her general comprehension ability. This individual would be expected to demonstrate weak relationship between reading and listening comprehension. There are specific reasons to believe that weak decoding skills would impair comprehension performance on the NARA. Firstly, children with poor reading accuracy may be unable to derive sufficient text information for comprehension. Although reading errors are corrected during administration of the NARA, children who require frequent corrections may find the interruptions disruptive. They may also find it confusing to amend their representations of story information. The quality of information derived from text during slow, disjointed and interrupted reading may not, therefore, match the quality of information derived during smooth, focused reading. Thus reading accuracy would be expected to influence reading comprehension, despite the correction of errors. Secondly, children are expected to distribute effort between reading accuracy and reading comprehension. However, the act of decoding may be so effortful for children with weak decoding skills that they adopt a strategy of not actively engaging comprehension processes. It is therefore possible that poor decoders are impaired in their comprehension of the NARA because of the concurrent demands made by the reading component. A hypothetical scenario resulting from the concurrent demands was described earlier. Furthermore, children progress no further through the NARA once they make a prescribed number of reading errors. Consequently, the number of questions over which comprehension ability is assessed is dependent on reading accuracy. Children with low reading accuracy, therefore, are assessed over a restricted number of questions. They are therefore denied the chance to attain higher scores that may provide a more accurate reflection of their comprehension ability. Furthermore, the restricted comprehension scores that less skilled decoders obtain may provide an 189

4 190 Alice L. R. Spooner et al. underestimation of their ability to comprehend written text during assisted reading. Experience administering the NARA has revealed that even when children make a large number of decoding errors near the cut-off level, they still tend to be able to answer some questions correctly. It is therefore reasonable to believe that they would continue to comprehend the more difficult texts despite making many decoding errors. In short, it is proposed that general comprehension and decoding are separate skills, and that reading and listening comprehension will be strongly related in the absence of any reading or listening impairments. However, it is predicted that weak decoding skills will result in reduced reading comprehension ability. In this case, the relationship between reading comprehension and listening comprehension will be limited, and reading comprehension will be dependent on decoding ability. As discussed above, this would weaken the power of the NARA to identify specific profiles across decoding and comprehension. The work presented here was conducted to ascertain the extent to which performance on the decoding and reading comprehension components of the NARA could be separated. In Study 1, the independence of the decoding and reading comprehension components of the NARA were examined. In Studies 2 and 3, closer examinations of the comprehension measure were conducted. STUDY 1 In Study 1, a large sample of children was tested both on the NARA and on an assessment of silent reading comprehension. This latter test differed from the NARA primarily in that children were not provided with any assistance: reading errors were corrected in the NARA. On the silent reading test, therefore, children with weaker decoding skills were expected to take incomplete or inaccurate information from text. This would affect their comprehension. The silent reading comprehension test, therefore, tapped both reading accuracy and comprehension ability. The contribution of the reading accuracy and comprehension measures of the NARA to silent reading comprehension was investigated. If these measures were independent, they should each account for a significant proportion of unique variance in silent reading comprehension. Method Participants A total of 114 Year 3 children (65 girls and 49 boys) participated in Study 1. The sample ranged in age from 7 years 2 months to 8 years 5 months, with mean age 7 years 10 months (SD 4 months). The sample was taken from four urban schools in Bristol, South- West England. Whole classes were tested, to include the range of abilities present in mainstream schools. All classes may be assumed to have been taught according to the National Curriculum, and therefore to have received comparable literacy teaching. Materials All children were administered the reading accuracy and reading comprehension

5 Reading accuracy and comprehension 191 components of the NARA, Form 1 (Neale, 1997). To provide a measure of silent reading comprehension ability, six stories of increasing difficulty and 44 questions were either derived from the Diagnostic Tutor form of the NARA (Neale, 1997) or written to reflect the style of the NARA stories. Pilot work confirmed that the difficulty of the stories increased smoothly. Children read the stories and questions silently and independently from individual booklets. This enabled group administration to whole classes. The stories and questions were presented on separate pages, and children were not permitted to turn back to the stories when answering the questions. These were presented as forced-choice true/false statements, that children answered in the booklet. No assistance was available to children in the silent reading comprehension test. They were encouraged to guess when they did not know the answer. An example story and its accompanying questions are presented in Appendix A. Performance on the reading accuracy and reading comprehension components of the NARA were scored both as raw scores and reading ages. Silent reading comprehension was scored as number of items correct. Internal reliability of the silent reading comprehension test was investigated over the first four stories which all children completed, yielding Cronbach s =.67. Procedure Children were administered the silent reading comprehension test as a group, in their usual classes in one session. All children were required to complete stories 1 to 4, and children attempted stories 5 and 6 according to their individual abilities. Where a child was unable to make any progress through story 5 or 6, the teacher s opinion was sought on whether the story was within the child s reading ability. If the teacher considered the text to be within the child s ability, the child was instructed to complete that text. The NARA was then administered to each child individually, over the few days following the administration of the silent reading comprehension test. Results The mean reading age for the NARA accuracy measure was 8 years 3 months (SD 1 year 7 months). The mean reading age for the NARA comprehension measure was 7 years 8 months (SD 1 year 4 months). The sample mean score on the silent reading comprehension test was 29.0 (SD 5.8). The maximum score possible on the silent reading comprehension test was 44. First, a series of Pearson correlations was conducted on silent reading comprehension score, NARA accuracy score, NARA comprehension score and age. Silent reading comprehension score correlated significantly with both NARA accuracy score, r=.60, and NARA comprehension score, r=.47, both p<.001. NARA accuracy and NARA comprehension scores correlated significantly with each other, r=.59, p<.001. Age correlated non-significantly with silent reading comprehension score, r=.13, NARA accuracy score, r=.02, and NARA comprehension score, r=.11, all p>.05. Two linear hierarchical multiple regression analyses were then conducted, with silent reading comprehension score as the dependent variable, and NARA accuracy score and NARA comprehension score as predictors. The complete models accounted for 38.1% of the variance. Changes in R 2 associated with the entry of each variable are provided in Table 1. NARA accuracy contributed significantly to silent reading comprehension, independent of NARA comprehension. NARA comprehension did not account for variance in silent reading comprehension over and above NARA accuracy.

6 192 Alice L. R. Spooner et al. Table 1. Hierarchical multiple regression analyses conducted in Study 1, with silent reading comprehension as the dependent variable Step Added variable Change in R 2 Significance of change in R 2 1 NARA comprehension.22 p< NARA accuracy.16 p< NARA accuracy.36 p< NARA comprehension.02 non-significant Discussion The Neale Analysis of Reading Ability offers separate assessments of reading accuracy and reading comprehension, but these measures were not found to be independent. In particular, NARA comprehension scores accounted for no unique variance in silent reading comprehension when reading accuracy was controlled, indicating that NARA comprehension score was dependent on NARA accuracy score. This finding has two possible explanations. Firstly, comprehension ability might be generally dependent on decoding ability. This would suggest that listening comprehension and picture comprehension as well as reading comprehension would all be dependent on decoding ability. This would be contrary to the predictions made earlier. Alternatively, this pattern of dependence may be peculiar to reading comprehension, so that reading comprehension is dependent on reading accuracy, without generalizing to other modalities. Study 2 was conducted to decide between these two explanations. In Study 2 the relationship between decoding and comprehension was examined using both the NARA and an alternative measure of comprehension. Children with low and high decoding abilities but matched for listening comprehension ability were identified. Performance on the NARA comprehension measure was compared between the decoding groups. If the results found in Study 1 generalize across comprehension measures, the relative performance of the decoding groups should remain consistent across comprehension measure. In other words, because the groups were matched on listening comprehension, they should score equivalently on NARA comprehension. However, if the results of Study 1 are specific to the NARA, decoding should mediate NARA comprehension: despite being matched for listening comprehension, the decoding groups should be separated on NARA comprehension. A strong relationship has been demonstrated between comprehension performance on the NARA and on tests of listening comprehension (e.g., Nation & Snowling, 1997; Stothard & Hulme, 1992). However, if NARA comprehension is dependent on decoding ability (as suggested by Study 1) children with weak decoding skills should perform at a lower level on the NARA comprehension measure than on a listening comprehension test which makes no decoding demands. Consequently the NARA might underestimate their general comprehension ability. Therefore, the second aim of Study 2 was to identify whether the NARA is a valid estimate of general comprehension ability in comparison with a listening comprehension test. A large sample was tested to survey a range of abilities. Children were divided into subgroups according to their performance across the NARA and listening comprehension measures. Thus children whose comprehension ability was underestimated by the NARA would form a subgroup attaining lower NARA

7 Reading accuracy and comprehension 193 comprehension scores than their listening comprehension scores. If such a group were matched in size to a group of children exhibiting the converse profile (i.e., high NARA comprehension scores and low listening comprehension scores), this would reflect random variation in one or both of the measures. If the underestimated group were greater in size than the converse group, this would suggest that the NARA comprehension measure systematically underestimates the comprehension ability of some children. STUDY 2 Method Participants A total of 212 Year 3 children (114 female and 98 male) from six urban Bristol schools participated in Study 2. The sample ranged in age from 7 years 2 months to 8 years 7 months, with mean age 8 years (SD 4 months). Eight whole classes were tested to include the mainstream range of abilities. All children may be assumed to have been taught according to the National Curriculum, and therefore to have received comparable literacy education. Materials All children were administered Form 1 of the Neale Analysis of Reading Ability (Neale, 1997) and a listening comprehension test. The latter test was constructed in the same manner as the silent reading comprehension test described in Study 1. An example story and its accompanying questions are presented in Appendix B. During administration of the test, children listened to four stories of increasing difficulty, followed by their accompanying true/false statements (also auditorily presented a total of 28 statements). Children answered the questions by ticking and crossing a worksheet. No assistance was available to children, and they were encouraged to guess when they did not know the answer. Performance was scored as number of items correct. The internal reliability of the listening comprehension test was investigated across all items, revealing Cronbach s =.62. Procedure Children were first administered the listening comprehension test, as a group in their usual classes. The Neale Analysis of Reading Ability was administered to each child individually, over the few days following administration of the listening comprehension test. Results The mean score on the listening comprehension test was (SD 2.86): maximum possible score was 28. The mean reading age for the NARA accuracy measure was 8

8 194 Alice L. R. Spooner et al. years 6 months (SD 1 year 7 months). The mean reading age for the NARA comprehension measure was 7 years 11 months (SD 1 year 4 months). Two decoding groups were formed according to listening comprehension and NARA accuracy performance. The skilled decoding group consisted of the 51 children who achieved above average listening comprehension score (24 and above), and NARA accuracy age of 6 months or more above their chronological age. The less skilled decoding group consisted of the 29 children with above average listening comprehension and NARA accuracy age 6 months or more below chronological age. Criteria of 6 months above or below chronological age were selected to provide a clear margin between the groups. The profiles of the groups are presented in Table 2. Table 2. Decoding group profiles from Study 2, with standard deviations in parentheses Decoding Listening NARA reading ages group N Age comprehension Accuracy Comprehension Skilled 51 8y 0m (4m) 25.9 (1.3) 10y 6m (1y 4m) 9y 3m (1y 6m) Less skilled 29 8y 1m (4m) 25.3 (1.4) 7y 1m (7m) 7y 6m (10m) A series of one-way analyses of variance, with group as factor, was conducted on age, listening comprehension score, NARA accuracy score and NARA comprehension score. The analyses revealed non-significant main effects of group on age, F(1,78)=1.29, and on listening comprehension score, F(1,78)=3.18, both p>.05, and significant main effect of group on NARA accuracy score, F(1,78)=201.80, p<.001: These results confirm satisfactory group selection. The analyses revealed significant main effect of group on NARA comprehension score, F(1,78)=35.97, p<.001, with skilled decoders achieving higher scores than less skilled decoders. Further analyses investigated whether the decoding groups attained reading ages on the NARA comprehension measure that differed significantly from chronological age. Age-related comprehension scores were derived by subtracting chronological age from NARA comprehension age. One-sample t test analyses (with test value zero) were conducted on the age-related NARA comprehension scores attained by each decoding group separately. The analyses revealed significant deviations from zero in both skilled decoders, t(50)=5.96, and less skilled decoders, t(28)=4.23, both p<.001: Skilled decoders attained higher NARA comprehension scores, and less skilled decoders attained lower NARA comprehension scores, respectively, than would be expected for their age. Next, the pattern of performance across the NARA and listening comprehension measures was investigated. First, a Pearson correlation was conducted on listening comprehension score and NARA comprehension score, with age partialled out. The analysis revealed significant relationship between listening comprehension and NARA comprehension in the general population, r=.57, N=212, p<.001. Figure 1 presents the scatterplot of scores attained on the two tests. To indicate age-related performance, NARA comprehension performance was plotted as the difference between comprehension age and chronological age. Reading comprehension performance was categorized as high if NARA comprehension age was equivalent to or above chronological age, as low if comprehension age was

9 Reading accuracy and comprehension 195 Figure 1. Scatterplot of performance in Study 2. Group boundaries are indicated 12 months or more below chronological age, and as intermediate if comprehension age fell between these two markers. Since the assessment of listening comprehension was not a standardized test, individual performance was categorized relative to mean performance of the Study 2 sample. Listening comprehension was termed high if above the mean (scores of 24/28 and above), as low if more than one standard deviation below the mean (20/28 and below), and as intermediate if scoring between these two cut-offs. The boundaries of these performance categories are marked on Figure 1. Of the children who attained high NARA comprehension, 75% attained high listening comprehension. Of the children who exhibited low listening comprehension 80% attained low NARA comprehension scores. However, of the children who attained low NARA comprehension scores, 32% attained high listening comprehension scores. A cross-tabulation analysis was conducted to explore the predictability of performance between the tests. Wilkes provides an estimate of the degree to which one variable accounts for the variance in another variable. The analysis revealed that while listening comprehension was a significant predictor of NARA comprehension, =.18, df=2,

10 196 Alice L. R. Spooner et al. p<.01, NARA comprehension was not a significant predictor of listening comprehension, =.02, df=2, p>.05. NARA accuracy performance was inspected in the subgroup of children who attained low NARA comprehension scores despite high listening comprehension. Of the 18 children in this subgroup, 11 attained NARA accuracy scores below that expected of their age, of whom 9 attained reading accuracy age at least 6 months below chronological age. However, only 2 children in the subgroup exhibited reading accuracy age more than 12 months below chronological age. The subgroup exhibited mean accuracy reading age 1 month below chronological age. In summary, despite being matched for listening comprehension, children of high and low decoding ability achieved different NARA comprehension scores: less skilled decoders exhibited lower comprehension scores and skilled decoders achieved higher scores. Secondly, almost one in three children who exhibited poor comprehension performance on the NARA scored above average on the listening comprehension test. The children in this subgroup were not generally poor readers, suggesting that poor decoding alone cannot account for this discrepant performance across comprehension test. Discussion A subgroup of less skilled decoders attained lower NARA comprehension scores than would be expected for their age, despite above average listening comprehension performance. In contrast, skilled decoders, with equivalent listening comprehension ability, scored higher on NARA comprehension than expected for their age. This supports the finding in Study 1 that NARA comprehension performance is dependent on reading accuracy. It further demonstrates that this dependency extends across the range of reading accuracy, with apparently boosted NARA comprehension performance in good decoders, and impaired performance in poor decoders. This confirms the utility of matching skilled and less skilled comprehension groups for reading accuracy when basing selection on the NARA (e.g., Cain & Oakhill, 1999; Nation et al., 1999; Nation & Snowling, 1998a, 1998b; Oakhill, 1982; Oakhill et al., 1986; Stothard & Hulme, 1992; Yuill et al., 1989). It also suggests that performance on the decoding and comprehension measures of the NARA should not be interpreted separately. Secondly, listening comprehension corresponded to NARA reading comprehension across the large sample. This replicated previous findings by Nation and Snowling (1997) and Stothard and Hulme (1992). However, nearly one in three children who attained low NARA comprehension scores exhibited above average listening comprehension, suggesting that the NARA can underestimate comprehension ability. Some such children had weak decoding skills, and the evidence from Studies 1 and 2 would suggest that their low reading accuracy accounted for their underachievement on NARA comprehension. Such children may have derived insufficient information from text for comprehension as a result of their inaccurate decoding. Or, being aware of their weak decoding skills, they may have focused on decoding at the cost of their comprehension. Alternatively, their comprehension of the texts may have been relatively unaffected by their weak decoding, but their comprehension scores were restricted as a result of their high numbers of decoding errors. There are multiple reasons why weak decoding may result in an underestimation of comprehension ability on the NARA. However, the underestimated group were not all poor decoders. It

11 Reading accuracy and comprehension appears that reading accuracy does limit NARA comprehension, but that some additional factor also influences performance. One additional factor may relate to the different demands placed by the open-ended questions in the NARA and the closed questions in the listening comprehension test. Some areas of research advocate the use of open-ended questions, with evidence suggesting that children s accuracy is greater with these questions than with closed questions. This is particularly important, for example, in eyewitness testimony, where children have been found to contradict themselves in response to closed but not open questions (Lamb & Fauchier, 2001), and to say don t know where appropriate to open but not closed questions (Peterson, Dowden, & Tobin, 1999). Further evidence suggests that closed questions can prompt children into fabricating information in order to fill gaps in their understanding (Erskine, Markham, & Howie, 2002). Where accuracy of individual responses is important, for example in testimony, open-ended questions appear to be preferable to closed or forced-choice questions. However, when accuracy is calculated across a number of questions such as in the comprehension tests employed in this work individual responses that involved contradictions or fabrications are no longer problematic. Rather, open-ended questions such as those used in the NARA comprehension measure might limit the performance of some children. The NARA comprehension questions required the formulation of answers that describe and explain what the child understood. It is therefore possible in Study 2 that some children attained lower NARA comprehension scores than their actual comprehension ability merited, due to some difficulty formulating answers to such questions. The listening comprehension test, on the other hand, removed these potential hurdles by requiring a forced-choice response, and therefore no longer depended on expressive speech. By administering a parallel version of the NARA in which the questions were transformed to true/false statements, Study 3 investigated whether some children who attained low comprehension scores on the traditional version of the NARA might have had some difficulty with producing answers to the open-ended questions rather than comprehension per se. Skilled and less skilled comprehenders were selected according to performance on the NARA, in a manner that is representative of the literature. To ensure that the less skilled group did not erroneously include children whose comprehension performance had been impaired by low reading accuracy, all children exhibited above average reading accuracy. 197 STUDY 3 Method Participants From the sample assessed on the NARA in Study 2, 32 children were selected to participate in Study 3. Sixteen skilled comprehenders were selected according to the following criteria: (a) At least age-appropriate reading accuracy and comprehension, (b) Reading accuracy and comprehension ages no more than two years above actual age, and (c) Reading accuracy and comprehension ages within 12 months of each other. Sixteen less skilled comprehenders were selected according to the following reading profile: (a) At least age-appropriate reading accuracy, and (b) Reading comprehension

12 198 Alice L. R. Spooner et al. age at least 12 months below actual age. The profiles of the two groups are summarized in Table 3. Table 3. Participant profiles from Study 3, with standard deviations in parentheses Comprehension N NARA reading ages group Boys Girls Age Accuracy Comprehension Skilled 8 8 7y 11m (3m) 8y 5m (4m) 8y 6m (5m) Less skilled y 11m (4m) 9y 0m (1y 4m) 6y 6m (5m) Materials The NARA, Form 1 (Neale, 1997) yielded standardized reading ages for accuracy and comprehension used in group selection, and the comprehension raw scores provided the measure of open-ended question performance. The NARA, Form 2, (Neale, 1997) formed the basis for the second test. While the stories were used in their original form, the questions were transformed to true/false statements. The content of original questions and their correct answers was closely adhered to. In the few cases that this was not possible, new true/false statements were constructed to tap the same story information. This version of the NARA provided the measure of forced-choice performance. An example of the forced-choice questions are presented in Appendix C. Scoring Performance on both tests was scored as the number of questions correct. To enable between-test comparison, children completed Form 2 of the NARA to the level at which they reached the accuracy cut-off in Form 1. In the few cases where children reached the accuracy cut-off in Form 2 at a lower level than in Form 1, comprehension scores were calculated to the lower level in both tests. Procedure Children had been administered the NARA, Form 1, as part of Study 2. On the basis of this performance groups were selected as described above. These children were then administered the transformed version of the NARA, Form 2, a few days after administration of Form 1. This test was administered according to the typical procedure, with the exception that the modified questions required a forced-choice response: True/false, yes/no and a nod or shake of the head were all acceptable responses. Answers were recorded at testing. Results Table 4 summarizes group performances. Skilled comprehenders attained similar scores across question type. Less skilled comprehenders scored below the skilled group with

13 Reading accuracy and comprehension 199 Table 4. Mean comprehension scores, and number of stories completed (in each test) in Study 3, with standard deviations in parentheses Question type Comprehension group Number of stories Open-ended Forced-choice Skilled 3.8 (0.40) 17.7 (2.3) 17.8 (3.7) Less skilled 4.2 (0.83) 7.3 (2.3) 16.8 (2.6) open questions, but attained similar scores to the skilled group with forced-choice questions. To validate comparisons between the skilled and less skilled comprehenders, it was first determined whether the groups completed the same number of questions. A oneway analysis of variance, with group as factor, was conducted on the number of stories completed by skilled and less skilled comprehenders. The analysis revealed nonsignificant differences between groups, F(1,30)=2.62, p>.05. The next analysis was conducted to ascertain whether comprehension groups were differentially affected by question type. A two-way analysis of variance, with group and question type as factors, was conducted on comprehension scores attained in the openended and forced-choice versions of the NARA. The analysis revealed significant main effect of group, F(1,30)=50.40, p<.001, with higher scores obtained by the skilled group, and significant main effect of question type, F(1,30)=76.17, p<.001, with more correct answers to forced-choice questions. The analysis also revealed significant interaction between group and question type, F(1,30)=72.79, p<.001. A series of posthoc Tukey comparisons was conducted on comprehension scores, and revealed that while skilled comprehenders were unaffected by question type, p>.05, less skilled comprehenders scored significantly higher with forced-choice questions than with open-ended questions, p<.05. In addition, while the groups differed in their ability to answer open-ended questions, p<.05, they did not differ when assessed by forcedchoice questions, p>.05. Discussion Groups of skilled and less skilled comprehenders were selected on the basis of their good and poor performance, respectively, on the NARA comprehension measure, in its traditional format with open-ended questions. However, these groups did not differ in comprehension performance on an equivalent test when assessed by ability to answer forced-choice questions. The less skilled comprehension group may have included some children who experienced difficulties in meeting the extra demands made by the open-ended questions, rather than comprehension deficits per se. Such demands may include meta-comprehension, expressive speech, and perhaps some degree of selfconfidence. This finding has important implications for the selection of comprehension groups for use in experimental research, and for teachers trying to understand a child s problems in reading. These implications are discussed below.

14 200 Alice L. R. Spooner et al. GENERAL DISCUSSION The work presented in this article set out to address two assumptions held about the Neale Analysis of Reading Ability (Neale, 1997): that the measures of reading accuracy and reading comprehension it offers are independent, and that low performance reliably reflects low ability rather than some artefact of the test. It was found that NARA comprehension performance was mediated by decoding ability. It was also found that a group who scored poorly on the NARA comprehension measure scored well when the question demands were reduced. Therefore, the NARA accuracy and comprehension measures are not independent, and low comprehension performance can reflect weak decoding ability and/or some difficulty with open-ended questions. One possible limitation of the work is that the internal reliabilities of the silent reading and listening comprehension tests employed in Studies 1 and 2 were not particularly high. However, this is likely to reflect the forced-choice design of the tests. Furthermore, silent reading comprehension did correlate significantly with NARA accuracy and NARA comprehension in Study 1. In addition, it was found in Study 2 that while some children scored highly on the listening comprehension test and poorly on the NARA comprehension measure, the converse profile did not arise. This discrepant performance, therefore, did not reflect low reliability of test materials but some systematic difference between the listening comprehension test and the NARA comprehension measure. These findings support the reliability of the silent reading and listening comprehension tests. Comprehension performance on the NARA was affected by reading accuracy, to the extent that children with above average listening comprehension ability but low reading accuracy achieved significantly low reading comprehension scores on the NARA. Several possible explanations for this finding were discussed in the introduction to this article. Firstly, less skilled decoders were expected to derive insufficient information from text from comprehension. The NARA attempts to control this by correcting children s decoding errors. However, it was proposed that the quality of information derived from text during slow, disjointed and interrupted reading might never match the quality of information derived from smooth, skilled reading. Secondly, it was proposed that some children might not distribute effort evenly across the decoding and comprehension components of the test. Thirdly, the design of the NARA introduces further reasons to expect that children with weak decoding skills would be at a disadvantage on the comprehension component of the test. Children are only permitted to attempt comprehension questions if they have made fewer than a certain number of decoding errors. The comprehension scores of children with weak or inaccurate decoding skills are therefore necessarily restricted. This is an important artefact of the NARA that should be appreciated. Experience administering the NARA has revealed that some children are clumsy in their decoding, making silly errors when reading aloud. The comprehension of such children is often unimpaired. Such children reach the accuracy cut-off early, due to their high number of errors, and are limited in their comprehension score despite exhibiting good comprehension so far. Though this is an anecdotal account, it demonstrates the risk that the cut-off artefact inherent in the NARA can underestimate reading comprehension ability. The findings from this work suggest that performance on the NARA should not be

15 Reading accuracy and comprehension used to make inferences about children s general comprehension ability. Where the NARA is employed in the selection of reading comprehension groups, these findings confirm the importance of controlling for decoding ability as is common practice. Without doing so, less skilled comprehension groups may include children whose difficulties lie instead with decoding. Educational assessors should be aware that low comprehension scores may reflect weak decoding skills, in order to correctly target special needs assistance. The NARA is not sufficient to separate comprehension ability from decoding ability. The results of Study 3 suggest that the NARA can underestimate comprehension ability as a result of its reliance on open-ended questions. In addition to comprehension, such questions require children to explain their understanding. Some children may therefore be viewed as having comprehension difficulties rather than difficulties with expressive speech. Selection of comprehension assessment must involve identification of the meaning of comprehension. If one considers comprehension to be the construction of some form of mental model of the information presented (e.g., Gernsbacher, 1990; Johnson- Laird, 1985; Kintsch, 1998) then comprehension is an input process. Open-ended questions make heavy output demands. Therefore, using open-ended questions to assess comprehension confounds input processing ability with output processing ability. By minimizing the output demands, forced-choice questions (particularly those which can be answered non-orally, as those used in the studies presented here) purify the assessment of the input processing ability. A large body of research has used the NARA to identify comprehension groups. In some cases, experimental techniques were then employed that used forced-choice questions, and failed to detect cognitive differences between the groups (e.g., Cain, Oakhill, Barnes, & Bryant, 2001; Nation & Snowling, 1997): these less skilled groups may have included children whose difficulties on the NARA resulted from the openended question demands, rather than comprehension per se. In other cases, group differences have been detected on tasks that involve open-ended questions (e.g., Cain, 1999; Cain & Oakhill, 1999; Cain et al., 2001; Oakhill, 1984, Stothard & Hulme, 1992; Yuill et al., 1989). Such research may be reporting circular effects of ability to answer open-ended questions. These hypotheses suggest that comprehension assessments involving open-ended questions be avoided in comprehension research. The NARA offers a standardized and widely used assessment of reading for comprehension that resembles successful reading in an interactive educational situation. However, when assessment requires independent measures of decoding and reading comprehension abilities, the NARA appears unable to separate those skills. Furthermore, it makes demands beyond comprehension that may result in an underestimation of a child s general language comprehension ability and reading comprehension ability. Thus, for experimental investigations of comprehension abilities the NARA may not be entirely appropriate. More accurate measures of general language comprehension ability should be provided by tests that have no reading component and minimize question demands, enabling more precise targeting of comprehension ability. Furthermore, decoding and reading comprehension should be more accurately assessed using separate texts for these two skills. 201

16 202 Alice L. R. Spooner et al. Acknowledgements This work was conducted as part of the first author s PhD at the University of Bristol, funded by the Medical Research Council. The first author wishes to thank Marketa Caravolas, Katy Lobley, and two anonymous reviewers for helpful comments on an earlier draft of this article. References Cain, K. (1999). Ways of reading: How knowledge and use of strategies are related to reading comprehension. British Journal of Developmental Psychology, 17, Cain, K., & Oakhill, J. V. (1999). Inference making ability and its relation to comprehension failure in young children. Reading and Writing, 11, Cain, K., Oakhill, J., & Bryant, P. (2000). Phonological skills and comprehension failure: A test of the phonological processing deficit hypothesis. Reading and Writing: An Interdisciplinary Journal, 13, Cain, K., Oakhill, J., & Sharp, N. (1998). Working memory, reading comprehension and the ability to learn new vocabulary items from context. Paper presented at the BPS Developmental Section, Lancaster. Cain, K., Oakhill, J., Barnes, M. A., & Bryant, P. E. (2001). Comprehension skill, inference-making ability, and their relation to knowledge. Memory and Cognition, 29, Carr, T. H., Brown, T. L., Vavrus, L. G., & Evans, M. A. (1990). Cognitive skill maps and cognitive skill profiles: Componential analysis of individual differences in children s reading efficiency. In T. H. Carr & B. A. Levy (Eds.), Reading and its development: Component skills approaches (pp. 1 55). San Diego, CA: Academic Press. Erskine, A., Markham, R., & Howie, P. (2002). Children s script-based inferences Implications for eyewitness testimony. Cognitive Development, 16, Gernsbacher, M. A. (1990). Language comprehension as structure building. Hillsdale, NJ: Erlbaum. Gernsbacher, M. A., Varner, K. R., & Faust, M. E. (1990). Investigating differences in general comprehension skill. Journal of Experimental Psychology: Learning, Memory and Cognition, 16, Johnson-Laird, P. N. (1985). Mental models (2nd ed.). Cambridge: Cambridge University Press. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press. Lamb, M. E., & Fauchier, A. (2001). The effects of question type on self-contradictions by children in the course of forensic interviews. Applied Cognitive Psychology, 15, Nation, K., Adams, J. W., Bowyer-Crane, C. A., & Snowling, M. J. (1999). Working memory deficits in poor comprehenders reflect underlying language impairments. Journal of Experimental Child Psychology, 73, Nation, K., & Snowling, M. (1997). Assessing reading difficulties: The validity and utility of current measures of reading skill. British Journal of Educational Psychology, 67, Nation, K., & Snowling, M. (1998a). Semantic processing and the development of wordrecognition skills: Evidence from children with reading comprehension difficulties. Journal of Memory and Language, 39, Nation, K., & Snowling, M. J. (1998b). Individual differences in contextual facilitation: Evidence from dyslexia and poor reading comprehension. Child Development, 69, Neale, M. D. (1958). Neale Analysis of Reading Ability. London: Macmillan. Neale, M. D. (1997). Neale Analysis of Reading Ability Revised. Windsor: NFER-Nelson. Neale, M. D., Christophers, U., & Whetton, C. (1989). Neale Analysis of Reading Ability Revised British Edition. Windsor: NFER-Nelson. Oakhill, J. (1982). Constructive processes in skilled and less skilled comprehenders memory for sentences. British Journal of Psychology, 73, Oakhill, J. (1984). Inferential and memory skills in children s comprehension of stories. British Journal of Educational Psychology, 54,

17 Reading accuracy and comprehension 203 Oakhill, J., Yuill, N., & Parkin, A. (1986). On the nature of the difference between skilled and lessskilled comprehenders. Journal of Research in Reading, 9, Peterson, C., Dowden, C., & Tobin, J. (1999). Interviewing preschoolers: Comparisons of Yes/No and Wh-questions. Law and Human Behaviour, 23, Stothard, S. E., & Hulme, C. (1992). Reading comprehension difficulties in children: The role of language comprehension and working memory skills. Reading and Writing, 4, Stothard, S. E., & Hulme, C. (1995). A comparison of phonological skills in children with reading comprehension difficulties and children with decoding difficulties. Journal of Child Psychology and Psychiatry, 36, Yuill, N., Oakhill, J., & Parkin, A. (1989). Working memory, comprehension ability and the resolution of text anomaly. British Journal of Psychology, 80, Received 20 May, 2002; revised version received 14 January, 2003 Appendix A Example story and accompanying true/false questions from the silent reading comprehension test employed in Study 1 (corresponds to NARA level 3). As soon as they reached the house, Jack jumped out of the fire-engine and ran inside. He knew a boy was trapped upstairs. He battled through flames to climb the stairs, and found Tim hiding behind a bed. Then they heard a loud crash. The stairs had vanished! Jack opened the window and shouted for a ladder. When they reached the ground, Tim hugged his mother, and Jack was a hero. Jack is a policeman. (False) Jack jumped out of a fire-engine. (True) The boy was trapped downstairs. (False) Tim was hiding in the bathroom. (False) The stairs collapsed. (True) Jack shouted for a hose. (False) They escaped down a ladder. (True) Tim hugged his mother. (True) Appendix B Example story and accompanying true/false questions from the listening comprehension test employed in Study 2 (corresponds to NARA level 3). Emma had never been camping before. She woke up in the dark, and heard snorting and rustling outside. As Emma listened, the noise came closer. She grabbed her sister and told her a huge monster was coming. But when her sister looked, she laughed. She told Emma to come and see. It was a hedgehog, hunting for slugs. Emma was amazed that such a small creature could make so much noise. This was Emma s first time camping. (True) The story happened in the day. (False) Emma was scared by a noise. (True) Emma shared a tent with her sister. (True) Emma s sister laughed when she saw the hedgehog. (True) Emma saw something scary. (False) The hedgehog was hunting for worms. (False) The hedgehog was really big. (False)

18 204 Alice L. R. Spooner et al. Appendix C Example questions from the forced-choice version of the NARA derived for Study 3 (to accompany NARA, Form 2, level 3 story Circus). The story took place in a zoo. (False) The lions had just started their show. (False) Jack was waiting to get the lions out of the ring. (True) The lions were restless because of the whip. (False) Tina dropped the whip. (True) Jack saved Tina. (True) Jack finished the act. (False) Jack then decided that he would become a lion tamer. (True)