Boston Naming Test Short Forms: A Comparison of Previous Forms with New Item Response Theory Based Forms

Transcription

1 Journal of Clinical and Experimental Neuropsychology 2004, Vol. 26, No. 7, pp Journal of Clinical and Experimental Neuropsychology Boston Naming Test Short Forms: A Comparison of Previous Forms with New Item Response Theory Based Forms R. E. Graves 1, S. C. Bezeau 1,2, J. Fogarty 2, and R. Blair 2 1 Department of Psychology, University of Victoria, Victoria, British Columbia, Canada, and 2 Elisabeth Bruyere SCO Health Service, Ottawa, Ontario, Canada ABSTRACT Two new short forms of the Boston Naming Test (BNT) were developed using item response theory (IRT) with data from 206 elderly outpatients. We evaluated the diagnostic ability of 12 short forms among the full sample and in a sub-sample of 69 patients diagnosed with mild Alzheimer s disease (AD) either alone or in combination with vascular dementia (VD). The full BNT (reliability alpha=.90) identified 44% of the AD/VD patients as abnormal on naming. Our 30 item short form (alpha=.90) also identified 44% of the AD/VD patients as abnormal, with 93% agreement with the full BNT on abnormal AD/VD patient classifications. Our 15 item short form (alpha=.84) identified 48% of the AD/VD patients as abnormal, with 90% agreement with the full BNT s abnormal classifications. An adaptive 30/15 item version equaled the performance of the full 30 item test while requiring only 15 items for 75% of the patients with normal naming ability. This study illustrates the utility of IRT for developing neuropsychological assessment tools. INTRODUCTION The 60 item Boston Naming Test (Kaplan, Goodglass, & Weintraub, 1983) is widely used to assess naming ability in clinical neuropsychology. Shortened versions for reduced administration time can be useful, especially for clients with limited attention or motivation and for screening purposes. While a number of shorter versions have been devised, there is limited information available on the psychometric properties of these short forms. Furthermore, none of the currently available short forms was developed using item response theory (IRT). IRT provides methods to shorten a test by identifying items that will maintain high reliability and maximize discrimination at an ability level appropriate for a desired population. We first review reliability and validity data for previous short forms, second describe the development of new 30 and 15 item short forms using IRT principles, and third evaluate the performance of the new short forms and 10 other previously published short forms of the Boston Naming Test (BNT) for identifying naming ability in a sample of mostly elderly individuals from an outpatient clinic. Review of Existing Short Forms The best established of the previously published short forms of the BNT is the Williams 30 item test (Williams, Mack, & Henderson, 1989). The items for this test were selected to be those that exhibited the largest difference in accuracy rate Address correspondence to: Dr. Roger Graves, P.O. Box 3050, Victoria, BC, V8W 3P5, Canada, rgraves@uvic.ca /04/ $16.00 Taylor & Francis Ltd.

2 892 ROGER GRAVES ET AL. between 15 Alzheimer Disease (AD) patients and 15 age and education matched normal control subjects. This test has been included in four previous cross-validation studies. The first (Mack, Freed, Williams, & Henderson, 1992) found large differences between 26 AD patients and 26 normal control subjects on this test, but reliability and correlation with the full BNT were not reported. The second study (Franzen, Haut, Rankin, & Keefover, 1995) used 320 non-normal clinically diagnosed participants and reported high reliability (alpha.95), high correlation with the full version (.99), and high agreement accuracy (92%) with the full BNT s classifications (normal, borderline, impaired). The third study reported good reliability (alpha.74) and high correlation with the full BNT (.96) with 219 mostly elderly normal adults (Tombaugh & Hubley, 1997). The fourth study (Lansing, Ivnik, Cullum, & Randolph, 1999) reported that the Williams 30 item test discriminated as well as did the full 60 item BNT between 237 AD patients and 717 normal control subjects, all mostly elderly. The two Saxton 30 item tests (Saxton, et al., 2000) were created by dividing the 60 BNT items into two sets, matched in difficulty according to the performance of 314 healthy community dwelling elderly volunteers. The original study reported high reliability (alpha.83,.82) and high correlations with the full BNT (.96,.96) for forms A and B. We are not aware of any previous cross-validation studies of these forms. The four 15 item short forms of Mack et al. (Mack, Freed, Williams, & Henderson, 1992) were created by assigning the 60 BNT items to four sets based on the order of items, which was intended to create four forms matched for word frequency. The original study examined performance on these versions by 26 AD patients and 26 age-matched, but not education-matched, controls. Reliability was not reported but correlations with the full 60 item BNT were reported as high (.97 to.98). Although the AD patients scored lower on all four short forms, there were significant differences in performance across the four forms. Franzen et al. (1995) reported high reliability for these forms (alpha.87,.84,.86,.84 respectively) and high correlations with the full BNT (.95 to.97), but unacceptably high misclassification rates (18% to 24%) and significant variations across forms with their 320 non-normal patients. These four Mack 15 item forms were studied by Tombaugh and Hubley (1997), who reported relatively low reliabilities (alpha.31,.41,.52,.49 for forms 1 through 4), but good correlations with the full 60 item BNT (.72 to.82) with 219 community dwelling mostly elderly volunteers. The four Mack 15 item forms were also studied by Fastenau et al. (1998), who also reported relatively low reliabilities (alpha.37,.40,.48,.67 for forms 1 through 4), fair correlations with the full BNT (.63 to.74), and significant differences among the four versions, with 108 community dwelling mostly elderly adults. Fastenau et al. recommended forms 3 and 4 as the best of the four. Lansing et al. (1999) included the four Mack 15 item forms in their study and reported that these short forms discriminated almost as well as did the full 60 item BNT between 237 AD patients and 717 normal controls, all mostly elderly, with form 3 showing the best discrimination. The latest published version of the BNT (Kaplan, Goodglass, & Weintraub, 2001) includes the Mack et al. (1992) Form 4. The Mack 15 item forms were combined by Fastenau, Denburg & Mauer (1998) into two 30 item forms, which we will refer to as the Mack 30 item forms A (form 1 plus form 2) and B (form 3 plus form 4). Fastenau et al. reported reliabilities (alpha) of.57 and.75 for versions A and B and correlations with the full 60 item BNT of.76 for both versions, with 108 community dwelling mostly elderly adults. Items for the 15 item test of Lansing et al. (1999) were selected using a stepwise discriminant function method to maximize discrimination between 237 AD patients and 717 normal control subjects. Lansing et al. did not report reliability data, but reported that discrimination between the groups was slightly higher for their 15 item version than for the full 60 item version. However, Lansing et al. s use of a stepwise discrimination method to select items will capitalize on random error variance, which tends to produce much higher discrimination on the original sample than would be found on a new sample that has a different pattern of random error (Thompson, 1995). Thus, the items selected for this latter test may not perform well on cross-validation, and no previous crossvalidation study of this test could be located.

3 BOSTON NAMING TEST SHORT FORMS 893 Of the 10 short forms reviewed above, 8 were developed on rational grounds in an attempt to preserve the content range of the full test (i.e., range of word frequency and item difficulty). While this approach is commendable in several respects, including retaining generalizability across all populations (cf. Smith, McCarthy, & Anderson 2000), this approach is also expected to produce a less reliable test since reliability decreases as test length is reduced when the original test structure is maintained (Embretson, 1996). Short forms can, however, be developed as special purpose instruments that attempt to retain the reliability and validity of the original as much as possible for a certain population or purpose, at the expense of generality. Two existing short forms can be seen as examples of the latter test development approach. The Williams 30 item test was developed by selecting items that discriminated best between AD patients and normal subjects. This approach might also be expected to show some unwanted capitalization on error, especially in view of the small sample size (N = 30), but the procedure did avoid stepwise methods. The Lansing 15 item test was also developed to discriminate between AD patients and normal subjects. However, as discussed above, the stepwise discrimination item selection method used by Lansing et al. (1999) has been strongly criticized (Thompson, 1995). It is noteworthy that the review of previous short forms clearly shows that the most successful short form for use in identifying abnormal naming in elderly dementia patients is the Williams test, which was developed specifically for this purpose and population. Although two of the short forms reviewed above were designed empirically for a special purpose, their design approach of selecting items to optimize discrimination between AD and normal groups can be questioned. The first step in designing a new test is to set a design goal. If the ultimate goal is to discriminate a group of AD patients from a group of normal patients, one would never start with just a naming test since many normal elderly have naming problems while not all early AD patients do so. Instead, one would include items reflecting other common symptoms of AD such as memory and semantic difficulty. The result would be a multidimensional test like the Mini-Mental State Exam (MMSE, Folstein, Folstein, & McHugh, 1975) or the Modified Mini-Mental State Exam (3MS, Teng & Chui, 1987). Why then create a BNT short form by selecting items to optimally discriminate AD patients from cognitively normal patients? The logic that the designers of these tests had in mind, although not stated explicitly, was probably: a) assume that the groups differ in naming ability and that naming ability is the only difference between the groups that affects naming item success, b) conclude that the items that best discriminate the groups must therefore do so because they actually are the best for measuring naming in this population. Thus, selecting items for best group discrimination would indirectly result in selecting items for best naming ability discrimination. Although the logic is understandable, the success of this indirect approach depends on the extent to which the groups actually differ in naming ability. However, early AD and elderly normal groups typically do not differ strongly in naming ability (substantial between group overlap) nor are the groups consistent in naming ability (large within group variability). Therefore, selecting items on the basis of group discrimination is not an ideal method for designing a short form test of naming ability. Criteria for Design of New Tests In our view, the design goal for the task of creating a short naming test is simply to produce the best possible test for measuring patients naming ability. In accord with this goal, the design process requires that items be selected so as to optimize the short form s capacity to assess naming ability at the individual patient level in agreement with the long form, possibly for a specific population (e.g., elderly referrals to a memory disorders clinic). The best strategy for doing this involves using an individual differences, correlational, analysis (Clark, 1986) rather than the group differences analysis approach used by Williams et al. (1989) and Lansing et al. (1999). Thus, for short form item selection we combined all patients into one group, regardless of diagnosis, since it does not matter whether a patient with poor naming was diagnosed as normal or as a dementia case if one is primarily interested in measuring naming ability. In addition to correlational methods (e.g., principal components analysis) we applied IRT principles

4 894 ROGER GRAVES ET AL. in our short form development. IRT is ideally suited for this purpose since it is intrinsically an individual differences type of analysis and because IRT provides a variety of techniques specifically intended for selecting test items to optimize measurement of a latent trait such as naming ability (Bond & Fox, 2001; Embretson, 1996). Validity Evaluation Since test validity should always be evaluated in the context of a particular use of a test (AERA, et al., 1999), we chose the common clinical application of using a naming test to classify performance dichotomously into normal versus impaired categories. If the classification into normal versus impaired naming performance by the full 60 item BNT is taken as the gold standard, then one type of evidence of a short form s validity is how well its classifications agree with those of the full BNT. Thus, in this study we first develop short forms using IRT principles, with the entire sample considered as a whole (therefore blind to diagnostic classifications), and with naming ability taken as a continuous measure. We then evaluate the utility of the tests by looking at a specific clinical application, dichotomous (abnormal vs. normal) classifications of naming ability of patients within diagnostic classifications. This approach to evaluation of short form performance is similar to that used by Franzen et al. (1995) who employed a three level classification scheme with a somewhat larger and substantially more impaired sample. METHODS Participants We examined archival data of all patients at a memory disorders clinic in Ottawa, Canada who received the complete 60 item BNT in English from 1998 to the time of the study in Patients were included if there was adequate diagnostic information, plus data on the MMSE and 3MS tests. Diagnoses were provided by a neurologist, based on medical and social history, laboratory and radiological data, neurological examination, and neuropsychological test data. Patients were not excluded on basis of concomitant psychiatric diagnoses. Of the 206 patients (see Table 1 for demographic information), 62 were diagnosed normal with no significant medical or neurological complications. Another 69 patients were diagnosed as having either AD, or as having a combination of AD and vascular dementia (VD), a group henceforth referred to as the AD/VD group. We did not attempt to separate a pure AD group in this study since both AD and VD lead to naming problems and there is increasing evidence of considerable overlap between these diagnoses (de la Torre, 2002). This AD/VD group was relatively mildly impaired as shown by the high scores on the BNT, MMSE, and 3MS (Table 1). Also included were a further 75 patients that included 53 with significant medical disorders but without neurological complications and 22 diagnosed with dementia of types other than AD/VD. Education was coded into eight categories. The 206 patients ranged from having no formal education (1%) to completed post-graduated programs (18%), with completed university (26%) being the most frequent category. Of the 62 normal patients, 68% had completed university, versus 36% for the 69 AD/VD patients. Performance on the BNT by our 62 normal Canadian outpatients (mean age 59, mean BNT score 54.4) is quite similar to the performance of community dwelling Americans reported by Van Gorp, Satz, Kiersch and Henry (1986, age 65 69, mean BNT 55.6) and by Mitrushina and Satz (1989, age 66 70, meant BNT 55.8). Development of New Short Forms Our intended use for BNT short forms was twofold, first to measure naming ability of persons with ability similar to our referral sample and second to use these measurements to discriminate between normal and abnormal levels of naming ability. Thus, our strategy was to select items similar in difficulty to the full range of abilities in our sample, and beyond this, to select items that would discriminate well at the normal/abnormal Table 1. Characteristics of the Patient Groups. Whole sample Normal AD/VD Other Diagnosis N Age (range, mean, SD) 38-93, 67.0, , 59.2, , 75.2, , 66.0, 11.7 MMSE (range, mean, SD) 19-30, 27.5, , 28.9, , 25.8, , 28.0, 2.0 3MS (range, mean, SD) , 91.4, , 96.1, , 85.8, , 92.7, 5.4 BNT 60 (range, mean, SD) 23-60, 51.3, , 54.4, , 47.7, , 52.0, 6.7

5 BOSTON NAMING TEST SHORT FORMS 895 cut-off level. Since the full BNT contained many items that were very easy for our sample, in practice this strategy resulted in deletion of easy items. We did, however, attempt to retain a wide range of item difficulty levels by keeping at least a few easy and hard items (those with the highest first component loadings) until the test length became so short that the second principle would be compromised. The Rasch, one parameter logistic, model (Bond & Fox, 2001; Rasch, 1960) was chosen because we intended that our short forms would be scored conventionally, as the total number of items answered correctly. Only the one parameter model is compatible with this scoring (tests based on the 2 and 3 parameter IRT models cannot be scored as number correct more complex scoring transformations are required). Our plan was to proceed in stages, constructing tests of progressively shorter length. In each stage we conducted a principal components analysis (PCA) based on the tetrachoric correlation matrix using TestFact (Wilson, Wood, & Gibbons, 1991) software. Assuming that the PCA provided evidence of the essential unidimensionality (which we defined as a first component explaining greater than 30% of variance and a ratio of the first to second component s explained variance of greater than 3.0), we would then conduct a Rasch, one parameter IRT, analysis using Winsteps version 3.23 software (Linacre & Wright, 2001). Item misfit to the Rasch measurement model was defined as information weighted standardized residual mean square (infit) above 1.3 plus a Z score for this statistic above 2.0. Satisfaction of fit criteria, in addition to providing evidence that the data satisfy assumptions of the measurement model, also provides evidence of unidimensionality from a different measurement perspective (Smith & Miao, 1994). The results of these analyses would be used to select items to omit in the next stage. The strategies described above imply that items would be selected for omission if either: a) they had a low loading on the first component of the PCA, b) failed the fit criteria to the Rasch model, or c) had a difficulty level outside the range for optimum discrimination of abnormal naming ability for our population. Item selection for our new short forms was done blind to diagnosis, using data from all 206 patients. Reliability and Validity Evaluation of Short Forms As an index of internal consistency reliability, Cronbach s alpha was calculated. Two major ways of looking at construct validity were adopted. First, from the perspective that scores from a short form should agree with the scores from the full test, correlations with the full BNT were calculated. Second, from the perspective that a desirable use of a short form would be to make a dichotomous classification of patients as either normal or abnormal on naming, the agreement of short form classifications with the classification of patients by the full BNT (taken as the gold standard) was evaluated. For this purpose, we established a cutoff for each test such that 10% of the 62 patients diagnosed as normal scored below the cutoff. (Percentiles were calculated according the methods described by Allen and Yen (1979, pp )). This cutoff was then used to classify naming ability as normal or abnormal for the 69 patients diagnosed with AD/VD. Percent agreement and Kappa indices were calculated. For the full BNT and the new tests, results for a cutoff such that 20% of the 62 normals scored below are also reported to give an indication of how the tests perform with a criterion providing more sensitivity for detecting abnormality in screening, but also more false positives. In addition, as indices relevant to convergent validity, correlations with MMSE and 3MS scores were calculated. RESULTS Development of New Short Forms Table 2 shows the items retained and the results of reliability and unidimensionality analyses for tests of each length. Table 3 shows the correspondence between scores on the new short forms and scores on the full 60 item BNT. Stage 1 The results for the full 60 item test showed that, for this sample, accuracy was quite high (see Table 1). The TestFact software could not calculate a tetrachoric correlation matrix (needed for the PCA) with items having essentially no variance (almost all persons getting the item correct). Thus, we could not employ our planned method of using the PCA results at this stage. In order to obtain a PCA, and in accord with the principle of omitting items with difficulty levels not appropriate for good discrimination, the thirteen items on which fewer than 3 of the 206 persons made an error were selected for omission. Omission of the 13 items having essentially no errors produced a 47 item test. Stage 2 The Rasch analysis revealed 3 items (items 50, 51, 56) failing our fit criteria. Four additional items were chosen for deletion based on the PCA results. These 7 items were the 7 with the lowest

6 896 ROGER GRAVES ET AL. Table 2. Development of New Short Forms of the Boston Naming Test. Number of items Reliability (alpha) Correlation with full test PCA a 1st (1:2 ratio) Items NA NA % (4.8) above less 1-10,12,14, % (5.9) above less 11,20,27,50-52, % (10.5) 13,17,19,21-25,28,29,31-36, 38,41-45,48,53-55, % (8.1) 19,24,31,32,35,41,42,44,45, 48,53-55,57,58 Notes: All analyses based on 206 cases; a Principal Components Analysis, results shown are the percentage of variance explained by the first component and the ratio of the first and second component s explained variances. Table 3. Correspondence a between scores on the new short forms and on the full BNT 60. New BNT30 Adaptive 30/15 New BNT15 Full BNT60 Percentile < Notes: a Scores matched according to equivalent percentiles in the 62 person normal group. loadings on the first PCA component (<.502). Omission of these 7 items resulted in a 40 item test with good evidence of unidimensionality for which all 40 items and all 206 persons satisfied the Rasch analysis fit criteria. Stage 3 Ten further items were selected for subsequent omission on the basis of lower loadings on the first component in the PCA and having low difficulty level. Omission of these 10 items resulted in a 30 item test for which all 30 items and all 206 persons satisfied the Rasch analysis fit criteria. This established our 30 item short form. Item loadings on the first PCA component were all high ( ) and all items fit the Rasch model, thus these two criteria provide little guidance for further item selection. The range of item difficulty on the Rasch logit scale was 5.0, a fairly wide range, but it was still targeted towards much more impaired individuals than our population since the mean person ability level was 1.9 logits above the mean item difficulty level. This new 30 item test contains 20 of the 30 items of the Williams et al. (1989) test. Stage 4 Optimum discrimination among individuals with a particular ability level occurs when there are many items with difficulty level matching the target ability level. Thus, for reduction to a 15 item version, we relied on our third item selection criterion. We selected for retention the 15 items with difficulty level closest (in logit value) to the difficulty level representing the 10th percentile of the normal patients. This involved selecting for deletion the 13 easiest and the 2 hardest items from the 30 item test. Omission of the final items resulted in a 15 item test for which all items and persons satisfied the Rasch analysis fit criteria. The range of item difficulty was 3.5 logits, with a mean item difficulty 0.9 logits easier than the mean person ability level.

7 BOSTON NAMING TEST SHORT FORMS 897 Starting and Stopping Rule Scored Version The data for the entire 60 item test were rescored using the scoring system described in the test manual (Kaplan, Goodglass, & Weintraub, 2001), hereafter referred to as the standard administration, which theoretically requires administration of between 16 and 60 items depending on the client s performance. For our sample, the number of items that would be given under this system ranged from 31 to 60, with means of: 38.9 items over all 206 patients, 35.4 items for the 157 patients scoring above the normal 10th percentile (i.e., a score above 47) on the full 60 item test, and 50.0 items for the 49 patients scoring below the 10th percentile. Reliability and Validity Evaluation Table 4 shows the results of the evaluation of the 12 short forms and the start/stop rule version using cutoffs set at the 10th percentile for the 62 normals. The correlation over the 206 patients with age was -.33 for the 60 item BNT and was very similar for all short forms (-.23 to -.35, -.32 Williams 30, -.34 new BNT30, -.35 new BNT15). The Spearman correlation between our categorized education variable and the 60 item BNT was -.28 for the 206 patients and was very similar for all short forms (-.22 to -.29, -.28 Williams 30, -.29 new BNT30 and BNT15). All these correlations are statistically significant at p <.001 with this sample size. Table 5 shows results for the new short forms using cutoffs set at the 20th percentile for the 62 normals. New 30/15 Item Adaptive Short Form Using IRT, we calculated a 95%, 2-tail, confidence interval for the 15 item test for a person with ability level at the 10th percentile of the normals. This indicates that a score of 12 or higher is unlikely (p <.05) for a person with ability less than the 10th percentile, while a score of 3 or less is unlikely (p <.05) for a person with ability greater than the 10th percentile. Accordingly, we defined an adaptive short form as follows. Administer the 15 item short form: if the score is 12 or greater, stop and give credit for 13 additional items, if the score is 3 or less, stop and use this score; otherwise administer the remaining items to complete the 30 item test. According to this rule, 100% of the sample was classified the same as for the complete 30 item test, that is, either above or below the cutoff shown in Table 4 for the 30 item test. Thus, the classification accuracy with this rule is identical to the results of the full 30 item test. With this rule, for patients scoring above the normal 10th percentile on the complete 60 item test, only 15 items would need be administered for 75% of those patients, with a mean number of items required of 19 (vs. 35 for the test manual start/stop rule version). For patients scoring below the normal 10th percentile on the complete 60 item test, only 15 items would be administered for 18% of those patients, with a mean number of items required of 27 (vs. 50 for the test manual version). DISCUSSION A review of previously published original and cross-validation reports of 10 different Boston Naming Test short forms revealed a variety of approaches to item selection and methods of evaluation. No previous short form had been developed using item response theory. We used IRT principles to develop new 30 item, 15 item, and adaptive 30/15 item short forms of the BNT based on data from 206 patients. Development of these short forms was done blind to patient diagnosis. Stepwise regression procedures were not used to select items, avoiding the pitfalls of the latter approach (Thompson, 1995). Evaluation of the starting and stopping rule scoring system recommended as the standard administration in the test manual (Kaplan, Goodglass, & Weintraub, 2001) showed that this scoring system provides scores that are close to, but not identical to, scores resulting from administering all 60 items, for this patient sample and for the purpose of identifying abnormal naming. The actual number of items that would be given under this system was 35 items on average for unimpaired patients in our sample, but 50 items for impaired patients. Thus, although the standard administration does reduce administration time for normal patients, it provides relatively little reduction in number of items for impaired patients, who typically are also considerably slower and require additional cues. The standard BNT administration with impaired patients may require at least twice as much time (to

8 898 ROGER GRAVES ET AL. Table 4. Reliability and validity data for the Full 60 item test, the start/stop rule scored version, and 12 short forms of the Boston Naming Test using 10th percentile cutoffs. Results ranked by agreement with the full BNT on whole sample abnormal classifications. Test Correlations a 10% Cutoff b Whole sample agreement a AD/VD Patients agreement e alpha a BNT60 MMSE 3MS Kappa nl c abnl d tot < cutoff f nl g abnl h tot 60 Item Full BNT60.90 NA <49 NA NA NA NA 44% NA NA NA BNT Stop Rule NA < % 100% 97% 48% 92% 100% 96% 30 Item Short Forms New BNT < % 92% 97% 44% 95% 93% 94% Williams < % 92% 98% 41% 97% 90% 94% Mack 30B < % 84% 95% 39% 92% 80% 87% Saxton 30A < % 80% 94% 36% 97% 80% 90% Mack 30A < % 73% 91% 32% 100% 73% 88% Saxton 30B < % 53% 89% 23% 100% 53% 80% New Adaptive 30/15 Item.90/ < % 92% 97% 44% 95% 93% 94% 15 Item Short Forms New BNT < % 90% 93% 48% 85% 90% 87% Mack < % 80% 93% 36% 97% 80% 90% Mack 15-4 i < % 76% 91% 38% 90% 73% 83% Lansing < % 65% 90% 33% 92% 67% 81% Mack < % 59% 89% 26% 97% 57% 80% Mack < % 41% 76% 16% 100% 37% 72% Notes: a Data in these columns are based on full sample of 206 patients. b Cutoff established for 10th percentile using data from the 62 normal patients. c This column shows the percentage of the 157 patients classified normal on naming in the full sample by the full 60 item BNT that were also classified normal by the short test (specificity). d This column shows the percentage of the 49 patients classified abnormal on naming by the full 60 item BNT that were also classified abnormal by the short test (specificity). e Data in rightmost three columns based on subset of 69 AD/VD patients. f This column shows the percentage of the 69 AD/VD patients classified abnormal by the test. g This column shows the percentage of the 39 AD/VD patients classified normal on naming by the full 60 item BNT that were also classified normal by the short test (specificity). h This column shows the percentage of the 30 AD/VD patients classified abnormal on naming by the full 60 item BNT that were also classified as abnormal by the short test (sensitivity). i This is also the 15 item short form included in Kaplan, et al. (2001).

9 BOSTON NAMING TEST SHORT FORMS 899 Table 5. Validity Data for the Full 60 Item Test, the Start/stop Rule Scored Version, and New Short Forms of the Boston Naming Test Using 20 th percentile Cutoffs. Test 20% Cutoff b Whole sample agreement a AD/VD Patients agreement e Kappa nl c abnl d tot <cutoff f nl g abnl h tot 60 Item Full BNT60 <52 NA NA NA NA 64% NA NA NA BNT Stop Rule < % 87% 95% 58% 100% 91% 94% New BNT30 < % 95% 94% 65% 88% 96% 93% New Adaptive 30/15 Item < % 95% 94% 65% 88% 96% 93% New BNT15 < % 79% 91% 57% 92% 84% 87% Notes: a Data in these columns are based on full sample of 206 patients. b Cutoff established for 20 th percentile using data from the 62 normal patients. c This column shows the percentage of the 121 patients classified normal on naming in the full sample by the full 60 item BNT that were also classified normal by the short test (specificity). d This column shows the percentage of the 85 patients classified abnormal on naming by the full 60 item BNT that were also classified abnormal by the short test (specificity). e Data in rightmost three columns based on subset of 69 AD/VD patients. f This column shows the percentage of the 69 AD/VD patients classified abnormal by the test. g This column shows the percentage of the 25 AD/VD patients classified normal on naming by the full 60 item BNT that were also classified normal by the short test (specificity). h This column shows the percentage of the 44 AD/VD patients classified abnormal on naming by the full 60 item BNT that were also classified as abnormal by the short test (sensitivity). give 50 items versus 35) and perhaps as much as four times as much time as for unimpaired patients. Thirty and 15 item short forms would therefore be especially useful in reducing test time for impaired patients, if the short forms were reliable and valid. Validity of the new 30 and 15 item short forms was considered with respect to possible use as screening tests for naming impairment among elderly clinic referred patients. Evidence of validity for this application was obtained from several sources, which are discussed according to current conceptions of validity (AERA, et al. 1999). First, evidence based on test content is that the short forms items were obtained from the full BNT and reflect the range of content of the full version except that, for the 15 item version, the range was reduced, mainly through deletion of the easiest items, which would reduce the validity for a more impaired or less educated population but should not affect validity for patients similar to our sample. Second, evidence based on internal structure is that the short form items are homogenous (high internal consistency reliability), unidimensional (PCA results showing a strong first component), and consistent with the one-parameter logistic (Rasch) IRT latent trait measurement model. Rasch model fit has been argued to have important implications for the scientific quality of measurement (cf., Karabatsos, 2001). The relevance of fit with the Rasch model to validity has been extensively discussed by Smith (2001). It is noteworthy that, to our knowledge, neither the full BNT nor any previous short versions of the BNT have been shown to satisfy Rasch or other IRT measurement models. Third, evidence based on relations to other variables consists of multiple results. The short forms scores correlated highly with the scores from the full BNT, indicating that the new tests are providing measures of the same construct (concurrent validity). Further, the short forms showed correlations with the 3MS and MMSE scores that were similar to the correlations of the full BNT with these other tests, indicating that the new tests perform similar to the full BNT in sharing variance with these broader indicators of cognitive impairment. Most importantly for the proposed application, the classifications into normal versus impaired categories that were made by the new short forms agreed well with the classifications made by the full BNT (high Kappa, similar percentage identified as impaired, high overlap with the full BNT s classifications). Our 30 item

10 900 ROGER GRAVES ET AL. form was equivalent to the full 60 item BNT in terms of internal consistency reliability and performed very similar to the full BNT in classifying naming ability of early dementia patients as normal versus impaired (below 10th percentile for normals). Our new 15 item form was nearly as reliable and successful in classification ability as the 30 item form and was superior to all previous 15 item forms. These results are very encouraging. However, cross-validation with a new sample is needed to confirm the agreement data. An adaptive short form was also developed, which involves giving the new 15 item version and stopping there if the score is outside of a confidence interval, otherwise continuing to give the complete 30 item short form. This method provided classification results exactly equivalent to that of the complete 30 item short form, with the benefit that for 75% of patients classified normal, and for 18% patients classified abnormal, only 15 items need be given to obtain very high classification accuracy. The average number of items required with this adaptive test was 19 for patients classified normal and 27 items for patients classified abnormal, a considerable saving over the standard administration. A computerized version of the BNT could likely improve on these results using full computerized adaptive testing based on IRT item difficulties. Evaluation of the short forms determined the degree to which the short form classifications agreed with normal versus impaired naming ability classifications of the full 60 item BNT. These classifications were based on cutoff levels selected in order to provide 90% specificity for each test with the 62 normal patients. As shown in Table 4, agreement with the normal classifications of the full BNT was very good for all 12 short forms. Thus, at least for our population and for our empirically derived cut-off levels, all of the short forms would appear to perform well in not making false positive types of errors with clients who have normal naming ability. There was considerable variation, however, among the short form tests in their capacity to avoid mis-classifying clients who do have impaired naming ability (according to the full BNT). This latter aspect of performance is, of course, of major importance for clinical utility. The AD/VD group was slightly older and less educated than was the normal group for which the cutoff level was derived. In view of the low, but significant correlations between all BNT forms and age and education, a few of the AD/VD patients may have fallen below cutoff (on all forms) because of high age and/or low education rather than because their naming ability was actually below the 10th percentile of normal, if some age and education correction were available. If present, such a spurious effect would not affect our evaluation of concurrent validity because the age and education correlations of the short forms were almost identical to those of the full BNT. Thus, the short forms should have exactly the same age and education bias problem as the full BNT and percent agreement with the full BNT should not be affected by age and education effects. Evaluation of 10 previously published short forms for the purpose of classifying naming ability showed varying success. Among the 30 item forms, the Williams test (Williams et al., 1989) was the standout, performing as well as our new 30 item test in reliability and classification agreement accuracy with the full BNT. The Williams version has now shown excellent results in each of five crossvalidation studies with mostly elderly participants. The Mack 30 item version B (Mack et al., 1992) and the Saxton 30 item version A (Saxton et al., 2000) tests also performed reasonably well with our data, although clearly not as well as the Williams test. Among the previously published 15 item forms, the Mack Form 3 and Mack Form 4 (Mack et al., 1992) showed results that approached our new version and which, in our view, might be considered adequate. Results of two previous studies have also shown support for Form 3 (Fastenau et al., 1998; Lansing et al., 1999). The Mack Form 4 is now included as a short form in the latest BNT (Kaplan et al., 2001). Since these short form results were obtained on a new sample, independent of the samples used in the original developments, our results support the potential utility of these previously proposed versions, at least for classifying naming impairment in an elderly Canadian early dementia clinic referred group. While several of the previously proposed short forms did not yield good results for our population and evaluation criteria, it is entirely possible that they might be very good for other populations or for other applications.

11 BOSTON NAMING TEST SHORT FORMS 901 Results of the short form evaluation indicate that BNT administration time could be reduced by half, with very little loss of accuracy, by using 30 item versions with populations similar to ours and for the purpose of screening for naming impairment. Our new 15 item version retains most of the more difficult items (which provide most of the discriminating power) from the larger 30 item version. Our adaptive 30 item version, which performed as well as the full 30 item short form but requires only 15 items for high and low ability patients, is of particular interest for clinical screening purposes. Potential users of any of these short forms should be familiar with the recommendations of Smith et al. (2000) that provide strong criteria for validation of short versions of clinical tests. For documenting a specific level of naming ability on a continuous scale for clients of a wider range of premorbid ability the full 60 item BNT would be preferred. Until cross-validation studies become available, users should exercise caution in basing clinical judgements on the new short forms, as well as on many of the existing short forms that have limited validation data. Although item response theory has become widely and successfully used in educational assessment, it has so far been minimally exploited in Neuropsychology. The present results provide an illustration of how recent advances in test development methodologies can aid in the creation of clinically useful assessment instruments. ACKNOWLEDGEMENTS Portions of this paper were presented at the 31st Annual Meeting of the International Neuropsychology Society, Honolulu, February REFERENCES Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association. Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum. Clark, C. M. (1986). Statistical models and their application in clinical neuropsychological research and practice. In: S. B. Filskov and T. J. Boll (Eds.), Handbook of clinical neuropsychology. New York: John Wiley & Sons. (Chapter 19, pp ) de la Torre, J. C. (2002). Alzheimer disease as a vascular disorder: Nosological evidence. Stroke, 33, Embretson, S. E. (1996). The new rules of measurement. Psychological Assessment, 8, Fastenau, P. S., Denburg, N. L., & Mauer, B. A. (1998). Parallel short forms for the Boston Naming Test: Psychometric properties and norms for older adults. Journal of Clinical and Experimental Neuropsychology, 20, Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). Mini-Mental State. Journal of Psychiatric Research, 12, Franzen, M. D., Haut, M. W., Rankin, E., & Keefover, R. (1995). Empirical comparison of alternate forms of the Boston Naming Test. The Clinical Neuropsychologist, 9, Kaplan, E., Goodglass, H., & Weintraub, S. (1983). Boston Naming Test. Philadelphia: Lea & Febiger. Kaplan, E., Goodglass, H., & Weintraub, S. (2001). Boston Naming Test, second edition. Philadelphia: Lippincott Williams & Wilkins. Karabatsos, G. (2001). The Rasch model, additive conjoint measurement, and new models of probabilistic measurement theory. Journal of Applied Measurement, 2, Lansing, A. E., Ivnik, R. J., Cullum, C. M., & Randolph, C. (1999). An empirically derived short form of the Boston Naming Test. Archives of Clinical Neuropsychology, 14, Linacre, J. M., & Wright, B. D. (2001). Winsteps. Chicago: MESA Press. Mack, W. J., Freed, D. M., Williams, B. W., & Henderson, V. W. (1992). Boston Naming Test: Shortened versions for use in Alzheimer s disease. Journal of Gerontology: Psychological Sciences, 47, Mitrushina, M., & Satz, P. (1989). Differential decline of specific memory components in normal aging. Brain Dysfunction, 2, Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research. (Expanded edition, Chicago: The University of Chicago Press, 1980.) Saxton, J., Ratcliff, G., Munro, C. A., Coffey, C. E., Becker, J. T., Fried, L., et al. (2000). Normative data on the Boston Naming Test and two equivalent 30- item short forms. The Clinical Neuropsychologist, 14,

12 902 ROGER GRAVES ET AL. Smith, E. V., Jr. (2001). Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective. Journal of Applied Measurement, 2, Smith, G. T., McCarthy, D. M., & Anderson, K. G. (2000). On the sins of short-form development. Psychological Assessment, 12, Smith, R. M., & Miao, C. Y. (1994). Assessing unidimensionality for Rasch measurement. In: M. Wilson (Ed.), Objective measurement: Theory into practice, volume 2. Norwood, NJ: Ablex. Teng, E. L., & Chui, H. C. (1987). The Modified Mini- Mental State (3MS) examination. Journal of Clinical Psychiatry, 48, Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55, Tombaugh, T. N., & Hubley, A. M. (1997). The 60- item Boston Naming Test: Norms for cognitively intact adults aged 25 to 88 years. Journal of Clinical and Experimental Neuropsychology, 19, Van Gorp, W. G., Satz, P., Kiersch, M. E., & Henry, R. (1986). Normative data on the Boston Naming Test for a group of older adults. Journal of Clinical and Experimental Neuropsychology, 8, Williams, B. W., Mack, W., & Henderson, V. W. (1989). Boston Naming Test in Alzheimer s Disease. Neuropsychologia, 27, Wilson, D. T., Wood, R., & Gibbons, R. (1991). Testfact: test scoring, item analysis, and item factor analysis. Chicago: Scientific Software International, Inc.