S peech recognition measures are widely. Evaluation of the Computer-Assisted Speech Perception Assessment Test (CASPA)

Transcription

1 J Am Acad Audiol 12 : (2001) Evaluation of the Computer-Assisted Speech Perception Assessment Test (CASPA) Carol L. Mackersie* Arthur Boothroyd* Donna Minniear* Abstract Interlist equivalency and short-term practice effects were evaluated for the recorded stimuli of the Computer-Assisted Speech Perception Assessment (CASPA) Test. Twenty lists, each consisting of 10 consonant-vowel-consonant words, were administered to 20 adults with normal hearing. The lists were presented at 50 db SPL (Leq) in the presence of spectrally matched steady-state noise (55 db SPL Leq). Phoneme recognition scores for the first list presented were significantly lower than for the second through the twentieth list presented, indicating a small practice effect. Phoneme scores for 4 of the lists (3, 6, 7, and 16) were significantly higher than scores for the remaining 16 lists by approximately 10 percentage points. Eliminating the effects of interlist differences reduced the 95 percent confidence interval of a test score based on a single list from 18.4 to 16.1 percentage points. Although interlist differences have only a small effect on confidence limits, some clinicians may wish to eliminate them by excluding lists 3, 6, 7, and 16 from the test. The practice effect observed here can be eliminated by administering one 10-word practice list before beginning the test. Key Words : Computer-Assisted Speech Perception Assessment (CASPA), list equivalency, outcome assessment, phoneme recognition, speech audiometry, speech perception, word recognition Abbreviations : ANOVA = analysis of variance, CASPA = Computer-Assisted Speech Perception Assessment, CASPA = Computer-Assisted Speech Recognition Assessment, CVC = consonant-vowel-consonant S peech recognition measures are widely used in audiologic practice for diagnostic assessment but are not widely used in hearing aid outcome assessment. Several limitations of performance measures have led to their decline in popularity as a performance component of a hearing aid outcome battery. First, it is widely recognized that conventional speech recognition measures have poor test-retest reliability (Shore et al, 1960 ; Boothroyd, 1968a ; Thornton and Raffin, 1978). Poor reliability compromises test sensitivity, making it difficult to detect differences between speech recognition scores under different listening conditions. Conventional speech recognition measures are also time consuming to *Department of Communicative Disorders, San Diego State University, San Diego, California Reprint requests : Carol L. Mackersie, Department of Communicative Disorders, San Diego State University, 5500 Campanile Drive, San Diego, CA administer and score. Many audiologists argue, therefore, that performance measures of hearing aid benefit are not a realistic option for a typical audiology practice. There are, however, several advantages to including performance measures in a hearing aid outcome battery. First, performance measures can provide objective documentation of speech recognition benefit attributable to hearing aid use. In addition, performance outcome measures can supplement subjective outcome assessments because they tap different and independent aspects of function (e.g., Bentler et al, 1993a, b). There are several considerations in choosing a test protocol for performance outcome assessment. First, to sample a range of listening situations important to the hearing aid user, listeners should be tested at several input levels and in both quiet and noise. Multilevel testing is particularly valuable in the evaluation of the effects of compression on speech recognition. Second, there are advantages to including both word and 390

2 Evaluation of CASPA/Mackersie et al sentence materials in the assessment. The use of word materials allows a valid measure of phonemic perception while minimizing the effects of language knowledge and the use of context (Boothroyd, 1968b ; Boothroyd and Nittrouer, 1988). The use of word materials also permits a detailed analysis of phoneme errors. Sentence materials, on the other hand, are more representative of speech encountered in the real world and allow for such factors as the influence of suprasegmental features and short-term memory. It is possible, however, to predict sentence recognition from both word and phoneme recognition scores (Boothroyd and Nittrouer, 1988). Finally, the reliability, sensitivity, and efficiency of the test must be considered as factors crucial to demonstrating statistically significant differences between speech recognition scores for different conditions within the time constraints of clinical practice. The reliability of' speech recognition tests can be improved by increasing the number of test items. Phoneme scoring increases the number of items in a word recognition test without requiring additional test time (Boothroyd 1968a; Olsen et al, 1997 ; Gelfand 1998). The use of phoneme scoring triples the number of test items in a consonant-vowel-consonant (CVC) word list because three items are scored for each word rather than just one. I The increase in the number of storable test items reduces (improves) the confidence limits of the test. This means that smaller differences between listening conditions can be detected (Boothroyd, 1968a ; Thornton and Raf'fin, 1978). Until recently, phoneme scoring has been regarded as awkward to implement because of extra time required to score the individual phonemes. Two recently developed computerassisted word recognition tests, however, provide the option to score individual phonemes easily and quickly. Gelfand (1998) developed the Computer-Assisted Speech Recognition Assessment (CASRA), a computer-administered test that was designed to provide 450 storable items in 50 three-word trials. At each trial, the subject is asked to repeat three CVC words. The tester then indicates on the computer keyboard which of the nine phonemes are correct. By using three words in each trial rather than one, the number of test items in a 50-trial test increases to 150 if whole-word scoring is used and to 450 if phoneme scoring is used. Each 50-trial test takes approximately 6 minutes to administer. Unfortunately, at the time of writing, this test was not expected to be commercially available in the foreseeable future (S Gelfand, personal communication, 2000). Another computer-assisted test that is currently in use by both researchers and clinicians is the Computer-Assisted Speech Perception Assessment Test (CASPA) (Boothroyd,1999). The CASPA test, which was originally designed to assess hearing aid outcome, permits multilevel testing in quiet and in noise. The interactive software interfaces with the tester to allow computer presentation and scoring of the CVC stimuli and provides separate scores for words, phonemes, consonants, and vowels. Like the CASPA test described above, the phoneme scoring feature of CASPA increases the number of test items to improve the confidence intervals of the test scores without additional administration time. It takes approximately 1 minute to administer one 10-word list, which yields a score based on 30 phonemes. By administering two to three lists under each condition, one can substantially improve the likelihood of detecting significant differences between phoneme recognition scores measured under different listening conditions. Moreover, multilevel testing with the CASPA test is faster than multilevel testing with a conventional word recognition test because of CASPA's ease of administration and automatic phoneme scoring. Although the CASPA test is increasing in popularity among researchers and clinicians, there is currently no information available on the list equivalency for the recorded stimuli or on practice effects for these materials. The purpose of this study was to gather this information. Participants METHOD Twenty normal-hearing adults served as listeners. Ages ranged from 19 to 40 years, with a mean of 23.4 years. All listeners had pure-tone thresholds of 20 db HL or less for octave frequencies between 250 and 8000 Hz CASPA Stimuli 'Lexical redundancy actually reduces the number of statistically independent items from 3 to approximately 2.5 for listeners with normal language skills (Boothroyd and Nittrouer, 1988) The CASPA stimuli consist of 20 lists of CVC words with 10 words in each list. The lists are isophonemic, that is, each list contains one 391

3 Journal of the American Academy of Audiology/Volume 12, Number 8, September 2001 example of each of the same 30 phonemes. The 20 CASPA lists include the original 15 AB isophonemic word lists (Boothroyd, 1968a, b) and 5 new isophonemic lists (Oruganti, 2000). The stimuli available with the CASPA test at the time of writing are digitized recordings of these 20 lists spoken by a female talker. The noise used in the test is steady state and is spectrally matched to the female talker's long-term average spectrum. When testing in noise, the speech items are temporally centered in the noise, the duration of which is fixed at 2 seconds. The word lists are provided in the Appendix. CASPA Software The software controls the presentation of the CASPA words and provides the option to present stimuli in quiet or in the presence of noise. The noise and speech can be delivered through separate channels (stereo) or mixed in a single channel (mono). The program can be used to provide speech input to an audiometer, an amplified loudspeaker, or another external device. The presentation level can be controlled either through the CASPA software or with the attenuator of an audiometer. If the CASPA software is used, the speech level is selectable in 5-dB steps between 45 and 75 db SPL (Leq), and the noise level is fixed at 55 db SPL. If an audiometer attenuator is used to control the presentation levels, then a wider range of intensities can be selected, and both the speech and noise levels can be adjusted. The software permits randomization of the order of presentation of the lists and of the items within the lists. Figure 1 shows an example of a test screen as it appears to the tester during administration. Following presentation of a test item, the tester types the listener's response into the computer. The computer then displays both the test item and the listener's response. Using the computer mouse, the tester selects the box(es) corresponding to the phonemes repeated correctly. In the particular example shown in Figure 1, the test item was "both" and the listener's response was "boat," so the tester marked the first consonant and the vowel correct. For testers who are not comfortable with the mouse, responses can be scored from the keyboard using either "1," "2," "3" or "C," "V" "B" to score the first, second, and third phonemes, respectively. The option also exists to postpone scoring until after the subject has left. For each list, the software tabulates the total number of words, phonemes, consonants, and vowels repeated correctly (see Fig. l, lower left). In addition, both the stimuli and corresponding subject responses are logged on the computer's hard drive for each test item. At the end of a list or series of tests, the results can be printed through the software in graphic (performance-intensity functions) or tabular format. Alternatively, the results can be read into a spreadsheet program for error analysis. Procedures The stimuli were played through a Creative- Labs SB-16 Sound Card at the rate of k/sec with 16-bit resolution. The output of the sound card was routed to the tape input of a Grason-Stadler GSI-16 audiometer and delivered monaurally to TDH-59 earphones. Speech stimuli were presented at 50 db SPL (Leq) in the presence of the spectrally matched noise (55 db SPL). A signal-to-noise ratio of -5 db was used to avoid ceiling- and floor-level performance. Avoiding ceiling- and floor-level performance increased the likelihood of detecting interlist differences. In the interest of efficiency, no carrier phrase was used. Listeners were instructed to repeat the words and were encouraged to guess. The tester scored the listeners'responses using the computer software described above. The presentation order of the 20 lists was counterbalanced among listeners using a Latinsquare design. The order of items within each list was randomized each time the list was presented. Testing was completed in one 40-minute session. Data were transformed to rationalized arcsine units prior to statistical analyses to stabilize the error variance (Studebaker, 1985). Order Effects RESULTS Group mean phoneme recognition is shown in Figure 2 plotted as a function of presentation order. Note that the score for the first list presented is lower than the scores for the subsequent lists. The solid line in the figure is a least-squares fit to an exponential growth function. The formula for this function is Y = a + (b - a)* (1- ec-cx-i)i~>) where a = intercept (score on trial 1), b = asymptote, and c = time constant (i.e., the number of lists required for the gap between the current score and the asymptote to be reduced by approx- 392

4 Evaluation of CASPA/Mackersie et al Subject ID (max. 8 characters) S1 Type subject's response and hit Enter If the whole word was correct just hit Enter. If the subject gave no Carrier None response type 0 (zero) or-. [ boat Speech level in dbspl (leq) 50 The stimulus word was Acoustics [ Noise SpeechiNoise mono J o ~ th Listening conditions for the two ears : Left J Right List # J Current word # Earph-~ Check correct sounds below and click OK Vowel Consonant 1 FV r r (B)Consonant 2 None (- All 3 Start over ~+ll Judge Loudness? F- Randomize wor d s? loo ear graph Q u i e t N i Postpone scoring? r - Randomize lists? p h Aided: 50 Other : o o c 1 v c 2 C ons Words Phonemes % unaided: o Return 0 Preferences Instruction MatrixTesting Normal: d BSP L(leq) BSI ~J Phoneme scoring o se Data retrieval Figure l CASPA test screen showing test parameters and sample trial for one subject. The test stimulus ("both") is shown in the second section at the right, the listener response ("boat") is shown at the top right, and the score for the test item is shown in the middle-right section. imately two-thirds). Note that the nonlinear correlation coefficient reflects a significant increase in phoneme recognition scores as the number of lists presented increased (r [171 =.56, p <.01). The mean score rises to within 3 percentage points (one phoneme) of the maximum after 0.7 lists. That is, by the second list presented, the mean score is close to the asymp- tote. A one-way repeated-measures analysis of variance (ANOVA) confirms that the significant effect of presentation order (F = 6.69, df = 19, 361, p <.0001) disappears when the first list presented is excluded from the analysis (F = 1.30, of = 18, 342, p >.05). Taken together, the nonlinear regression and ANOVAs indicate the presence of a small practice effect, but performance is stabilized by the time the second list is presented. List Differences Presentation Order Figure 2 Mean scores - 1 standard error for the first through the twentieth list presented, arranged in order of presentation. The solid line in the figure is the curve resulting from a least-squares fit to an exponential growth function. The mean scores for individual lists, shown in Figure 3, ranged from 58 to 74 percent, with a mean of 64.7 percent. A one-way repeatedmeasures ANOVA showed a significant effect of list (F = 6.38, df = 1, 19, p <.00001). Based on key's honest significant difference post hoc analyses, the lists can be divided into two categories : "easier" lists with higher mean scores, shown as solid squares in Figure 3 (lists 3, 6, 7, and 16), and the remaining 16 "harder" lists, shown as open circles in the figure. The mean scores for the easier and harder lists were 72.0 and 62.9 percent, respectively. The middle, top, and bottom horizontal lines in the figure indicate the mean of the 16 harder lists ± 2 SD, respectively. The post hoc analyses showed that 393

5 Journal of the American Academy of Audiology/Volume 12, Number 8, September Table 1 Standard Error of Repeated Scores 90 Based on a Single List 80 V d o ~.. ~ Tn o --- Source of Variability Eliminated Standard Error (%) -" None* U Subjectt 9.2 N 40 List and subject' 8.1 E 30 d C *Overall test variability with all sources of variability O 20 included ; 'test variability with only the variability attributable to t a 10 subjects eliminated ; 'test variability with both list and subject sources of variability eliminated List Number Figure 3 Mean scores ± 1 standard error for each list arranged by list number. The filled squares indicate the scores for four "easier" lists. The open circles indicate scores for the remaining 16 "harder" lists. The solid line represents the mean of the 16 "harder" lists, and the broken lines represent 2 SD above and below this mean. mean scores within the easier and harder list groups did not differ significantly from one another (p >.05). These findings suggest that selecting lists from either the group of easier or harder lists will minimize list-difference effects. Using lists from both groups, however, could lead to differences between scores that reflect interlist differences rather than differences between listening conditions. Contribution of List Equivalency to Overall Variability Although these list differences are statistically significant, it is important to consider whether these differences are substantial in light of the overall variability of the test measures and the interaction between subject and list. From the ANOVA, the influence of list differences on the confidence limits of the test scores was examined by calculating the standard error of an individual subject's mean score with and without the contribution of list differences. These standard errors were calculated by first determining the sums of squares for list, subject, and the list-by-subject interaction. Standard errors were then calculated and compared for various combinations of effects. As shown in Table 1, the standard error improves by 0.5 percentage points when the variability attributable to subject is accounted for. When the additional factor of list was accounted for, the variability decreases from 9.2 to 8.1, an additional 1.1 percentage points. The slight reduction in test variability with the elimination of the effect of interlist differences means that the test would be slightly more sensitive to differences between listening conditions. By doubling the standard error, one arrives at an estimate of the 95 percent confidence limits of a score. Based on the analysis described above, the 95 percent confidence limits of a score based on a single list can be reduced by 2 percentage points (roughly from -!- 18 to ± 16 percentage points) if the effect of interlist differences is removed. This effect, however, is small compared with the effect of using two or more lists under each condition. Practice Effects DISCUSSION The finding that the scores for the first list presented were lower than scores for subsequent lists suggests that under the conditions used in the study, normal-hearing listeners need a small amount of practice before their performance stabilizes. The improvement of scores observed after the first list was presented may be a result of adaptation to the talker, adaptation to the noise, improved attention, or a combination of these factors. Based on the practice effects observed in this study, we recommend that one 10-word practice list be administered before collecting clinical or research data for a listener. Administration of a practice list should only require an additional minute of test time. It is important to note that the minimal practice effects observed in this study are for administration of different lists. It is possible that greater practice effects would be observed if the same list was used more than once. For example, if a listener is exposed to a list under optimal listening conditions and then tested using the same list under a more difficult condition, prior exposure to the list may inflate the second score. 394

6 Evaluation of CASPA/Mackersie et al List Differences There are several possible explanations for the interlist differences observed in this study. One possibility relates to acoustic differences between test items in the different lists. Care was taken to control the acoustic content of the items by balancing the lists for phonemic content and equalizing the overall root mean square of the test items in recording. Nevertheless, it is possible that differences in the frequency and range of formant transitions made consonants easier to recognize in some vowel contexts than in others. Another possibility is that linguistic factors influenced the difficulty of the lists. For example, items in the lists may differ in word frequency and/or neighborhood density characteristics. Words that occur more frequently in a language are known to be easier to recognize than those occurring less frequently (see Luce and Pisoni, 1998, for a review). In addition, the number of words that are in the lexical neighborhood (i.e., that are phonetically similar to test items) has been shown to influence the likelihood that a given word will be recognized correctly (Luce and Pisoni, 1998 ; Meyer and Pisoni, 1999). It is possible that a revision of the AB word lists to balance them for phonotactic characteristics, word frequency, and lexical neighborhood would eliminate the interlist differences observed here. The present findings, however, suggest that the quantitative impact on the confidence limits of a test score would be small. The reason is that these confidence limits are dominated by the interaction between list and subject. In other words, the relative difficulty of the lists differs considerably from listener to listener. By eliminating list differences that are common to all listeners, one would improve the confidence limits only by a small amount. Although the current findings indicate only a small benefit, clinicians and researchers may wish to control CASPA interlist differences by simply avoiding the four "easier" lists (3, 6, 7, 16). An alternative approach would be to apply a correction factor to the scores obtained using the four easy lists to reduce the differences. A suitable correction, based on the work of Boothroyd and Nittrouer (1988), would be PC = [1- (1- Pu/100) 0.7s] " 100, where Pc = correct phoneme recognition score in percent and Pu = uncorrected phoneme recognition score in percent. Note that this equation leaves scores of 0 or 100 percent unchanged. It is important to bear in mind that the findings described in this article are based on data from normal-hearing listeners tested with a specific type of noise at a single signal-tonoise ratio. It is possible that a different pattern of practice effects and interlist differences would be observed for a clinical population of children, older adults, and listeners with hearing loss. Furthermore, the effects of cochlear damage and/or hearing aid filtering are different than those of the noise used in the current study and therefore could potentially have a different impact on both practice and interlist differences. The findings observed in this study are but one example of how list differences and practice can affect speech recognition performance. Numerous examples exist in the literature for speech recognition tests that are commonly used by audiologists (e.g., Her Kirk et al, 1999 ; Bentler, 2000 ; Stockley and Green, 2000). These examples remind audiologists that issues related to practice effects and interlist differences need to be considered for any speech recognition measure used in clinical practice. CONCLUSIONS 1. Under the conditions of this study, normalhearing adults listening in noise do not demonstrate significant practice effects after the first list is presented. 2. Under the conditions of this study, normalhearing listeners find 4 of the lists significantly easier than the remaining 16 lists. 3. The interlist difference effect can be eliminated either by avoiding the easier lists or by applying a correction factor. In terms of the confidence limits of test scores, however, the benefit of controlling for interlist differences will be small compared with the benefit of using more than one list under each test condition. REFERENCES Bentler R. (2000). List equivalency and test-retest reliability of the Speech in Noise Test. Am JAudiol 9:1-17. Bentler R, Neibuhr D, Getta J, Anderson C. (1993a). Longitudinal study of hearing aid effectiveness I. Objective measures. J Speech Hear Res 36 : Bentler R, Neibuhr D, Getta J, Anderson C. (1993b). Longitudinal study of hearing aid effectiveness II. Subjective measures. J Speech Hear Res 36 : Boothroyd A. (1968a). Developments in speech audiometry. Br JAudiol 2:

7 Journal of the American Academy of Audiology/Volume 12, Number 8, September 2001 Boothroyd A. (1968b). Statistical theory of the speech discrimination score. J Acoust Soc Am 43: BoothroydA. (1999). Computer-Assisted Speech Perception Assessment (CASPA), Version 3.0 [computer software]. Boothroyd A, Nittrouer S. (1988). Mathematical treatment of context effects in phoneme and word recognition. JAcoust Soc Am 84 : Gelfand S. (1998). Optimizing the reliability of speech recognition scores. J Speech Lang Hear Res 41 : Iler Kirk K, Eisenberg LS, Martinez AS, Hay-McCutcheon M. (1999). Lexical Neighborhood Test: test-retest reliability and interlist equivalency. J Am Acad Audiol 10 : Luce P, Pisoni D. (1998). Recognizing spoken words : the neighborhood activation model. Ear Hear 19 :1-36. Meyer T, Pisoni D. (1999). Some computational analyses of the PB-K test : effects of frequency and lexical density on spoken word recognition. Ear Hear 20: Olsen WO, Van Tasell DJ, Speaks CE. (1997). Phoneme and word recognition for words in isolation and in sentences. Ear Hear 18 : Oruganti B. (2000). The Effects of High-Frequency Emphasis and Amplitude Compression on the Short-Term Intensity Range of Speech. Doctoral dissertation, City University of New York, New York. Shore 1, Bilger R, Hirsh I. (1960). Hearing aid evaluation : reliability of repeated measures. J Speech Hear Disord 25: Studebaker G. (1985). A "rationalized" arcsine transform. J Speech Hear Res 28 : Stockley KB, Green WB. (2000). Interlist equivalency of the Northwestern University Auditory Test No. 6 in quiet and noise with adult hearing-impaired individuals. JAm Acad Audiol 11 : Thornton A, Raffin M. (1978). Speech discrimination scores modeled as a binomial variable. J Speech Hear Res 21: Appendix CASPA Word Lists List 1 List 2 List 3 List 4 List 5 List 6 List 7 List 8 List 9 List 10 ship fish thug fun fib fill badge bath hush jug rug duck witch will thatch catch hutch hum gas latch fan path teak vat sum thumb kill dig thin wick cheek cheese wrap shape heel heap thighs five fake faith haze race vice wreath wide wise wave ways chime sign dice hive jail hide rake rave reap reach weave beep both bone hen guess goes got foam joke jet hem well wedge shows comb shop shown goose noose rob rod jot log food choose vet bed not pot dope vote move tomb bomb job june juice shed shell lose shoes List 11 List 12 List 13 List 14 List 15 List 16 List 17 List 18 List 19 List 20 math have kiss wish hug wage jade shave vase cave hip wig buzz Dutch dish rag cash jazz cab rash gun buff hash jam ban beach thief theme teach tease ride mice thieve heath rage chef set fetch death jell siege teeth gate laze chief dime wine height nice guide veil jays wife bike pies thick give win fig pin chose poach pole rove wet love rub suck rush fuss shoot rule wretch pet cove zone hole robe hope home web den dodge fog loose hop chop dog lodge watch cough shock moon soon moth suit zoom pool womb booth