Brain responses reveal the learning of foreign language phonemes

Psychophysiology, 36 ~1999!, 638 642. Cambridge University Press. Printed in the USA. Copyright 1999 Society for Psychophysiological Research Brain responses reveal the learning of foreign language phonemes ISTVÁN WINKLER, a,b TEIJA KUJALA, b HANNU TIITINEN, b PÄIVI SIVONEN, b PAAVO ALKU, c ANNE LEHTOKOSKI, b ISTVÁN CZIGLER, a VALÉRIA CSÉPE, a RISTO J. ILMONIEMI, d and RISTO NÄÄTÄNEN b,d a Institute for Psychology, Hungarian Academy of Sciences, Budapest, Hungary b Cognitive Brain Research Unit, Department of Psychology, University of Helsinki, Finland c Department of Applied Physics, Electronics and Information Technology, University of Turku, Finland d BioMag Laboratory, Medical Engineering Centre, Helsinki University Central Hospital, Finland Abstract Learning to speak a new language requires the formation of recognition patterns for the speech sounds specific to the newly acquired language. The present study demonstrates the dynamic nature of cortical memory representations for phonemes in adults by using the mismatch negativity ~MMN! event-related potential. We studied Hungarian and Finnish subjects, dividing the Hungarians into a naive ~no knowledge of Finnish! and a fluent ~in Finnish! group. We found that the MMN for a contrast between two Finnish phonemes was elicited in the fluent Hungarians but not in the naive Hungarians. This result indicates that the fluent Hungarians developed cortical memory representations for the Finnish phoneme system that enabled them to preattentively categorize phonemes specific to this language. Descriptors: Language learning, Phoneme system, Vowel contrast, Event-related brain potentials, Mismatch negativity, Brain plasticity This work was supported by the Hungarian National Research Fund ~OTKA T022681!, the Academy of Finland, and the University of Helsinki, Finland. Address reprint requests to: Dr. István Winkler, Institute for Psychology, Hungarian Academy of Sciences, P. O. Box 398, Budapest, H-1394, Hungary. E-mail: winkler@cogpsyphy.hu. Acquiring fluent command of a language requires plastic changes in the neuronal circuitry of the human brain that enable one to correctly perceive the new speech sounds ~Kuhl, 1993; Näätänen & Tiitinen, 1997!. Recent studies ~Merzenich et al., 1996; Tallal et al., 1996! have emphasized the significance of accurate perception of speech elements, such as phonemes, for the understanding of spoken language. The inability to discriminate speech elements causes problems in language processing in children, whose linguistic abilities were improved by discrimination training on these elements. Thus, to reach fluency in a foreign language, one has to learn to discriminate its fine acoustic details. Näätänen et al. ~1997! showed that speech-sound representations can be probed with a cortical response called the mismatch negativity ~MMN, for a review, see Näätänen, 1992! and its magnetic equivalent ~MMNm!, which provide an objective index of the bottom-up processing of auditory stimulus events. The MMN is elicited by a change ~deviant! in an otherwise regular sequence of discrete auditory stimuli ~standards!, reflecting incongruence between the deviant and the representation formed by the standard sounds. The MMN is generated in the auditory cortex ~for a review, see Alho, 1995! for any discriminable change in simple acoustic and complex phonetic stimulus features. The elicitation of the MMN component does not require the subject s attention to be focused on the target sounds nor is it sensitive to top-down processes ~e.g., Alho & Sinervo, 1997; for reviews, see Näätänen, 1992; Schröger, 1997!. Therefore the MMN component can be used to index the outcome of preattentive auditory processing. The cerebral process generating the MMN is of crucial importance for subsequent higher order cognitive processes. The MMN governs the timing of attentive behavioral responses ~Tiitinen, May, Reinikainen, & Näätänen, 1994!. Conversely, blocking the memorytrace formation with a masking sound abolishes the MMN and degrades voluntary stimulus discrimination ~Winkler, Reinikainen, & Näätänen, 1993!. The memory traces reflected by the MMN are adaptive. Originally indiscernible complex tonal patterns can become discernible via active training, with the MMN emergence paralleling this learning process ~Kraus et al., 1996; Näätänen, Schröger, Karakas, Tervaniemi, & Paavilainen, 1993!. Our recent study showed that the MMN reflects memory traces specific to the mother tongue ~Näätänen et al., 1997!. These traces are formed as a result of early exposure to one s native language environment ~Cheour et al., 1998!. Language-specific memory traces were demonstrated by presenting vowels of two closely related language groups, Estonian and Finnish, to speakers of these languages ~Näätänen et al., 1997!. Estonians responded with a large MMN to a vowel of their own language ~0õ0! when it was presented as a deviant stimulus. In contrast, the same deviant stimulus elicited a much smaller MMN response in Finnish subjects, who 638

MMN to learned vowel contrasts 639 lacked a proper memory representation for this particular phoneme. In the present study, we tested the formation of new vowel representations in adults as a result of learning a foreign language. Subjects were selected from speakers of two remotely related languages, Hungarian and Finnish. Hungarians were divided into two subgroups, one having acquired a fluent command of Finnish language after childhood ~fluent! and the other with no prior exposure to Finnish ~naive!. We compared these groups on the MMNs elicited by a vowel contrast that is relevant in Finnish but not in Hungarian ~0æ0 vs. 0e0! and by another vowel contrast ~0y0 vs. 0e0! that is relevant in both languages. In addition, we measured how well subjects could identify the two Finnish e vowels ~0æ0 and 0e0!. Methods Vowel synthesis and selection. Forty-two isolated vowels were synthesized with their F1 F4 parameters equidistantly covering the four-dimensional straight line connecting the centers of the Finnish 0e0 and 0æ0 area in the F1-F4 formant frequency space. In addition, a near-prototype 0y0 was also created. Stimuli were generated using the semisynthetic approach, enabling production of natural-sounding phonetic material with adjustable formant structure. This approach is based on the source-filter theory ~Fant, 1960! of speech, which assumes that vowel sounds are produced as a cascade of three processes ~glottal excitation, vocal tract filtering, and the lip-radiation effect!. In the semisynthetic scheme, an estimate for the glottal excitation is first computed from a natural speech sound using an inverse filtering technique ~Alku, 1992!. Speech sounds are then generated by processing this glottal waveform with a digital filter modeling the vocal tract and finally with another filter that models the lip-radiation effect. In the present study, the glottal excitation was estimated from a male voice ~vowel 0a0, normal phonation, F0 114 Hz!. The vocal tract was estimated using an eighth-order digital all-pole filter and the lipradiation effect with a fixed differentiator. Twelve native Finnish speakers ~7 females; 20 35 years of age! and 14 native Hungarian speakers with no training in the Finnish language ~3 females; 18 24 years of age! categorized the synthesized vowels ~10 repetitions of each in a randomized sequence! and evaluated on a 5-point scale the similarity of each of them with a typically pronounced vowel of the selected category ~the procedure was described in detail in a previous study by Winkler et al., 1999!. Two synthesized vowels were selected for the present study, one reliably categorized as 0e0, the other as 0æ0 by Finns, both being perceived as 0E0 by Hungarians ~Table 1!. This was possible because the Hungarian 0E0 vowel overlaps both Finnish e sounds. Table 1. F1 F4 Formant Frequencies of the Synthesized 0e0, 0æ0, and 0y0 Vowels Frequency ~Hz! Vowels F1 F2 F3 F4 0e0 550 1955 2576 3338 0æ0 583 1874 2535 3285 0y0 290 1960 2380 3300 Electrophysiological experiment. Ten native Finnish speakers ~4 females; 21 42 years of age!, 10 naive ~3 females; 19 36 years of age!, and 10 fluent native Hungarians ~4 females; 21 44 years of age! participated in this experiment. Hungarians in the naive group had no prior exposure to Finnish, whereas the fluent Hungarians have been living in Finland for a period of 2 13 years prior to the experiment and have acquired fluent command of Finnish. All fluent Hungarians started to learn Finnish between 13 and 32 years of age with the exception of one man, who started his Finnish training at the age of 7. None of the subjects participated in the above vowel categorization experiment. Finnish and fluent Hungarian subjects were tested in Helsinki, and naive Hungarians were tested in Budapest using identical equipment and applying the same experimental procedures. Experiments were performed in an acoustically and electrically shielded room. Two blocks of 700 vowels were binaurally presented through headphones to reading subjects at a stimulus onset asynchrony ~SOA! of 1,200 ms. The 0e0 vowel was the standard ~82.5%!, and 0æ0 ~15%! and 0y0 ~2.5%! were the deviant stimuli. All stimuli were 70 db in intensity and 165 ms in duration. The nose-referenced electroencephalogram ~EEG, sampling rate 250 Hz! was recorded with SYNAMPS amplifiers ~Neuro Scan! from Fpz, Fz, Cz, and Pz ~10-20 system!, the two mastoids ~Lm and Rm!, and two electrodes over each hemisphere at the 103 and 203 points of the coronal line connecting the mastoids to Fz. Eye movements were monitored with electrodes placed above and below the right eye ~vertical electrooculogram @EOG#! and with electrodes placed near the outer canthi of the eyes ~horizontal EOG!. Trials with artifacts exceeding 150 mv at any EEG or EOG channel were discarded. The EEG was digitally filtered ~passband 1.5 30 Hz!. Epochs of 400 ms duration ~including a 100-ms prestimulus interval! were separately averaged for each stimulus class. MMN amplitudes were measured from the frontal ~Fz! deviantminus-standard differences as the mean amplitude in the 130 190-ms interval referred to the mean amplitude in the prestimulus interval. The duration of the MMN response to the 0e0 to 0æ0 change was determined separately for each subject as the interval around the MMN peak within which the response amplitude exceeded 40% of the individual MMN peak amplitude. MMN duration was only measured in those subjects who showed a prominent frontocentral negative deviant-minus-standard difference wave peaking in the 100 200-ms poststimulus interval. Vowel-identification experiment. Ten native Finnish speakers ~5 females; 22 38 years of age!, 10 naive ~6 females; 20 38 years of age!, and 10 fluent Hungarians ~4 females; 21 34 years of age! participated in this experiment. The fluent Hungarians had lived in Finland for a period of 2 13 years prior to the experiment and acquired good command of Finnish. Seven fluent Hungarian and 7 Finnish subjects and 1 naive Hungarian subject also participated in the electrophysiological experiment, which took place prior to the vowel-identification experiment. None of the subjects participated in the vowel categorization experiment. At the beginning of the stimulus block, the subject was presented 5 times with the 0e0 0æ0 pair, the two vowels always being delivered in the same order. Subjects were instructed to carefully listen to these two different sounds so they could identify them in the subsequent stimulus sequence. The test sequence was a randomized series of the 0e0 and 0æ0 vowels delivered equiprobably at 1,200 ms SOA. In a forced-choice reaction task, subjects had to press one of two buttons after each stimulus to indicate whether the sound was identical to the first or the second sound of the initial pairs.

640 I. Winkler et al. Results The experiments revealed a striking difference in the response patterns of the three groups. Finns ~9 of10!and fluent Hungarians ~9 of 10!showed sizable MMN responses to the 0æ0 sound infrequently appearing in the repetitive series of 0e0 vowels, whereas naive Hungarians displayed no detectable MMN response ~except 1of10!in this situation ~Figure 1 and Table 2!. The MMN amplitude was significantly larger in the Finns and fluent Hungarians than in the naive Hungarians, one-way analysis of variance @ANOVA#: F~2,27! 8.03, p.01, with t~27! 3.33 and 3.60, p.01 both, for post hoc comparisons of the naive with the Finnish and fluent groups, respectively. The Finns and the fluent Hungarians did not significantly differ from each other either in the amplitude, post hoc comparison, t~27! 0.27, or in the duration, two-sample t-test, t~16! 1.53, between the 9 Finns ~M 60.67! Table 2. Frontal (Fz) Deviant-Minus-Standard Difference Amplitudes Measured from the 130 190-ms Poststimulus Interval Vowel Finns Difference Amplitude ~mv! Fluent Hungarians Naive Hungarians 0æ0 deviant 0.51 6 0.11* 0.58 6 0.17* 0.36 6 0.25 0y0 deviant 3.21 6 0.71* 3.12 6 0.50* 3.12 6 0.30* Note: Values are group averaged 6 SEM. *p.01, one-sample t-test. and 9 fluent Hungarians ~M 46.22!, of the MMN response elicited by the 0æ0 0e0 contrast. All three groups ~and each subject within the groups!, however, displayed MMNs of very similar amplitude, one-way ANOVA: F~2,27! 0.01, to the rare 0y0 vowel ~Table 2!. Finns and fluent Hungarians were significantly faster and more accurate in identifying the Finnish 0æ0 and 0e0 than were naive Hungarians ~Figure 2!, one-way ANOVA for the hit rates: F~2,27! 19.56, p.001, with t~27! 5.99 and 4.56, p.001 both, for post hoc comparisons of the naive with the Finnish and the fluent groups, respectively. No significant difference was found between the Finns and the fluent Hungarians, t~27! 1.43. Oneway ANOVA for the reaction times: F~2,27! 4.34, p.05, t~27! 2.72 and 2.33, p.05 both, and t~27! 0.39 ~testing structure as above!. Naive Hungarians performed at chance level. These subjects could not identify the Finnish vowels. Discussion The present results demonstrate that learning a foreign language gives rise to long-term plastic changes of the brain mechanisms of phonetic analysis in adults. We found that an across categoryboundary vowel contrast specific to the Finnish language elicited Figure 1. MMN to vowel contrasts. Group-average frontal ~Fz! MMN responses in Finns ~dashed line! and in fluent Hungarians ~Finnish speaking; thick continuous line! and naive Hungarians ~not Finnish speaking; thin continuous line! to rare 0æ0 ~top panel! and 0y0 ~bottom panel! vowels presented in repetitive series of the 0e0 vowel. The 0e0 versus 0æ0 contrast, which is relevant in Finnish but not in Hungarian, did not elicit an MMN in naive Hungarians, whereas fluent Hungarians showed an MMN very similar to that of the native Finnish speakers. The 0e0 versus 0y0 contrast, which is relevant in both languages, elicited almost identical MMN responses in all three groups of subjects. Figure 2. Vowel identification. The group-averaged correct identification rates ~left panel, SEM values marked above each bar! of Finns approached 90% and that of fluent Hungarians was 80%, demonstrating that they were able to reliably identify both Finnish vowels, whereas naive Hungarians performed at chance level. Finns and fluent Hungarians were also faster in the identification task than were naive Hungarians ~right panel: groupaveraged reaction times, SEM values above the bars!.

MMN to learned vowel contrasts 641 a large MMN in Finnish-speaking Hungarians, whereas Hungarians who had not learned Finnish showed no MMN. The very similar MMNs elicited in all three subject groups by the change that fell across vowel category boundaries in both languages indicate that there were no systematic differences between subjects of the different groups in their ability to detect auditory and0or phonetic deviance. Therefore, the difference found between the two Hungarian groups reflects the effect of learning in the fluent Hungarians who developed representations for the Finnish phonemes during their language training. Because the MMN reflects preattentive processing, the present results suggest that acquiring fluent command of a second language in adulthood enables one to process the phonemes of this language preattentively, similarly to native speakers of this language. The MMN was elicited in fluent Hungarians by the Finnish phoneme contrast despite the fact that they were engaged in a task not related to the target stimuli. Thus, the phonetic aspects of learning one s mother tongue ~Cheour et al., 1998! or a second language might be similar. The close correspondence between the MMN responses and the identification performance in the present experiment supports the hypothesis that the preattentive MMNgenerating process indexes stimulus representations that also underlie voluntary discrimination. The larger MMN amplitude elicited by the rare 0y0 than by the 0æ0 sound in the Finns and the fluent Hungarians and in the naive Hungarians is probably caused by two factors: ~a! the lower probability of the 0y0 than the 0æ0 sound within the stimulus blocks ~see Näätänen, 1992! and ~b! the larger physical separation between 0y0 and 0e0 than between 0æ0 and 0e0. Näätänen et al. ~1997! and Winkler et al. ~1999! found that the MMN to acrosscategory phonetic changes reflects both phonetic and auditory deviance. This finding brings up the question of why no MMN was elicited by the change between the two Finnish e sounds in the naive Hungarians. The lack of an MMN suggests that these subjects did not detect the acoustic difference between the present exemplars of the two Finnish e sounds, which they perceived as variants of the Hungarian 0E0. The chance-level identification performance of this group supports this conclusion. Near-prototype vowel variants are less discriminable than variants with the same amount of acoustic separation that are, however, farther from the prototype ~the perceptual magnet effect; see Kuhl, 1993!. The MMN reflects the perceptual magnet effect, demonstrating that this effect originates in preattentive auditory processing ~Aaltonen, Eerola, Hellstrom, Uusipaikka, & Lang, 1997!. The prototype of the Hungarian 0E0 lies close to the border between the two Finnish es. Because the present variants of 0e0 and 0æ0 were near to this border and thus to the prototype of 0E0, Hungarians with no prior training in Finnish could not discriminate between them. However, for the fluent Hungarians, who developed representations for the two Finnish e sounds ~and thus for them these two vowel variants fell across a phonetic category boundary!, the same physical difference became easily discriminable. The reduced discriminability between variants of the same phoneme might stem from the reduction of the sensory resolution of those phonetic feature ranges that do not include category boundaries in one s mother tongue ~e.g., Dehaene-Lambertz, 1997!. If this were true, then the present results suggest that the sensory resolution of phonetic features can be improved by training even in adults, which exemplifies the plasticity of the mature human brain ~see also Merzenich et al., 1996!. New phoneme representations are formed over a period of several months or years. However, previous observations have shown that when subjects try to learn to discriminate between two slightly different stimuli, representations for these stimuli can be developed within a time frame of up to only a few hours ~Kraus et al., 1996; Näätänen et al., 1993!. Once established, these representations function preattentively in a bottom-up fashion, providing sensory information for higher order cognitive processes. These observations clearly accentuate the role of bottom-up processes as a prerequisite for cognition proper. Recent experimental evidence suggests that learning a second language is associated with an increased variability in the activated brain areas, especially with regard to the lateralization of brain activity ~Dehaene et al., 1997; Kim, Relkin, Lee, & Hirsch, 1997!. Previous results ~Näätänen et al., 1997!, however, have demonstrated that phoneme-level operations occur in a dedicated neuronal network located in the auditory cortex. The full range of complex lexical, syntactic, and semantic operations cause more variable and widespread brain activity. Therefore, studies focusing on specific subfunctions of the language system are required to reveal the details of speech processing. REFERENCES Aaltonen, O., Eerola, O., Hellstrom, Å., Uusipaikka, E., & Lang, A. H. ~1997!. Perceptual magnet effect in the light of behavioral and psychophysiological data. Journal of the Acoustical Society of America, 101, 1090 1106. Alho, K. ~1995!. Cerebral generators of mismatch negativity ~MMN! and its magnetic counterpart ~MMNm! elicited by sound changes. Ear and Hearing, 16, 38 51. Alho, K., & Sinervo, N. ~1997!. Preattentive processing of complex sounds in the human brain. Neuroscience Letters, 233, 33 36. Alku, P. ~1992!. Glottal wave analysis with pitch synchronous iterative adaptive inverse filtering. Speech Communications, 11, 109 118. Cheour, M., Ceponiene, R., Lehtokoski, A., Luuk, A., Allik, J., Alho, K., & Näätänen, R. ~1998!. Development of language-specific phoneme representations in the infant brain. Nature Neuroscience, 1, 351 353. Dehaene, S., Dupoux, E., Mehler, J., Cohen, L., Paulesu, E., Perani, D., van de Moortele, P.-F., Lehericy, S., & Le Bihan, D. ~1997!. Anatomical variability in the cortical representation of first and second language. NeuroReport, 8, 3809 3815. Dehaene-Lambertz, G. ~1997!. Electrophysiological correlates of categorical phoneme perception in adults. NeuroReport, 8, 919 924. Fant, G. ~1960!. The acoustic theory of speech production. The Hague: Mouton and Co. Kim, K. H., Relkin, N. R., Lee, K.-M., & Hirsch, J. ~1997!. Distinct cortical areas associated with native and second languages. Nature, 388, 171 174. Kraus, N., McGee, T. J., Carrell, T. D., King, C., Tremblay, K., & Nicol, T. ~1996!. Central auditory system plasticity associated with speech discrimination training. Journal of Cognitive Neuroscience, 7, 25 32. Kuhl, P. K. ~1993!. Early linguistic experience and phonetic perception: Implications for theories of developmental speech perception. Journal of Phonetics, 21, 125 139. Merzenich, M. M., Jenkins, W. M., Johnston, P., Schreiner, C., Miller, S. L., & Tallal, P. ~1996!. Temporal processing deficits of language-learning impaired children ameliorated by training. Science, 271, 77 81. Näätänen, R. ~1992!. Attention and brain function. Hillsdale, NJ: Erlbaum. Näätänen, R., Lehtokoski, A., Lennes, M., Cheour-Luhtanen, M., Huotilainen, M., Iivonen, A., Vainio, M., Alku, P., Ilmoniemi, R. J., Luuk, A., Allik, J., Sinkkonen, J., & Alho, K. ~1997!. Language-specific phoneme representations revealed by electric and magnetic brain responses. Nature, 385, 432 434. Näätänen, R., Schröger, E., Karakas, S., Tervaniemi, M., & Paavilainen, P. ~1993!. Development of a memory trace for a complex sound in the human brain. NeuroReport, 4, 503 506. Näätänen, R., & Tiitinen, H. ~1997!. Auditory information processing as

642 I. Winkler et al. indexed by the mismatch negativity. In M. Sabourin, F. I. M. Craik, & M. Robert ~Eds.!, Advances in psychological science: Biological and cognitive aspects ~pp. 145 170!. Hove, U.K.: Psychology Press. Schröger, E. ~1997!. On the detection of auditory deviants: A pre-attentive activation model. Psychophysiology, 34, 245 257. Tallal, P., Miller, S. L., Bedi, G., Byma, G., Wang, X., Nagarajan, S. S., Schreiner, C., Jenkins, W. M., & Merzenich, M. M. ~1996!. Language comprehension in language-impaired children improved with acoustically modified speech. Science, 271, 81 84. Tiitinen, H., May, P., Reinikainen, K., & Näätänen, R. ~1994!. Attentive novelty detection in humans is governed by pre-attentive sensory memory. Nature, 372, 90 92. Winkler, I., Lehtokoski, A., Alku, P., Vainio, M., Czigler, I., Csépe, V., Aaltonen, O., Raimo, I., Alho, K., Lang, A. H., Iivonen, A., & Näätänen, R. ~1999!. Pre-attentive detection of vowel contrasts utilizes both phonetic and auditory memory representations. Cognitive Brain Research, 7, 357 369. Winkler, I., Reinikainen, K., & Näätänen, R. ~1993!. Event-related brain potentials reflect traces of the echoic memory in humans. Perception & Psychophysics, 53, 443 449. ~Received October 16, 1998; Accepted January 26, 1999!