Attention and emotional face-voice perception

Transcription

1 Attention and emotional face-voice perception (top-down & bottom-up) ATTENTION? Emotional face Emotional voice Coherent & unified percept HOW early do audiovisual emotional signals interact? EEG To what degree do these interactions require attention?

2 Early audiovisual emotional interactions Facial and vocal emotional information interacts early, i.e., within ms after voice onset N1 (~100 ms) smaller amplitude to incongruent than congruent emotional face-voice combinations Pourtois et al. (2000) NeuroReport MMN ( ms) congruent and incongruent emotional face-voice combinations elicit a Mismatch Negativity (MMN) de Gelder et al. (1999) Neurosci Lett GBA ( ms) greater Gamma band activity (GBA) ~40-50 Hz to congruent than incongruent non-emotional stimuli Scheider et al. (2008) Neuroimage, Senkowski et al. (2009) Exp Brain Res All ERP studies employed a passive viewing or listening task

3 Early audiovisual emotional interactions Facial and vocal emotional information interacts early, i.e., within ms after voice onset N1 (~100 ms) smaller amplitude to incongruent than congruent emotional face-voice combinations Pourtois et al. (2000) NeuroReport MMN ( ms) congruent and incongruent emotional face-voice combinations elicit a Mismatch Negativity (MMN) de Gelder et al. (1999) Neurosci Lett Pre-attentive change detection Oddball paradigm Standard Deviant Näätänen et al. (2001) TICS

4 Early audiovisual emotional interactions Facial and vocal emotional information interacts early, i.e., within ms after voice onset N1 (~100 ms) smaller amplitude to incongruent than congruent emotional face-voice combinations Pourtois et al. (2000) NeuroReport MMN ( ms) congruent and incongruent emotional face-voice combinations elicit a Mismatch Negativity (MMN) de Gelder et al. (1999) Neurosci Lett Pre-attentive change detection These studies employed a passive viewing or listening task Oddball paradigm Standard Deviant Näätänen et al. (2001) TICS

5 Effect of task on emotional face-voice perception Ho, Schöger, & Kotz (under review) JCN

6 Effect of task on emotional face-voice perception Grand average N1 topography N1 mean amplitude N1 ( Task performance ms) N1 mean amplitude Attend-synchrony Attend-congruence Potential (µv) Potential (µv) -4 FCz N Voice onset Time (ms) -4 FCz Time (ms) Congruent (C) Incongruent (I) Difference (I-C) Congruent (C) Incongruent (I) Difference (I-C) Congruent (C) Incongruent (I) Smaller amplitude to incongruent stimuli BUT: only when attention was directed to facial and/or vocal expression No N1 modulation when attention was directed to the a/synchrony of audiovisual speech Top-down attention modulates early audiovisual emotional interactions = Congruent (C) Incongruent (I) Congruent Incongruent Difference (I-C) Within-subject 95 % CI Difference (I-C) Ho, Schöger, & Kotz (under review) JCN

7 Three-stimulus oddball paradigm Videos: 1-2 seconds Face neutral neutral neutral angry neutral neutral neutral neutral neutral neutral neutral neutral neutral neutral neutral neutral Voice

8 Three-stimulus oddball paradigm Face Congruent standard Incongruent deviant neutral neutral neutral angry neutral neutral neutral neutral Voice neutral neutral neutral neutral neutral neutral neutral neutral ~80% ~10% 2 Blocks: neutral voice 2 Blocks: angry voice

9 Three-stimulus oddball paradigm Face Congruent standard Incongruent deviant Asynchronous target neutral neutral neutral angry neutral neutral neutral neutral Voice neutral neutral neutral neutral neutral neutral neutral neutral ~80% ~10% ~10% 2 Blocks: neutral voice 2 Blocks: angry voice

10 Three-stimulus oddball paradigm Face Congruent standard Incongruent deviant Asynchronous target neutral neutral neutral angry neutral neutral neutral neutral Voice neutral neutral neutral neutral neutral neutral neutral neutral ~80% ~10% ~10% 2 EEG sessions: Passive (1 st ) session No task Active (2 nd ) session Target detection task

11 Three-stimulus oddball paradigm Face Congruent standard Incongruent deviant Asynchronous target neutral neutral neutral angry neutral neutral neutral neutral Voice neutral neutral neutral neutral neutral neutral neutral neutral ~80% ~10% ~10% 29 Participants 15 female years old High-density EEG 256 electrodes Nose reference Active (2 nd ) session Target detection task

12 MMN: ms Congruent Incongruent Passive Potential (µv) -4 Fz Voice onset Time (ms) -4 Fz 400 Potential (µv) -4 Fz Voice onset Time (ms) -4 Fz 400 No MMN to congruent or incongruent deviants Absence of MMN due to stimulus material? Interference of asynchronous videos with deviance detection mechanism? Active Potential (µv) Voice onset Time (ms) 400 Potential (µv) Voice onset Time (ms) 400 Why can the MMN response to audiovisual emotional stimuli be elicited only in certain context? Standard Deviant Difference (I-C)

13 Early gamma band activity (GBA)

14 Early gamma band activity (GBA) Explicit cross-modal matching paradigm Enhanced lower GBA (30-40 Hz) within ms after auditory onset Reduced ERP amplitude within ms after auditory onset Implicit cross-modal matching paradigm Schneider et al. (2008) Neuroimage Enhanced lower GBA (40-50 Hz) within ms after auditory onset But no early ERP effects Senkowski et al. (2009) Exp Brain Res

15 Early gamma band activity (GBA) Congruent deviant Incongruent deviant 45 Hz 65 Hz Active Cz Cz Congruent deviant Incongruent deviant 100 ms 150 ms Congruent deviants show enhanced GBA starting from ~50 ms post voice onset Incongruent deviants show less activity in the same frequency band and time window

16 Early gamma band activity (GBA) Congruent deviant Cz Incongruent deviant 45 Hz 65 Hz Cz Active Congruent deviant Incongruent deviant 100 ms Cz Congruent deviant Passive Cz 150 ms Incongruent deviant 100 ms 150 ms

17 Early gamma band activity (GBA) Congruent deviant Incongruent deviant Hz Hz Cz Cz * * ms (anterior) ms Active Passive Cz Cz * ms * (anterior) ms Congruent Incongruent Within-subject 95 % CI

18 Summary and conclusion MMN Neither congruent nor incongruent deviants elicited a MMN GBA Enhanced activity to congruent deviants within ms Greater increase in active than passive condition Congruent and incongruent standards show no such difference Is early audiovisual emotional interaction pre-attentive? ERP provided no evidence that early audiovisual emotional interaction is pre-attentive Time-frequency results suggest that early audiovisual emotional interactions require some degree of attention

19 Acknowledgements Methods Maren Grigutsch Francois Tadel Technical assistance Johnson Chen Donovan Govan Data collection Leo Cheng Peta Mills Tim Paris

20 Early audiovisual emotional interactions Facial and vocal emotional information interacts early, i.e., within ms after voice onset N1 (~100 ms) smaller amplitude to incongruent than congruent emotional face-voice combinations Pourtois et al. (2000) NeuroReport MMN ( ms) congruent and incongruent emotional face-voice combinations elicit a Mismatch Negativity (MMN) de Gelder et al. (1999) Neurosci Lett GBA ( ms) greater Gamma band activity (GBA) ~40-50 Hz to congruent than incongruent non-emotional stimuli Scheider et al. (2008) Neuroimage, Senkowski et al. (2009) Exp Brain Res => irrespective of whether early ERP effects are present

21 Early audiovisual emotional interactions Facial and vocal emotional information interacts early, i.e., within ms after voice onset N1 (~100 ms) smaller amplitude to incongruent than congruent emotional face-voice combinations Pourtois et al. (2000) NeuroReport MMN ( ms) congruent and incongruent emotional face-voice combinations elicit a Mismatch Negativity (MMN) de Gelder et al. (1999) Neurosci Lett GBA ( ms) greater Gamma band activity (GBA) ~40-50 Hz to congruent than incongruent non-emotional stimuli Scheider et al. (2008) Neuroimage, Senkowski et al. (2009) Exp Brain Res => irrespective of whether early ERP effects are present All ERP studies employed a passive viewing or listening task

22 N1: ms Deviant Standard N1 Fz Difference (I-C) Mean amplitude Fz Active Deviant Standard Voice onset Congruent (C) Voice onset Incongruent (I) Difference (I-C) Significant increase in N1 amplitude to incongruent deviant stimuli Congruent and incongruent standards showed no difference Enhanced N1 likely reflects increased attention to incongruent deviants Congruent Incongruent

23 N1: ms Deviant Standard N1 Fz Difference (I-C) Mean amplitude Fz Active Deviant Standard Voice onset Voice onset Fz Fz Passive Deviant Standard Voice onset Congruent (C) Voice onset Incongruent (I) Difference (I-C) Congruent Incongruent Within-subject 95 % CI

24 Concluding remarks Role of attention in emotional face-voice integration No evidence that early audiovisual emotional interaction is pre-attentive Early modulation by emotional face-voice congruence constrained to deviant stimuli => Deviant stimuli typically capture involuntary attention => Early audiovisual emotional interactions require some degree of attention Voluntary and involuntary attention influence multisensory perception differently What is the function underlying early GBA at ~45 Hz? Matching bottom-up information with top-down information Herrmann et al. (2004) TICS Could this GBA enhancement be related to N1 modulation? => N1 suppression is only elicited by predictable stimuli Vroomen & Stekelenburg (2010) JCN => Both N1 modulation and early GBA enhancement require some degree of attention

25 Concluding remarks Role of attention in emotional face-voice integration No evidence that early audiovisual emotional interaction is pre-attentive Early modulation by emotional face-voice congruence constrained to deviant stimuli => Deviant stimuli typically capture involuntary attention => Early audiovisual emotional interactions require some degree of attention Voluntary and involuntary attention influence multisensory perception differently What is the function underlying early GBA at ~45 Hz? Matching bottom-up information with top-down information Herrmann et al. (2004) TICS Could this GBA enhancement be related to N1 modulation? => N1 suppression is only elicited by predictable stimuli Vroomen & Stekelenburg (2010) JCN => Both N1 modulation and early GBA enhancement require some degree of attention