1 Experience-grounded conceptualsemantic representations and their modulation by linguistic context Marco Tettamanti San Raffaele Scientific Institute, Milan, Italy Firenze, MODELACT 2015, June 11
2 CONCEPTUAL KNOWLEDGE Represented by amodal and arbitrary symbols Complete independence Independent but associated Partial dependence Complete dependence Binder JR & Desai RH (2011) Grounded in content-specific distributed representations
3 PARTIALLY GROUNDED The reactivation of the neural nodes contributing to distributed semantic networks varies in a flexible manner, depending on the type of concept retrieval required by the task, by the focus on specific sensorymotor features, and by the linguistic context in which it occurs (Hoenig et al., 2008; Ghio and Tettamanti, 2010; van Dam et al., 2012; Kiefer & Pulvermueller, 2012).
4 Action Motion Color Shape Sound Patterson K. et al. Nat. Rev. Neurosci. 8: (2007).
5 Action-related concepts Abstract concepts?
6 I grasp the apple LdPCG LSMG LIFG-PT LpITG I do not grasp the apple
7 Concepts in semantic memory Concepts in semantic memory
8 Concepts in semantic memory Dominant channel of experience for a given object. PERCEPTUAL Information about perceptual features Memory of visual, auditory, olfactory, gustatory properties of objects is stored in the same sensory systems activated when experiencing that information.
9 Concepts in semantic memory FUNCTIONAL ACTION-RELATED Abstract properties about object function and context of use Visuo-motor information about the use and manipulation of objects WHAT FOR? HOW? The use of an object and the way it is manipulated are reflected in the neuropsychological distinction between semantic dementia and apraxia.
10 Concepts in semantic memory FUNCTION > ACTION Areas associated with semantic dementia and contextual features of objects ACTION > FUNCTION Parieto-premotor system for action control Canessa N, Borgo F, Cappa SF, Perani D, Falini A, Buccino G, Tettamanti M, Shallice T. The Different Neural Correlates of Action and Functional Knowledge in Semantic Memory: An fmri Study. Cerebral Cortex 18(4): (2008).
11 Concepts in semantic memory Dominant channel of experience for a given object. Direct evidence?
12 Concepts in semantic memory INVENTED OBJECTS/TOOLS Increased fronto-parieto-temporal activations for trained vs untrained tools Early frontal motor and late occipito-parietal effects for pantomimed actions vs. pointing T=Trained NT=Not Trained Weisberg J. et al. Cereb. Cortex 17: (2007). Kiefer M. et al. J. Cogn. Neurosci. 19: (2007).
13 Concepts in semantic memory MANIPULATION vs VISUAL EXPERIENCE Manipulation training objects Visual training objects No-Training objects Pictures of objects Scrambled object images
14 Concepts in semantic memory Task Post > Pre Training Post Training specific effects V N M Bellebaum C, Tettamanti M, Marchetta E, Della Rosa P, Rizzo G, Daum I, Cappa SF. Neural representations of unfamiliar objects are modulated by sensorimotor experience. Cortex 49(4): (2013).
15 Concepts in semantic memory OBSERVATION vs VISUAL EXPERIENCE Rüther NN, Tettamanti M, Cappa SF, Bellebaum C. Observed Manipulation Enhances Left FrontoParietal Activations in the Processing of Unfamiliar Tools. PLoS ONE 9(6): e99401 (2014).
16 Concepts in semantic memory DCM FUNCTIONAL INTEGRATION Winning model LIFG LIFG LIPL LMTG LPPC LPPC LIPL LMTG LIPL LIFG 2nd Best model LMTG LPPC M V Bellebaum C, Tettamanti M, Marchetta E, Della Rosa P, Rizzo G, Daum I, Cappa SF. Neural representations of unfamiliar objects are modulated by sensorimotor experience. Cortex 49(4): (2013).
17 Semantic knowledge and language Semantic knowledge and language
18 Semantic knowledge and language ACTION-RELATED SENTENCES Task: passively listening to auditorily presented sentences 4s AFFERRO UN ADDENTO UNA MELA COLTELLO LECCO IL GELATO SPEZZO IL PANE... Mouth 20 s CALCIO IL PALLONE... Hand Abstract RICORDO IL FATTO VALUTO GLI ERRORI... INFILO GLI ZOCCOLI Leg Abstract MASTICO IL PANE... SOFFIO SUL FUOCO Mouth Abstract TEMO LA PROVA CONFONDO I RICORDI IMMAGINO LA COLLERA Abstract DESIDERO LA GIOIA... ADORO L OZIO APPREZZO LE DOTI...
19 0 L-A SFS IFS H-A Signal change (%) The left IFC (BA 44/45) was activated irrespective of body part 0.1 M-A H-A L-A SF x=-44 y=2 z=24 * IPS ITS M-A H-A * 0.2 CS STS 0.1 x=-28 y=-4 z=48 PoCS 0.2 M-A L-A Signal change (%) 0.3 x=-26 y=4 z=64 * PrCS Signal change (%) Semantic knowledge and language M-A H-A L-A Body-part specific activations in premotor area (BA 6) Tettamanti M, Buccino G, Saccuman MC, Gallese V, Danna M, Scifo P, Fazio F, Rizzolatti G, Cappa SF, Perani D. Listening to action-related sentences activates fronto-parietal motor circuits. J. Cogn. Neurosci. 17(2): (2005).
20 Semantic knowledge and language CONCRETE vs ABSTRACT Raichle ME, TICS 14(4): (2010). LdPCG P=0.99 P=1.00 LSMG P=1.00 P=1.00 P=0.95 P=1.00 LRpCC P=1.00 LIFG-PT LSTG LSTG LIFG-PB LpITG LpMTG P=1.00 P=1.00 P=0.97 LaITG Action > Abstract Abstract > Action P=1.00 Ghio M. & Tettamanti M. Semantic domain-specific functional integration for action-related vs. abstract concepts. Brain and Language 112 (3): (2010).
21 Semantic knowledge and language ABSTRACT & EMOTIONS 1446 English words Abstract words are less imageable, less familiar, acquired later, but they are more emotional than concrete words.
22 Semantic knowledge and language 60 abstract words 60 concrete words A& C A> C Vigliocco G, Kousta ST, Della Rosa PAD, Vinson DP, Tettamanti M, Devlin JT, Cappa SF. The neural representation of abstract words: the role of emotion. Cereb. Cortex 24(7): (2014).
23 Semantic knowledge and language A DEMOLITION TEST FOR GROUNDED THEORY... Fine-grained semantic categorization: 3 abstract and 3 concrete semantic domains. Robust stimulus set: at least 35 sentences per semantic domain. Sentence stimuli, to mitigate lexical-semantic ambiguities. Third-person singular form, to emphasize narrative linguistic comprehension and test stronger grounded cognition accounts. No passive listening but rather non-interfering overt motor response to keep control over subjects' performance. Semantic-domain specific functional localizer scans.
24 Semantic knowledge and language Abstract domain Concrete domain 35 Mental states-related sentences (Ms) e.g., Lei ricorda il passato (She remembers the past) 35 Mouth-related sentences (Mo) e.g., Lei schiocca la lingua (She clicks the tongue) 35 Emotion-related sentences (Em) e.g., Lei prova il disgusto (She feels disgust) 35 Hand-related sentences (Ha) e.g., Lei ricama il fazzoletto (She embroiders the handkerchief) 35 Mathematics-related sentences (Ma) e.g., Lei calcola la somma (She determines the sum) 35 Leg-related sentences (Le) e.g., Lei calcia il pallone (She kicks the ball) All sentences consisted of four words and had the same structure: 3rd person feminine singular + verb simple present tense + object complement Matched psycholinguistic variables Ms Em Ma Mo Ha Le N. of syllables χ 2 = N. of letters p = Noun frequency p = Verb frequency p = Nouns and verbs frequency p = 0.110
25 Semantic knowledge and language PSYCHOLINGUISTIC RATINGS Correspondence analysis of Concreteness ratings Ghio M, Vaghi MMS, Tettamanti M. Fine-grained semantic categorization across the abstract and concrete domains. PLoS ONE 8(6): e67090 (2013).
26 Semantic knowledge and language 1-BACK CROSS-MODAL TASK WITH OVERT RESPONSE Lei rievoca il ricordo Lei rievoca il ricordo 500 ms 500 ms + + Lei calcia il pallone Lei calcia il pallone 500 ms 500 ms + + Lei apre la porta 500 ms Lei suona il campanello 500 ms?? 2000 ms 2000 ms Lei apre la porta Lei sente il campanello CLOSE ONCE CLOSE TWICE Lei calma la rabbia 500 ms + Lei calma la rabbia 500 ms +
27 Semantic knowledge and language SEMANTIC-DOMAIN SPECIFIC FUNCTIONAL LOCALIZERS Mental-state localizer Emotion localizer REST * 2 =? REST REST 24 s 170 s Mouth localizer REST Mathematics localizer Move your tongue Hand localizer REST Leg localizer Move your right hand REST Move your right foot
28 Semantic knowledge and language Mental-state localizer Emotion localizer Mathematics localizer Action localizers FWE 0.05
29 Semantic knowledge and language Main effect of Concreteness n = 36 Abstract > Concrete Concrete > Abstract FWE 0.05 FWE 0.05 No predicted semantic domain effects. Ms Em Ma FWE 0.05 Mo L amygdala (SF) x = -20, y = -4, z = -16 * * * S.V.C. FWE 0.05 Ha Le
30 Semantic knowledge and language MULTIVARIATE PATTERN ANALYSIS Main effect of Concreteness Whole brain SVM classification Mean accuracy: 94.4 % (against chance level: 50 %) Sensitivity map Chi-square: 56.89, P = 2.7 e-12 Monte-Carlo permutation test: P =
31 Semantic knowledge and language 6 semantic domains Whole brain joint meta linear SVM classification Mean accuracy: 41.7 % (against chance level: 16.6 %) Chi-square: , P = 2.7 e-19 Bayesian class separation: [Ms] [Em] [Ma] [Mo] [Ha] [Le] Category-specific sensitivity conjunction: conjunction of the 5 sensitivity maps between each category and each of the other 5 categories. Ghio M, Vaghi MMS, Perani D, Tettamanti M. Submitted.
32 Semantic knowledge and language EXPERIENCE & LANGUAGE SCREWDRIVER Direct evidence?
33 Semantic knowledge and language EXPERIENCE ENHANCES COMPREHENSION Beilock S.L. et al. PNAS 105(36): (2008).
34 Semantic knowledge and language TRAINING ACTION = TRAINING CONCEPT? Piegare ( to fold ) Manipolare ( to handle ) Agganciare ( to hook ) Allacciare ( to fasten ) Infilare ( to thread ) Annodare ( to tie ) Digitare ( to key in ) Ricamare ( to embroider ) Rammendare ( to darn ) Cucire ( to sue )
35 Semantic knowledge and language PRE-TRAINING Minnesota Manual Dexterity Test ms The individual threads + The individual scratches match nonmatch The individual draws match nonmatch TARGET match nonmatch NONTARGET RT RT RT TRAINING 30 min x 5 days x 3 weeks POST-TRAINING Minnesota Manual Dexterity Test + + The individual threads + The individual scratches match nonmatch The individual draws match nonmatch TARGET match nonmatch NONTARGET RT RT RT
36 Semantic knowledge and language Means (n = 19) PreTraining Score PostTraining Score Paired t-student Minnesota MDT 65 7 ± ± 7 4 P = Origami 24 9 ± ± 2 4 P < Coin 0.7 ± ± 2.5 P < Knots 35 6 ± ± 4 0 P = Tastiera 10.1 ± ± 0.5 P = Ricamo 7.2 ± ± 2.1 P < Main effect of Verbs: P = Main effect of Training: P = Verbs x Training: P = TargetPost - NonTargetPost: P = Locatelli M, Gatti R, Tettamanti M. Training of manual actions improves language understanding of semantically related action sentences. Front. Cogn. Sci. 3(547): 1-10 (2012).
37 Modulatory effects of negation Modulatory effects of negation
38 Modulatory effects of negation SENTENTIAL NEGATION Priming effects are reduced for negated vs. non-negated nouns, suggesting that negation affects the semantic activation level of the negated propositions (MacDonald & Just, 1989). I grasp the apple Abstract sentences as a control I do not grasp the apple
39 Modulatory effects of negation CONCRETENESS X POLARITY
41 Modulatory effects of negation CONCRETENESS X POLARITY I grasp the apple I (do not) remember the past LdPCG LSMG LIFG-PT LpITG I do not grasp the apple Tettamanti M, Manenti R, Della Rosa PA, Falini A, Perani D, Cappa SF, Moro A. Negation in the brain: modulating action representations. NeuroImage 43(2): (2008).
42 Modulatory effects of negation INDEPENDENT VERIFICATIONS GRASP! vs DON'T GRASP! Tomasino B. et al., Neuroscience 169: (2010). Liuzza M.T. et al., PloS ONE 6(2): e16855 (2011).
43 Modulatory effects of negation MOTOR CORRESPONDENCE THEORY I grasp the apple I do not grasp the apple I grasp Reaction times I grasp I do not grasp
44 Modulatory effects of negation KINEMATIC ANALYSIS affirmative negative abstract proximal distal I wish I grasp I pinch I do not wish I do not grasp I do not pinch Exp 1 Experiment 1: (reach & grasp) n=18 Experiment 2: (grasp) n=24 Exp 2
45 Modulatory effects of negation Experiment 2 Experiment 1 * P = n.s. n.s. n.s. P = Concreteness x Polarity P = P = Concreteness x Polarity P = Bartoli E, Tettamanti A, Farronato P, Caporizzo A, Gatti R, Moro A, Perani D, Tettamanti M. Sentential negation improves concurrent motor performance. J. Neurophysiol. 109(7): (2013).
46 To sum up
47 Semantic memory for an object is encoded through the dominant channel of experience with concurrent inhibition of the non-dominant grounded systems. When processing domain-specific linguistic utterances, depending on the depth of lexical-semantic access and on the level of semantic experience, the corresponding grounded systems may be functionally integrated with perisylvian language regions. In the case of negative sentences, the interactions between perisylvian language regions and grounded systems are reduced, leaving neural resources more free for concurrent motor performance.
6 Defining the relation between linguistics and neuroscience David Poeppel and David Embick University of Maryland College Park and University of Pennsylvania The popularity of the study of language and
Human volition: towards a neuroscience of will Patrick Haggard Abstract The capacity for voluntary action is seen as essential to human nature. Yet neuroscience and behaviourist psychology have traditionally
Review The what, when, where, and how of visual word recognition Manuel Carreiras 1,2, Blair C. Armstrong 1, Manuel Perea 1,3, and Ram Frost 1,4,5 1 Basque Center for Cognition, Brain, and Language, San
Embodied Construction Grammar Benjamin Bergen Nancy Chang 1. Introduction 1.1 A cognitive science approach to grammar Theories of grammar are defined by the questions they ask. In the analytical linguistic
Review TREND in Cognitive ciences Vol.8 No.5 May 2004 ocial cognitive neuroscience: where are we heading? arah-jayne Blakemore 1, Joel Winston 2 and Uta Frith 1 1 Institute of Cognitive Neuroscience, 17
Open Source Visualization Tools to Enhance Reading Comprehension and Concept Attainment Carol A. Brown, EdD Associate Professor, East Carolina University Department of Mathematics, Science, and Instructional
Bilingualism: Language and Cognition 10 (2), 2007, 131 146 C 2007 Cambridge University Press doi:10.1017/s1366728907002891 131 Reading in two writing systems: Accommodation and assimilation of the brain
Review 119 Overlapping mechanisms of attention and spatial working memory Edward Awh and John Jonides Spatial selective attention and spatial working memory have largely been studied in isolation. Studies
Review TRENDS in Cognitive Sciences Vol.7 No.7 July 2003 313 Human recognition memory: a cognitive neuroscience perspective Michael D. Rugg 1 and Andrew P. Yonelinas 2 1 Institute of Cognitive Neuroscience
COGNITION AND EMOTION, 2002, 16 (3), 331 354 Induced processing biases have causal effects on anxiety Andrew Mathews MRC Cognition and Brain Sciences Unit, Cambridge, UK Colin MacLeod University of Western
THEORETICAL CONTRIBUTION What Do Connectionism and Social Psychology Offer Each Other? Eliot R. Smith Purdue University Social psychologists can benefit from exploring connectionist or parallel distributed
The Essential Role of Broca's Area in Imitation Marc Heiser 1,7, Marco Iacoboni 1,2,6,*, Fumiko Maeda 1,3, Jake Marcus 1, John C. Mazziotta 1,3,4,5,6 1 Ahmanson-Lovelace Brain Mapping Center, Neuropsychiatric
Emotion Copyright 2003 by the American Psychological Association, Inc. 2003, Vol. 3, No. 4, 378 393 1528-3542/03/$12.00 DOI: 10.1037/1528-35188.8.131.528 Effect of Negative Emotional Content on Working Memory
More than one language in the brain. Itziar Laka * University of the Basque Country (UPV/EHU) 0. Introduction What difference does it make to have one language in the brain or to have more than one? This
Jommtl of Experimental Paychology: General Copyright 2000 by the American Psychological Association, Inc. 2000, Vol. 129, No. 2, 242-261 0096-3445/00/$5.00 DOI: 10.1037//0096-34184.108.40.206 Are Affective
Journal of Child Psychology and Psychiatry 47:3 (2006), pp 296 312 doi:10.1111/j.1469-7610.2006.01611.x Development of the adolescent brain: implications for executive function and social cognition Sarah-Jayne
CORRECTED JUNE 7, 2010; SEE LAST PAGE Psychological Review 2010 American Psychological Association 2010, Vol. 117, No. 2, 440 463 0033-295X/10/$12.00 DOI: 10.1037/a0018963 Construal-Level Theory of Psychological
Brain and Cognition 53 (2003) 433 440 www.elsevier.com/locate/b&c Category-specific visual agnosia: Lesion to semantic memory versus extra-lesional variables in a case study and a connectionist model E.
Coding and use of tactile signals from the fingertips in object manipulation tasks Roland S. Johansson* and J. Randall Flanagan Abstract During object manipulation tasks, the brain selects and implements
Information and Cognitive Process: communication research for design Peter K. Storkerson M.F.A. Ph.D. Southern Illinois University www.communicationcognition.com email: firstname.lastname@example.org telephone-fax: 716
articles Differential responses in the fusiform region to same-race and other-race faces Alexandra J. Golby, John D. E. Gabrieli, Joan Y. Chiao and Jennifer L. Eberhardt Departments of Radiology and Psychology,
Flexible Shortcuts: Linguistic Distributional Information Affects both Shallow and Deep Conceptual Processing Louise Connell (email@example.com) School of Psychological Sciences, University
J Neurol Neurosurg Psychiatry 2000;69:479 486 479 Interference between gait and cognitive tasks in a rehabilitating neurological population Patrick Haggard, Janet Cockburn, Josephine Cock, Claire Fordham,
www.elsevier.com/locate/ynimg NeuroImage 34 (2007) 801 814 Investigating the functional interaction between semantic and episodic memory: Convergent behavioral and electrophysiological evidence for the