3BatS. The aim. The sign. 1: In the word 1/13/10. On the perception of allegedly word-like units in signed language

Transcription

1 VIPP Symposium 2010: Seeing Red Aalto University/School of Art and Design, 14 January BatS On the perception of allegedly word-like units in signed language Tommi Jantunen University of Jyväskylä, Department of Languages/3BatS Postdoctoral Researcher s project Funded by the Academy of Finland and Department of Languages, JyU Aim is to investigate critically the ontological basis of modern sign language (SL) research by testing certain well-established presuppositions governing the research into sign, syllable and sentence in (Finnish) SL The sign The aim...is assumed to be a unit that corresponds to spoken language word; for example: Throughout this account [---] the term sign will be used as a rough equivalent to word in spoken language (Johnston & Schembri 1999: 115/fn. 1) [---] the word and the sign are situated at an equivalent level of linguistic organisation [---] (Zeshan 2002: 154) [---] as far as I can see, signs would be partly equivalent to semantic units in spoken language (Blanche-Benveniste 2007: 86) of this presentation is to investigate the correspondence between the sign and the word from the perspective of two foundational word-defining characteristics and perceptual experiments 1: In the word ja sitte meaning always associates with the form ja [ja] [sitte] 1

2 2: The (form of the) word ja sitte includes the maximally perceivable part(s) of the signal The prototypical The method of identifying signs Word Has a conventionalised meaning (semantics) Contains two syllables/is bimoraic (phonology) Has a fixed order of meaningful elements, can form a sentence by itself, can change place in a sentence etc. (grammar) Sign Has a conventionalised meaning (semantics) Is a monosyllable, includes only one set of selected fingers, is produced at one major place of articulation (phonology) Can form a sentence by itself etc. (grammar) Look for meaningful sequences (semantics) Observe changes in the posture and/or movement of the hands and nonmanual articulators (phonology) Use any grammatical knowledge you have access to (grammar) (e.g. McCarthy & Prince 1993, Karlsson 1994) (e.g. Sandler 1999, Zeshan 2002) E.g., in the Corpus NGT ELAN annotation A sign starts: at the first frame in which the hand starts to move away from the initial location of the sign to the final location of the sign; or (in case the hand does not move through space): at the first frame in which the handshape starts to change, e.g. closing the hand in the sign for MAN ; or (in case the hand does not move through space and the handshape does not change): at the first frame in which the orientation of the hand starts to change, e.g. turning the hand in the sign for OVERLEDEN ( PASS_AWAY ); A sign ends: at the first frame in which the handshape starts to change after the sign was finished; or at the first frame in which the hand starts to move away from the final location of the sign. (Crasborn & Zwitserlood 2008: 6) 2

3 What does she sign? Demo 1 Comprehension & perception test (adapted from Suvi) What does he sign? Explanation & the result Superficially signing-like video clips, edited, however, to contain only the parts between signs (i.e. transitions); note that in the first clip, the durations of sign sequences are relatively short whereas the durations of transitions are relatively long in the second clip, the durations of sign sequences are relatively long whereas the durations of transitions are relatively short The artificial deletion of signs from the video does not crucially impair comprehension of the intended message (adapted from Suvi) Demo 1: interim discussion & conclusion The demo suggests that, in the sign, meaning is not crucially associated with the (alleged) form; the meaning can be deciphered, up to a certain point, on the basis of transitions (cf. information concerning Aktionsart, other semantic nyances, and near-homonym disambiguation) The finding seems to go against the conclusion made, for example, by Grosjean (1981) who in his perception study of isolated signs claimed that the "recognition point" is always inside the sign (cf., however, Grosjean's sign recognition technique) The main conclusion made by Clark & Grosjean (1982) is implicitly supported; they found that in contextual signing, only 1/3 of a sign is needed to recognise it (cf. 80% of the word in speech) The existence of a strong modality effect is implied: in SL (unlike in spoken language), the articulators are completely visible during the production of the signal; this allows the meaning to be continuously "guessed" and disambiguated In the word, meaning is always associated with the form and cannot be deciphered on the basis of transitions (i.e. the sequences of silence) Ergo: signs and words cannot be compared to each other in this respect Demo 2 Motion analysis & perception 3

4 1/13/10 Background A group of people from the Aalto University's School of Science and Technology/Dept. of Information and Computer Science (Markus Koskela & Jorma Laaksonen), the Finnish Association of the Deaf (Päivi Rainò), and Jyväskylä University/Dept. of Languages (Tommi Jantunen) A semi-automatic analysis of various characteristics of articulators' movements on digital SL video The basis of the method is quantitative computer vision analysis The method consists of three distinct phases: detection of the target articulators on the video, tracking of the motion of the selected articulators, and the representation of the motion of the articulators with statistical descriptors Issues that still need addressing: movements away from or towards the body, finer detection of articulators Filter (1) and mtip (2) (Jantunen, Koskela, Laaksonen & Rainò, submitted) (excerpts from Jantunen, Koskela, Laaksonen & Rainò, submitted) Motion descriptors (3) The importance of acceleration [A] visual beat is communicated by periods of acceleration [ ] (Luck & Sloboda 2008: 237) Visual beats = those events that are felt to be more forcefully produced and around which the other events in the sequence are organized (Allen & al. 1991: 197) "[A]cceleration is also important for the parsing of one event from another" (Shaw & Cutting 1980: 77) Current operationalisation: the peak of acceleration = the most perceivable moment (analysis of Suvi's article 1038, example 3) Signs & acceleration peaks (1) (analysis of Suvi's article 1038, example 3) Signs & acceleration peaks (2) (analysis of the first 512 frames of Pessi ja Illusia) 4

5 Demo 2: interim discussion & conclusion Given that the notion of perceptivity is validly operationalised and the forms of signs are correctly determined, the distribution of acceleration peaks suggests that the most perceivable moments in a signed signal tend to associate with transitions, not with signs Similar findings have been presented, from the perspective of sign production, for example, by Wilbur (1990) and Wilcox (1992) (concerning stressed sign production and fingerspelling, respectively) The (form of the) word always includes the maximally perceivable moment(s) of the signal Ergo: signs and words are not equivalent units in this respect General discussion The results indicate a conflict between that what the sign is argued to be in and by the linguistic theory (i.e. a unit equivalent to word) and that what the sign is according to the perceptual experiments (i.e. a unit not equivalent to word) Note that crucial differences between the sign and the word have also been indicated by the more traditonal grammatical research; however, this has not led to the questioning of sign=word assumption Spoken language bias in sign linguistics (cf. Per Linell's WLB) Cont'd Still cont'd & conclusion Perhaps only the applied method of identifying and indicating signs from the video is misguided and in the need of revision? Indeed, the method contains a certain degree of vagueness However, the method does not exist in the vacuum; it is ontologically grounded Therefore, also the current ontological conception of the sign is suggested to be misguided and in the need of revision In general, it is implied that signs and their borders are ontologically more gradient and fuzzier than words and their borders (cf. modality effect); note that the notions of gradience and fuzziness have already gained ground in other areas of SL research More work on signs, transitions and their borders is needed. For example, it is generally assumed that transitions in SL are non-linguistic in the sense similar to word-to-word transitions in spoken language. However, the present data implies that SL transitions have a linguistic role. References (1) References (2) Allen, G. D.; Wilbur, R. B. & Schick, B. B. (1991). Aspects of rhythm in ASL. Sign Language Studies 72, Blanche-Benveniste, C. (2007). Linguistic units in signed and verbal languages. In E. Pizzuto, P. Pietrandrea & R. Simone (Eds.), Verbal and signed languages. Comparing structures, constructs and methodologies, Mouton de Gruyter: Berlin Crasborn, O. & Zwitserlood, I. (2008). Annotation of the video data in the "Corpus NGT". Department of Linguistics & Centre for Language Studies, Radboud University Nijmegen. Online publication: Grosjean, F. (1981). Sign & word recognition: a first comparison. Sign Language Studies 32, Clark, L. E. & Grosjean, F. (1982). Sign recognition processes in American Sign Language: the effect of context. Language and Speech 25, Jantunen, T.; Koskela, M.; Laaksonen, J. & Raino, P. (submitted). Towards automated visualization and analysis of signed language prosody: method and linguistic issues. Manuscript, 10 September Johnston, T. & Schembri, A. (1999). On defining lexeme in a signed language. Sign Language & Linguistics 2, Karlsson, F. (1994). Yleinen kielitiede. Yliopistopaino: Helsinki. Luck & Sloboda (2008). Exploring the spatio-temporal properties of simple conducting gestures using a synchronization task. Music Perception 25, McCarthy, J. & Prince, A Prosodic morphology II: constraint interaction and satisfaction. Manuscript. University of Massachusetts, Amherst; Rutgers University, New Brunswick, N.J. Sandler, W. (1999). Cliticization and prosodic words in a sign language. In T. Hall & U. Kleinhenz (Eds.), Studies on the Phonological Word, Johns Benjamins: Amsterdam. Shaw, R. E. & Cutting, J. E. (1980). Clues from an ecological theory of event perception. In U. Bellugi & M. Studdert-Kennedy (Eds.), Signed and spoken languages: biological constraints on linguistic form, Weinheim: Verlag Chemie. Suvi = Suvi The online dictionary of Finnish Sign Language. [Helsinki:] The Finnish Association of the Deaf, Online publication: Wilbur, R. B. (1990). An experimental investigation of stressed sign production. International Journal of Sign Language 1, Wilcox, S. (1992). The phonetics of fingerspelling. Amsterdam: John Benjamins. Zeshan, U. (2002). Towards a notion of 'word' in sign languages. In R. M. W. Dixon & A. Y. Aikhenvald (Eds.), Word. A cross-linguistic typology, Cambridge University Press: Cambridge. 5