Sign Language Avatars Animation and Comprehensibility

Transcription

1 Sign Language Avatars Animation and Comprehensibility Michael Kipp* Alexis Heloir Quan Nguyen DFKI Embodied Agents Research Group Exzellenzcluster Multimodal Computing and Interaction Universität des Saarlandes * University of Applied Sciences Augsburg research funded by:

2 Sign language avatars... for the internet

3 Deaf Sign language is primary means of communication Sign language is a real language [Stokoe 1960] Specific SL for every country (ASL, DGS, LSF, BSL...) 500,000

4 Deaf = what is your name? Sign language is primary means of communication Sign language is a real language [Stokoe 1960] Specific SL for every country (ASL, DGS, LSF, BSL...) Sign language as a first language Spoken language is a foreign language 80% of deaf pupils leave school with significant reading/ writing problems 500,000

5 inexpensive editable possibly interactive comprehension limited commercial expensive not editable not interactive comprehensible 95%

6 Prior Work No standard writing system for sign language Glosses: based on meaning, tool for learning Notation: based on form, tool for science (Stokoe notation, HamNoSys) Milestones ViSiCAST ( ): face-to-face translation // mocap esign ( ): internet // procedural animation SiGML 60% comprehensibility Recent projects More flexible notations: Zebedee (LIMSI), PDTS-SiGML (U East Anglia) Avatars for American SL (Huenerfauth et al. // DePaul Univ.), Italian SL (ATLAS project), Czech SL (U West Bohemia)... YOUR NAME WHAT

7 What's your Agent's Native Language? Greta, U Paris 8 (2006) LIMSI, SNCF, web sourds (2006) SmartBody ICT (2004) Paula GeSSyCa... GUIDO, esign, Televirtual (2003) EMBR, DFKI (2009) Max Elckerlyc Marc...

8 What's your Agent's Native Language? - rich form vocabulary - validation by "understanding" Greta, U Paris 8 (2006) - speech-gesture sync. - lip syncing - locomotion LIMSI, SNCF, web sourds (2006) Universal Communicators SmartBody ICT (2004) Paula GeSSyCa... GUIDO, esign, Televirtual (2003) - control language - validating quality EMBR, DFKI (2009) Max Elckerlyc Marc...

9 Point of Departure Goal: Make every ECA "sign language ready" EMBR: EMBodied agent Realizer open source own animation language EMBRScript Comprehensiblity?

10 Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...)

11 Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises

12 Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises Facial expression: higher amplitude Mouth: sophisticated viseme set, should allow lipreading, use text-to-speech for visemes

13 Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises Facial expression: higher amplitude Mouth: sophisticated viseme set, should allow lipreading, use text-to-speech for visemes Gaze: separate eye-ball from head movement

14 Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises Facial expression: higher amplitude Mouth: sophisticated viseme set, should allow lipreading, use text-to-speech for visemes Gaze: separate eye-ball from head movement

15 Animating Sign Language: Attempt I Video: human signer's utterance Imitate utterance using EMBRScript Show EMBR animation

16 Failed!

17 Reasons Ambiguity in sign language fewer grammatical constructs Single sign level formational manual features situational nonmanual features (almost impossible) mouthing especially important in German SL Utterance level facial expression for sentence mode eyebrows + posture for information structure face as a visual focus point Casual signing style makes sign harder to read human signers compensate with all of the above

18 Consequences Working hypothesis: Avatars with current animation methods are unable to produce understandable "spontaneous" sign language Therefore: Overarticulate Involve Deaf experts Focus on nonmanual features Consider random facial movement

19 Original Overarticulated Remake Avatar

20 Attempt II Overarticulated remake transcribe glosses recording Gloss-based animation (lexicalized) compatible with EMBRScript tool support implications for HamNoSys

21 Video HamNoSys Animation

22 Video HamNoSys BML Animation

23 Video HamNoSys BML Animation Heloir, Kipp 2010 Kipp et al Video BML EMBRScript Animation

24 Video HamNoSys BML Animation Heloir, Kipp 2010 Kipp et al Video HamNoSys BML EMBRScript Animation

25 Video HamNoSys BML Animation Heloir, Kipp 2010 Kipp et al Video EMBRScript Animation

26 Sample utterance: YOUR NAME WHAT gloss = sequence many sequences utterance pose seq. pose seq. pose seq. single pose pose sequence pose pose pose pose YOUR NAME WHAT

27 Evaluation Corpus 11 utterances (154 glosses) from German Deaf e-learning portal quite complex sentences Animation higher duration for remake (factor 1.8) and for animation (factor 2.3) gloss reuse factor = 1.6 (95 gloss lexemes) Experiment 13 Deaf test subjects (6m / 7f), aged Each session hrs (videotaped) Pure sign language environment: Deaf assistant, use of pictograms Warm-up: 3 easy avatar sentence

28 Delta Testing

29 Analysis Analysis of videos by Deaf experts Subjects' own rating usually misleading [Huenerfauth et al. 2008] Objective measure: count correctly recalled glosses only partial understanding? Subjective measure: expert rates understanding for each utterance Combine measures [Sheard et al. 2004]

30 Results avatar / absolute: 41.4 % avatar / relative: 58.4 %

31 Discussion Comprehensibility original video = 71 % "shockingly low" overarticulated remake = 82 % avatar = 58.4 % => close to state of the art Novel aspects: complex content direct comparison with human signers Delta testing factors out difficulties inherent to the material (dialect, speed, bad grammar) focus on the real "delta" between avatar and human

32 Conclusions How to make an ECA sign! EMBRScript as an interface language SL synthesis research: nonmanuals and prosody Thanks! Delta testing for comprehensibility Signing avatars can profit from ECAs, and vice versa 2nd workshop on Sign Language Translation and Avatar ASSETS 2011 Dundee! First workshop, Berlin, January 2011 Thanks to: Peter Schaar Iris König Silke Matthes Thomas Hanke