Sign Language Avatars Animation and Comprehensibility Michael Kipp* Alexis Heloir Quan Nguyen DFKI Embodied Agents Research Group Exzellenzcluster Multimodal Computing and Interaction Universität des Saarlandes * University of Applied Sciences Augsburg research funded by:
Sign language avatars... for the internet
Deaf Sign language is primary means of communication Sign language is a real language [Stokoe 1960] Specific SL for every country (ASL, DGS, LSF, BSL...) 500,000
Deaf = what is your name? Sign language is primary means of communication Sign language is a real language [Stokoe 1960] Specific SL for every country (ASL, DGS, LSF, BSL...) Sign language as a first language Spoken language is a foreign language 80% of deaf pupils leave school with significant reading/ writing problems 500,000
inexpensive editable possibly interactive comprehension limited commercial expensive not editable not interactive comprehensible 95%
Prior Work No standard writing system for sign language Glosses: based on meaning, tool for learning Notation: based on form, tool for science (Stokoe notation, HamNoSys) Milestones ViSiCAST (2000-2003): face-to-face translation // mocap esign (2002-2004): internet // procedural animation SiGML 60% comprehensibility Recent projects More flexible notations: Zebedee (LIMSI), PDTS-SiGML (U East Anglia) Avatars for American SL (Huenerfauth et al. // DePaul Univ.), Italian SL (ATLAS project), Czech SL (U West Bohemia)... YOUR NAME WHAT
What's your Agent's Native Language? Greta, U Paris 8 (2006) LIMSI, SNCF, web sourds (2006) SmartBody ICT (2004) Paula GeSSyCa... GUIDO, esign, Televirtual (2003) EMBR, DFKI (2009) Max Elckerlyc Marc...
What's your Agent's Native Language? - rich form vocabulary - validation by "understanding" Greta, U Paris 8 (2006) - speech-gesture sync. - lip syncing - locomotion LIMSI, SNCF, web sourds (2006) Universal Communicators SmartBody ICT (2004) Paula GeSSyCa... GUIDO, esign, Televirtual (2003) - control language - validating quality EMBR, DFKI (2009) Max Elckerlyc Marc...
Point of Departure Goal: Make every ECA "sign language ready" EMBR: EMBodied agent Realizer open source own animation language EMBRScript Comprehensiblity?
Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...)
Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises
Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises Facial expression: higher amplitude Mouth: sophisticated viseme set, should allow lipreading, use text-to-speech for visemes
Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises Facial expression: higher amplitude Mouth: sophisticated viseme set, should allow lipreading, use text-to-speech for visemes Gaze: separate eye-ball from head movement
Toward "sign language ready" Hand shapes: 10 => 60+ (finger alphabet...) Torso: lean/orientation, shoulder raises Facial expression: higher amplitude Mouth: sophisticated viseme set, should allow lipreading, use text-to-speech for visemes Gaze: separate eye-ball from head movement
Animating Sign Language: Attempt I Video: human signer's utterance Imitate utterance using EMBRScript Show EMBR animation
Failed!
Reasons Ambiguity in sign language fewer grammatical constructs Single sign level formational manual features situational nonmanual features (almost impossible) mouthing especially important in German SL Utterance level facial expression for sentence mode eyebrows + posture for information structure face as a visual focus point Casual signing style makes sign harder to read human signers compensate with all of the above
Consequences Working hypothesis: Avatars with current animation methods are unable to produce understandable "spontaneous" sign language Therefore: Overarticulate Involve Deaf experts Focus on nonmanual features Consider random facial movement
Original Overarticulated Remake Avatar
Attempt II Overarticulated remake transcribe glosses recording Gloss-based animation (lexicalized) compatible with EMBRScript tool support implications for HamNoSys
Video HamNoSys Animation
Video HamNoSys BML Animation
Video HamNoSys BML Animation Heloir, Kipp 2010 Kipp et al. 2010 Video BML EMBRScript Animation
Video HamNoSys BML Animation Heloir, Kipp 2010 Kipp et al. 2010 Video HamNoSys BML EMBRScript Animation
Video HamNoSys BML Animation Heloir, Kipp 2010 Kipp et al. 2010 Video EMBRScript Animation
Sample utterance: YOUR NAME WHAT gloss = sequence many sequences utterance pose seq. pose seq. pose seq. single pose pose sequence pose pose pose pose YOUR NAME WHAT
Evaluation Corpus 11 utterances (154 glosses) from German Deaf e-learning portal quite complex sentences Animation higher duration for remake (factor 1.8) and for animation (factor 2.3) gloss reuse factor = 1.6 (95 gloss lexemes) Experiment 13 Deaf test subjects (6m / 7f), aged 33-55 Each session 1.5-2 hrs (videotaped) Pure sign language environment: Deaf assistant, use of pictograms Warm-up: 3 easy avatar sentence
Delta Testing
Analysis Analysis of videos by Deaf experts Subjects' own rating usually misleading [Huenerfauth et al. 2008] Objective measure: count correctly recalled glosses only partial understanding? Subjective measure: expert rates understanding for each utterance Combine measures [Sheard et al. 2004]
Results avatar / absolute: 41.4 % avatar / relative: 58.4 %
Discussion Comprehensibility original video = 71 % "shockingly low" overarticulated remake = 82 % avatar = 58.4 % => close to state of the art Novel aspects: complex content direct comparison with human signers Delta testing factors out difficulties inherent to the material (dialect, speed, bad grammar) focus on the real "delta" between avatar and human
Conclusions How to make an ECA sign! EMBRScript as an interface language SL synthesis research: nonmanuals and prosody Thanks! Delta testing for comprehensibility Signing avatars can profit from ECAs, and vice versa 2nd workshop on Sign Language Translation and Avatar Technology @ACM ASSETS 2011 Dundee! First workshop, Berlin, January 2011 Thanks to: Peter Schaar Iris König Silke Matthes Thomas Hanke