Sign Language Recognition, Generation and Modeling with Application in Deaf Communication. Project Presentation

Transcription

1 Sign Language Recognition, Generation and Modeling with Application in Deaf Communication

2 UniS, Guildford UEA, Norwich UHH, Hamburg LIMSI, Orsay UPS, Toulouse WebSourd, Toulouse NTUA, Athens ILSP, Athens

3 Project Synergies Sign Recognition Sign Generation NTUA UniS UPS UEA ILSP UHH LIMSI WebSourd Sign Language Resources

4 Dicta-Sign addresses the need for communication of Deaf Communities via natural language Project goal To research and develop recognition and synthesis engines for sign languages (SLs) at a level of detail necessary for recognising and generating authentic signing. Research outcomes will be integrated in three laboratory prototypes: a Search-by-Example tool a SL-to-SL translator a sign-wiki Project demonstrator aimed to validate the interaction between sign recognition, linguistic modelling, and sign synthesis Prerequisite: Adequate linguistic knowledge and coded SL resources.

5 Overall Strategy of the Work Plan Research, technology-development, and evaluation/dissemination activities of the project are decomposed into 9 workpackages: WP 1: Visual Tracking and Feature Extraction WP 2: Continuous Sign Recognition WP 3: Synthesis and Animation WP 4: Linguistic Modelling WP 5: Annotation Tools WP 6: Sign Language Corpora WP 7: Bidirectional Integration WP 8: SL-to-SL Terminology Translator WP 9: Evaluation and Dissemination + WP10: Management

6 Graphical presentation of the Dicta-Sign components and their interdependencies

7 Goal Visual Tracking and Feature Extraction To develop Image processing and computer vision tools required for the sign video analysis, sign image detection and feature extraction. Targeted objectives: To develop the image processing and computer vision tools to locate and track key aspects of humans in video. To extract key image features from video for the annotation and classification of Sign. To detect and classify body movement; To detect and classify hand shape; To detect and classify facial expression To develop tools to assist in the HamNoSys notation of video by mapping body & facial motion features to HamNoSys features.

8 Video-based recognition: typical approach

9 Continuous Sign Recognition Goal: To research and develop an integrated continuous sign recognition system Intermediate tasks to this goal: Model Training, Temporal Alignment and Sign Segmentation by using as visual front-end the features produced in WP1 Incorporation of Language and Context Modelling Multimodal Fusion of manual and non-manual sign features Robustness in sign recognition through Signer Adaptation

10 Segmentation and tracking of the image sequence from the sign return in GSL using computer vision techniques

11 Synthesis and Animation Goal: To advance the state of the art in synthetically generated real-time sign language animation using virtual humans.

12 Signing avatar performance and skeleton modeling

13 Goal: Linguistic Modeling Definition of a high level linguistic framework relevant to sign languages Development of a common linguistic framework suitable to support both sign language recognition (WP1-WP2) and sign language generation (WP3) Development of sign language lexicon models to assist in: sign sequence generation and sign sequence recognition sign language-to-sign language translation. By using as input data the lemmatised corpora built in WP6.

14 Signing space editor

15 Goal: Annotation Tools To support integration of image processing and sign recognition (from WP1, WP2) as well as sign language knowledge (from WP4) through the development of annotation tools adapted for multi-disciplinary sign language studies. Targeted objectives: To specify a model for the annotation able to describe sign languages from linguistics and computer science points of view. To specify an architecture for programs to be able to accept plugins that facilitate the annotation process and to engineer WP1 and WP2 and related results into plugin modules. To integrate these results in existing annotation tools like ilex and AnColin and to develop an interconnection between them. increase efficiency and quality of sign language transcription

16 Annotation environment

17 Goal: Sign Language Corpora Parallel corpus construction for each of the four Dicta-Sign national sign languages: BSL, DGS, GSL, LSF. Existing lexica (DGS, GSL) will be adapted, and new ones (BSL, LSF) will be developed, so as to meet this objective. Common content without compromising authenticity Detailed work in this package will involve video recording of signers in the four languages, transcription and annotation of the resulting signed discourses. The annotation tools developed in WP5 will be used to help with this task. The corpora produced in WP6 will be input to WPs 1 4 and also as a value by themselves for any future sign linguistic analysis.

18 Instantiation of search in an annotated corpus

19 Bidirectional Integration Integration of results from WPs 1 5 into bidirectional prototypes will provide: Test-beds for the different technologies in meaningful contexts Proof that these technologies and models integrate well. Three laboratory prototypes will be implemented: Search-by-example in lexical databases for sign language Automatic sign language-to-sign language translation Sign-Wiki where instead of using text as the output medium, a signing avatar presents information. Goal: Parallel action in evaluation of all modules.

20 SL-to-SL Terminology Translator Dicta-Sign s showcase: Sign language-to-sign language terminology translator A domain specific demonstrator of the capabilities and industrial potential of the different technologies developed in the project. Goal: To demonstrate how sign language recognition, synthesis and Machine Translation based on linguistic knowledge combine to make possible communication between deaf users via natural language. Parallel action: showcase evaluation

21 Dissemination Project outcomes will become available: to the industry (providers of multimodal multilingual software applications) to the international scientific community to Dicta-Sign support organisations. Targeted objectives : To disseminate the project s developed tools and have them recognized by the deaf communities of the partner countries; To make scientific innovations available to the international scientific community and standardization bodies; To communicate innovations and elicit feedback from project support organizations interested in signed content production, To effectively construct the project exploitation plan.

22 Management Two major characteristics affect Dicta-Sign s management strategy: Basic scientific and technological research Innovation targets basic research rather than applied/industrial research Strong inter-disciplinary character tight interconnections among research tasks and work-packages Two-level management structure : Top-level management, also dealing with aspects of Scientific and technological development Dissemination