Tools & Resources for Visualising Conversational-Speech Interaction

Transcription

1 Tools & Resources for Visualising Conversational-Speech Interaction Nick Campbell NiCT/ATR-SLC Keihanna Science City, Kyoto, Japan.

2 Preamble large corpus data examples new stuff conclusion Greetings from the NiCT/ATR labs in Kyoto (from

3 Introduction With ever-growing increases in the amount of data available for speech technology research, it is now increasingly difficult for any one individual to become personally familiar with all of the data in any given corpus. Yet without the insights provided by first-hand inspection of the types and variety of speech material being collected, it is difficult to ensure that appropriate models and features are being used in the processing of the speech data.

4 Introduction (cont.) It is perhaps not necessary (and often physically very difficult) to listen to all of the speech in a given corpus but it is essential to be able to select in a non-random manner specific sections of the corpus for closer inspection and analysis. If the data is transcribed, the transcriptions will provide the first key into the speech data but there are many aspects of a spoken message that are not well described by a plain text rendering of the linguistic content.

5 The JST/ATR ESP Corpus 5-years of recordings 50 paid volunteer speakers close-talking head-mounted mics worn throughout the day 1,500 hours of manually transcribed conversational speech = a lot of noise to analyse!!

6 The ATR SCOPE/Kaken Dialogue Corpora 3-years of dialogue recordings both camera & microphone recordings multi-person business and social interactions also annotated for discourse moves = a lot of data to associate!!

7 New ways to access the data an interactive web-based interface viewing dialogue structure - patterns of interaction accessing movement data & aligning it with speech segments finding synchrony in discourse participation measuring rapport in a conversation

8 Hierarchical structure to the data & transcriptions all transcriptions are (manually) time-aligned multiple views of the same discourse in separate directories all files linked through cgi scripts using timing infomation accessed through ordered search new annotations linked as data grows and is shared

9 : top FEAST: Feature Extraction & Analysis for Speech Technology

10 The top-level web-pages for ESP C ESP-C: a 100-hour telephone conversation corpus

11 Multimodal Dialogue Data (with Kobe University) Multimodal data - annotated video & audio

12 Visualising a conversation Speech activity for the two participants, showing overlaps & turns

13 View of a Multi-Party Dialogue In a multi-party discourse, each speaker is colour-coded

14 Text of a multi-person dialogue Transcription lines are clickable - each can be listened to

15 Display of a multi-person dialogue Colour-coded bar-plots allow a better view of the interaction

16 Labelling the intensity of a discourse (rapport) Three labellers indicate discourse involvement for 1 speaker

17 Viewing the video clip per utterance segment By clicking on a bar, its video sequence can be viewed

18 Plotting head and body motion Or movement data plotted using the same colour-coding

19 Gesture individual examples of a given gesture can be listed for viewing

20 Searching by Gesture Label Gestures have been labelled and can be searched for by type

21 SWISH - a Google-type interface to large text data A fast search-engine interface is included for text-based search

22 Feature-based (targeted) search Search conditions can be specified in a discourse-specific way

23 Multinational English Conversational Data Our latest data is very multi-source (and multi-modal)

24 Multiple views (and recordings) of the same scene

25 Subtitles and multi-view composite video

26 Transcriptions scroll as the video plays Speech overlaps can be interactively visualised in this way

27 Scanning Discourse Segments bar-plot displays can be scrolled with the video (karaoke)

28 Rapport / Discourse Intensity Subjective (and automatic) measures of discourse activity

29 Animated data streaming (synced with the video) Automatically-determined measures of head/body activity (x,y,z)

30 topic labels

31 topic labels for day-2

32 topic labels for day-3

33 outside

34 Conclusion We are working with large quantities of dialogue speech including business meetings, friendly discourse, and telephone conversations, and have produced web-based tools for the visualisation of non-verbal and paralinguistic features of the speech data. This paper has described some of our tools and techniques for accessing large quantities of speech data and for the visualisation of discourse interactions and events at levels above that of linguistic content. In essence, they provide higher-level displays so that specific sections of speech, text, or other annotation can be accessed by the researcher and provide an interactive interface to the large amount of data through an Archive Browser.

35 Conclusion (cont.) thanks Thank you for listening further information data and examples can be found at