Social Signal Processing Understanding Nonverbal Behavior in Human- Human Interactions

Transcription

1 Social Signal Processing Understanding Nonverbal Behavior in Human- Human Interactions A.Vinciarelli University of Glasgow and Idiap Research Institute vincia e- mail: 1

2 Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 2 2

3 Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 3 3

4 Socialism and Big Data Events showing the word social in their title (from dbworld, a multimedia retrieval mailing list) 4 4

5 Socialism and Big Data Events showing the word social in their title (from dbworld, a multimedia retrieval mailing list) 4 4

6 Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 5 5

7 Social Signals Vinciarelli, Pantic and Bourlard, Social Signal Processing: Survey of an Emerging Domain, Journal of Image and Vision Computing, 27(12): ,

8 Social Signals Height Mutual Gaze Vocal Behavior Forward Posture Nonverbal Behavioral Cues Forward Posture Interpersonal Distance Gestures Vinciarelli, Pantic and Bourlard, Social Signal Processing: Survey of an Emerging Domain, Journal of Image and Vision Computing, 27(12): ,

9 Social Signals Height Mutual Gaze Vocal Behavior Forward Posture Nonverbal Behavioral Cues Forward Posture Social Signals Interpersonal Distance Gestures Vinciarelli, Pantic and Bourlard, Social Signal Processing: Survey of an Emerging Domain, Journal of Image and Vision Computing, 27(12): ,

10 Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 7 7

11 Conflict [Conflict is a] mode of interaction [where] the attainment of the goal by one party precludes its attainment by the others. C.M. Judd, Cognitive Effects of Attitude Conflict Resolution, Journal of Conflict Resolution, 22(3): ,

12 Data Samples Samples Length Subjects 1430 clips from Canal9 30 seconds (11h:55m) 138 (5 moderators) Subjects per Sample At least 2 Assessors Questionnaire 551 (10 per sample via MTurk) 15 items Total Items 214,500 Vinciarelli, Kim, Valente and Salamin, Automatic Detection of Conflicts in SpokenConversations, Proc. of IEEE International Conference on Audio, Speech and Signal Processing, pp. 1-4,

13 Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown 10 10

14 Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown

15 Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown

16 Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown

17 Measuring Conflict Kim et al., Predicting the Conflict Level in Television Political Debates: an Approach Based on Crowdsourcing, Nonverbal Communication and Gaussian Processes, Proc. of ACM MM, pp ,

18 Speech Analysis Clip-based statistics (pitch and intensity) 13 13

19 Speech Analysis Clip-based statistics (pitch and intensity) Turn-based statistics (pitch and intensity) 14 14

20 Speech Analysis Clip-based statistics (pitch and intensity) Turn-based statistics (pitch and intensity) Speaker activity statistics 15 15

21 Speech Analysis Clip-based statistics (prosody) Turn-based statistics (prosody) Speaker activity statistics (time budget) Overlapping speech statistics (prosody and length) 16 16

22 Results 0.85 Correlation Actual / Predicted Conflict Level Manual Automatic w.o.s Automatic BLR GPR RBF GPR ARD SVR LIN SVR RBF 17 17

23 Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 18 18

24 Conclusions 19 19

25 Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena 19 19

26 Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena Big Data is an opportunity: SSP approaches are data driven and can benefit from large corpora 19 19

27 Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena Big Data is an opportunity: SSP approaches are data driven and can benefit from large corpora Big Data is a challenge: Psychologically oriented methodologies need to be adapted 19 19

28 Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena Big Data is an opportunity: SSP approaches are data driven and can benefit from large corpora Big Data is a challenge: Psychologically oriented methodologies need to be adapted

29 Thank You! 20