Social Signal Processing Understanding Nonverbal Behavior in Human- Human Interactions A.Vinciarelli University of Glasgow and Idiap Research Institute http://www.dcs.gla.ac.uk/~ vincia e- mail: vincia@dcs.gla.ac.uk 1
Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 2 2
Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 3 3
Socialism and Big Data 90 67.5 45 22.5 0 2004 2005 2006 2007 2008 2009 2010 2011 2012 Events showing the word social in their title (from dbworld, a multimedia retrieval mailing list) 4 4
Socialism and Big Data 90 67.5 45 22.5 0 2004 2005 2006 2007 2008 2009 2010 2011 2012 Events showing the word social in their title (from dbworld, a multimedia retrieval mailing list) 4 4
Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 5 5
Social Signals Vinciarelli, Pantic and Bourlard, Social Signal Processing: Survey of an Emerging Domain, Journal of Image and Vision Computing, 27(12):1743-1759, 2009 6 6
Social Signals Height Mutual Gaze Vocal Behavior Forward Posture Nonverbal Behavioral Cues Forward Posture Interpersonal Distance Gestures Vinciarelli, Pantic and Bourlard, Social Signal Processing: Survey of an Emerging Domain, Journal of Image and Vision Computing, 27(12):1743-1759, 2009 6 6
Social Signals Height Mutual Gaze Vocal Behavior Forward Posture Nonverbal Behavioral Cues Forward Posture Social Signals Interpersonal Distance Gestures Vinciarelli, Pantic and Bourlard, Social Signal Processing: Survey of an Emerging Domain, Journal of Image and Vision Computing, 27(12):1743-1759, 2009 6 6
Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 7 7
Conflict [Conflict is a] mode of interaction [where] the attainment of the goal by one party precludes its attainment by the others. C.M. Judd, Cognitive Effects of Attitude Conflict Resolution, Journal of Conflict Resolution, 22(3):483-498, 1978. 8 8
Data Samples Samples Length Subjects 1430 clips from Canal9 30 seconds (11h:55m) 138 (5 moderators) Subjects per Sample At least 2 Assessors Questionnaire 551 (10 per sample via MTurk) 15 items Total Items 214,500 Vinciarelli, Kim, Valente and Salamin, Automatic Detection of Conflicts in SpokenConversations, Proc. of IEEE International Conference on Audio, Speech and Signal Processing, pp. 1-4, 2012. 9 9
Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown 10 10
Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown + + 11 11
Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown + + 11 11
Measuring Conflict The atmosphere is relaxed People wait for their turn before speaking One or more people talk fast One or more people fidget People argue One or more people raise their voice One or more people shake their heads and nod People show mutual respect People interrupt one another One or more people gesture with their hands One or more people are aggressive The ambience is tense One or more people compete to talk People are actively engaged One or more people frown + + 11 11
Measuring Conflict Kim et al., Predicting the Conflict Level in Television Political Debates: an Approach Based on Crowdsourcing, Nonverbal Communication and Gaussian Processes, Proc. of ACM MM, pp. 793-796, 2012 12 12
Speech Analysis Clip-based statistics (pitch and intensity) 13 13
Speech Analysis Clip-based statistics (pitch and intensity) Turn-based statistics (pitch and intensity) 14 14
Speech Analysis Clip-based statistics (pitch and intensity) Turn-based statistics (pitch and intensity) Speaker activity statistics 15 15
Speech Analysis Clip-based statistics (prosody) Turn-based statistics (prosody) Speaker activity statistics (time budget) Overlapping speech statistics (prosody and length) 16 16
Results 0.85 Correlation Actual / Predicted Conflict Level 0.8 0.75 0.7 0.76 0.74 0.69 0.80 0.78 0.73 0.81 0.80 0.77 0.77 0.77 0.71 0.71 Manual Automatic w.o.s Automatic 0.81 0.74 0.65 BLR GPR RBF GPR ARD SVR LIN SVR RBF 17 17
Outline Socialism and Big Data Social Signal Processing The Conflict Example Conclusions 18 18
Conclusions 19 19
Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena 19 19
Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena Big Data is an opportunity: SSP approaches are data driven and can benefit from large corpora 19 19
Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena Big Data is an opportunity: SSP approaches are data driven and can benefit from large corpora Big Data is a challenge: Psychologically oriented methodologies need to be adapted 19 19
Conclusions Nonverbal behaviour provides machine detectable evidence of social and psychological phenomena Big Data is an opportunity: SSP approaches are data driven and can benefit from large corpora Big Data is a challenge: Psychologically oriented methodologies need to be adapted http://www.sspnet.eu 19 19
Thank You! 20