VideoStory A New Mul1media Embedding for Few- Example Recogni1on and Transla1on of Events

Transcription

1 VideoStory A New Mul1media Embedding for Few- Example Recogni1on and Transla1on of Events Amirhossein Habibian, Thomas Mensink, Cees Snoek ISLA, University of Amsterdam

2 Problem statement Recognize and translate video events Learning from few examples Provide seman1c interpreta1on of videos Event AMemp1ng bike trick Video Descrip1on 2

3 Video events in ACM Mul1media News events: earthquake, abdica/on, product launch Sport events: scoring goal, ace serve, slam dunk Social events: concert, debates, exhibi/ons Every day events: interac1ons of people and objects Repairing an appliance Working on sewing project Grooming an animal Birthday party 3

4 Recognizing events Represen1ng videos as histograms of low- level features Local descriptors Visual descriptors SIFT, HOG, GIST, Video descriptors MBH, STIP, Audio descriptors MFCC, AIM, Feature embedding Bag-of-words VLAD Fisher vector Audio-visual BoW Problem: very high- dimensional and non seman1cally 4 [Jiang et al., TRECVID 2010] [Natarajan et al., CVPR 2012] [Chen et al., MM 2013]

5 Recognizing and transla1ng events Represen1ng videos as histograms of concept scores Deep convolu1onal neural network Local descriptors Visual descriptors SIFT, HOG, GIST, Video descriptors MBH, STIP, Audio descriptors MFCC, AIM, Feature embedding Bag-of-words VLAD Fisher vector Audio-visual BoW Classification Attribute detection Concept detection Problem: define, annotate and train concept classifiers 5 [Smith et al., ICME 2003] [Hauptmann et al., TMM 2007] [Merler et al., TMM 2012] [Ma et al., MM 2012]

6 Recogni1on and transla1on by embedding W Embedding A Stunt Bike Motorcycle x i Joint space where x i W y i A Explicitly relate training W and A from mul1media y i A = Iden1ty matrix A = Projec1on matrix individual term classifiers select/group terms 6 [Rasiwasa et al., MM 2010] [Weston et al., IJCAI 2011] [Akata et al., CVPR 2013] [Das et al., WSDM 2013]

7 VideoStory: Embed the story of a video Stunt Bike Motorcycle x i W s i Embedding A y i Design criteria: learn W and A such that Descrip/veness: preserve video descrip1ons Predictability: recognize terms from video content 7

8 Key observa1on: Compelling forces Descrip1veness en predictability are compelling All grouped VideoStory No grouping Stunt/Bike/Motorcycle/ Stunt Bike/Motorcycle Stunt Bike Motorcycle 8

9 Why is this important? Grouping terms: Number of classes is reduced Training classifiers per group: More posi1ve examples available per group We can train from freely available web data 9

10 Key contribu1on: Joint op1miza1on Jointly op1mize for descrip1veness and predictability Video classifiers W Textual projec1on matrix A VideoStory embedding S VideoStory connects the two loss func1ons 10

11 VideoStory: Descrip1veness Reconstruct term vectors from VideoStory + A Textual projec1on matrix A VideoStory embedding S Term vectors y i Relates to: regularized latent seman1c indexing 11

12 VideoStory: Predictability Predict VideoStory from video features + W Video classifiers W VideoStory embedding S Video features x i Relates to: ridge regression 12

13 VideoStory: Training (1) VideoStory Training Set of videos and their cap1ons Video and descrip1ons Encode video features x i VideoStory Algorithm A W Fisher Vectors of MBH [Wang ICCV 13] Encode video descrip1ons y i Bag- of- words of terms 13

14 VideoStory: Training (2) VideoStory Training Video and descrip1ons Using Stochas/c Gradient Descent: Choose random sample Compute sample gradient wrt objec1ve VideoStory Algorithm W A Update parameters with step- size η 14 [BoMou ICCS 2010]

15 YouTube46K dataset Videos and 1tle descrip1ons from YouTube 46K videos, 19K unique terms in descrip1ons Seeded from video event descrip1ons Filters to remove low quality videos 15 Available for download:

16 VideoStory: Event classifier training VideoStory Training Event Classifier Training Video Labels Video and descrip1ons VideoStory Construc<on S Event Training VideoStory Algorithm A W Model Event classifiers: SVM with RBF kernel 16

17 Datasets for evalua1on TRECVID Mul1media Event Detec1on K videos - 20 events - 10 posi1ves train videos Columbia Consumer Video 9K videos - 15 events - 10 posi1ves train videos 17 [Jiang et al. ICMR 2011][Strassel et al. LREC 2012]

18 VideoStory: Recogni1on and transla1on VideoStory Training Event Classifier Training Video Labels Video and descrip1ons VideoStory Construc<on S Event Training Model VideoStory Algorithm W Recogni<on and Transla<on Evalua1on: A VideoStory Construc<on Video S Event Recogni<on Event score Mean Average Precision Event Transla<on Descrip<on Rouge- 1 18

19 Experiment 1: Effect of Embedding Frequent terms: train classifier for most frequent terms Grouping first: first descrip1veness; then predictability VideoStory: joint descrip1veness and predictability VideoStory outperforms other embeddings 19

20 Experiment 2: Story Quality vs. Quan1ty Expert10K: 10K TRECVID videos with expert descrip1ons YouTube10K: 10K random subset of YouTube46K dataset YouTube46K: 46K YouTube videos and descrip1ons Web supervision on par with expert provided descrip1ons 20

21 Experiment 3: VideoStory vs Others TRECVID MED Columbia CV AMributes [Habibian & Snoek CVIU 14] Low- Level MBH [Wang & Schmid ICCV 13] VideoStory - MBH

22 New Experiment: VideoStory with DeepNet AMributes [Habibian & Snoek CVIU 14] Low- Level MBH [Wang & Schmid ICCV 13] VideoStory - MBH CNN features [Zeiler & Fergus ECCV 14] VideoStory - CNN TRECVID MED

23 Experiment 4: VideoStory transla1on Getting Gettingaavehicle vehicleunstuck unstuck Predictions Predictions water water people people truck truck dump dump dog dog snow snow Term Term ss drive drive mud mud Predictions Predictions car car Rock Rockclimbing climbing climb climb hang hang dog dog fail fail rock rock boy boy wall wall indoor indoor Terms Terms Figure Figure 8: 8: VideoStory VideoStory event event recognition recognition and and translation translation results results on on REFERENCES REFERENCES [22] [22] Z. Z.Ma, Ma,Y. Y.Yang, Yang,Y

24 Experiment 4: VideoStory transla1on Evaluate on TRECVID MED Ground- truth: provided descrip1ons Measure with ROUGE- 1 VideoStory outperforms predefined amributes 24

25 Conclusions VideoStory a seman1c mul1media embedding Jointly op1mizes descrip1veness & predictability Training event classifiers from few examples Translate videos to textual descrip1on 25 Thank you!