Bag-of-Words models. Lecture 9. Slides from: S. Lazebnik, A. Torralba, L. Fei-Fei, D. Lowe, C. Szurka

Transcription

1 Bag-of-Words models Lecture 9 Sldes from: S. Lazebnk, A. Torralba, L. Fe-Fe, D. Lowe, C. Szurka

2 Bag-of-features models

3 Overvew: Bag-of-features models Orgns and motvaton Image representaton Dscrmnatve methods Nearest-neghbor classfcaton Support vector machnes Generatve methods Naïve Bayes Probablstc Latent Semantc Analyss Extensons: ncorporatng spatal nformaton

4 Orgn 1: Texture recognton Texture s characterzed by the repetton of basc elements or textons For stochastc textures, t s the dentty of the textons, not ther spatal arrangement, that matters Julesz, 1981; Cula & Dana, 2001; Leung & Malk 2001; Mor, Belonge & Malk, 2001; Schmd 2001; Varma & Zsserman, 2002, 2003; Lazebnk, Schmd & Ponce, 2003

5 Orgn 1: Texture recognton hstogram Unversal texton dctonary Julesz, 1981; Cula & Dana, 2001; Leung & Malk 2001; Mor, Belonge & Malk, 2001; Schmd 2001; Varma & Zsserman, 2002, 2003; Lazebnk, Schmd & Ponce, 2003

6 Orgn 2: Bag-of-words models Orderless document representaton: frequences of words from a dctonary Salton & McGll (1983)

7 Orgn 2: Bag-of-words models Orderless document representaton: frequences of words from a dctonary Salton & McGll (1983) US Presdental Speeches Tag Cloud

10 Bags of features for mage classfcaton 1. Extract features

11 Bags of features for mage classfcaton 1. Extract features 2. Learn vsual vocabulary

12 Bags of features for mage classfcaton 1. Extract features 2. Learn vsual vocabulary 3. Quantze features usng vsual vocabulary

13 Bags of features for mage 1. Extract features classfcaton 2. Learn vsual vocabulary 3. Quantze features usng vsual vocabulary 4. Represent mages by frequences of vsual words

14 1. Feature extracton Regular grd Vogel & Schele, 2003 Fe-Fe & Perona, 2005

15 1. Feature extracton Regular grd Vogel & Schele, 2003 Fe-Fe & Perona, 2005 Interest pont detector Csurka et al Fe-Fe & Perona, 2005 Svc et al. 2005

16 1. Feature extracton Regular grd Vogel & Schele, 2003 Fe-Fe & Perona, 2005 Interest pont detector Csurka et al Fe-Fe & Perona, 2005 Svc et al Other methods Random samplng (Vdal-Naquet & Ullman, 2002) Segmentaton-based patches (Barnard et al. 2003)

17 1. Feature extracton Compute SIFT descrptor [Lowe 99] Normalze patch Detect patches [Mkojaczyk and Schmd 02] [Mata, Chum, Urban & Pajdla, 02] [Svc & Zsserman, 03] Slde credt: Josef Svc

18 1. Feature extracton

19 2. Learnng the vsual vocabulary

20 2. Learnng the vsual vocabulary Clusterng Slde credt: Josef Svc

21 2. Learnng the vsual vocabulary Vsual vocabulary Clusterng Slde credt: Josef Svc

22 K-means clusterng Want to mnmze sum of squared Eucldean dstances between ponts x and ther nearest cluster centers m k D( X, M ) clusterk pont n clusterk ( x m k Algorthm: Randomly ntalze K cluster centers Iterate untl convergence: Assgn each data pont to the nearest center Recompute each cluster center as the mean of all ponts assgned to t 2 )

23 From clusterng to vector quantzaton Clusterng s a common method for learnng a vsual vocabulary or codebook Unsupervsed learnng process Each cluster center produced by k-means becomes a codevector Codebook can be learned on separate tranng set Provded the tranng set s suffcently representatve, the codebook wll be unversal The codebook s used for quantzng features A vector quantzer takes a feature vector and maps t to the ndex of the nearest codevector n a codebook Codebook = vsual vocabulary Codevector = vsual word

24 Example vsual vocabulary Fe-Fe et al. 2005

25 Image patch examples of vsual words Svc et al. 2005

26 Vsual vocabulares: Issues How to choose vocabulary sze? Too small: vsual words not representatve of all patches Too large: quantzaton artfacts, overfttng Generatve or dscrmnatve learnng? Computatonal effcency Vocabulary trees (Nster & Stewenus, 2006)

27 frequency 3. Image representaton.. codewords

28 Image classfcaton Gven the bag-of-features representatons of mages from dfferent classes, how do we learn a model for dstngushng them?

29 Dscrmnatve and generatve methods for bags of features Zebra Non-zebra

30 Image classfcaton Gven the bag-of-features representatons of mages from dfferent classes, how do we learn a model for dstngushng them?

31 Dscrmnatve methods Learn a decson rule (classfer) assgnng bagof-features representatons of mages to dfferent classes Decson boundary Zebra Non-zebra

32 Classfcaton Assgn nput vector to one of two or more classes Any decson rule dvdes nput space nto decson regons separated by decson boundares

33 Nearest Neghbor Classfer Assgn label of nearest tranng data pont to each test data pont from Duda et al. Vorono parttonng of feature space for two-category 2D and 3D data Source: D. Lowe

34 K-Nearest Neghbors For a new pont, fnd the k closest ponts from tranng data Labels of the k ponts vote to classfy Works well provded there s lots of data and the dstance functon s good k = 5 Source: D. Lowe

35 Functons for comparng hstograms L1 dstance χ 2 dstance Quadratc dstance (cross-bn) N h h h h D ) ( ) ( ), ( Jan Puzcha, Yoss Rubner, Carlo Tomas, Joachm M. Buhmann: Emprcal Evaluaton of Dssmlarty Measures for Color and Texture. ICCV 1999 N h h h h h h D ) ( ) ( ) ( ) ( ), ( j j j h h A h h D, )) ( ) ( ( ), (

36 Earth Mover s Dstance Each mage s represented by a sgnature S consstng of a set of centers {m } and weghts {w } Centers can be codewords from unversal vocabulary, clusters of features n the mage, or ndvdual features (n whch case quantzaton s not requred) Earth Mover s Dstance has the form EMD( S 1, S 2 ) where the flows f j are gven by the soluton of a transportaton problem, j f j d( m f 1 j, m 2 j ) Y. Rubner, C. Tomas, and L. Gubas: A Metrc for Dstrbutons wth Applcatons to Image Databases. ICCV 1998

37 Lnear classfers Fnd lnear functon (hyperplane) to separate postve and negatve examples x x postve: negatve : x x w w b b 0 0 Whch hyperplane s best? Slde: S. Lazebnk

38 Support vector machnes Fnd hyperplane that maxmzes the margn between the postve and negatve examples C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1998

39 Support vector machnes Fnd hyperplane that maxmzes the margn between the postve and negatve examples x x postve ( y negatve ( y 1) : 1) : x x w w b b 1 1 For support vectors, x w b 1 x w Dstance between pont and hyperplane: w b Therefore, the margn s 2 / w Support vectors Margn C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1998

40 Fndng the maxmum margn hyperplane 1. Maxmze margn 2/ w 2. Correctly classfy all tranng data: x x postve ( y negatve ( y 1) : Quadratc optmzaton problem: Mnmze 1) : 1 2 w T w Subject to y (w x +b) 1 x x w w b b 1 1 C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1998

41 Fndng the maxmum margn w hyperplane Soluton: y x learned weght Support vector C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1998

42 Fndng the maxmum margn hyperplane Soluton: w b = y w x y x for any support vector Classfcaton functon (decson boundary): w x b Notce that t reles on an nner product between the test pont x and the support vectors x Solvng the optmzaton problem also nvolves computng the nner products x x j between all pars of tranng ponts C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1998 y x x b

43 Nonlnear SVMs Datasets that are lnearly separable work out great: 0 x But what f the dataset s just too hard? 0 x We can map t to a hgher-dmensonal space: x 2 0 x Slde credt: Andrew Moore

44 Nonlnear SVMs General dea: the orgnal nput space can always be mapped to some hgherdmensonal feature space where the tranng set s separable: Φ: x φ(x) Slde credt: Andrew Moore

45 Nonlnear SVMs The kernel trck: nstead of explctly computng the lftng transformaton φ(x), defne a kernel functon K such that K(x,x j ) = φ(x ) φ(x j ) (to be vald, the kernel functon must satsfy Mercer s condton) Ths gves a nonlnear decson boundary n the orgnal feature space: yk( x, x) C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 1998 b

46 Kernels for bags of features Hstogram ntersecton kernel: Generalzed Gaussan kernel: 1 K( h, h2 ) exp D( h1, h A D can be Eucldean dstance, χ 2 dstance, Earth Mover s Dstance, etc. N I( h1, h2 ) mn( h1 ( ), h2 ( )) ) J. Zhang, M. Marszalek, S. Lazebnk, and C. Schmd, Local Features and Kernels for Classfcaton of Texture and Object Categores: A Comprehensve Study, IJCV 2007

47 Summary: SVMs for mage classfcaton 1. Pck an mage representaton (n our case, bag of features) 2. Pck a kernel functon for that representaton 3. Compute the matrx of kernel values between every par of tranng examples 4. Feed the kernel matrx nto your favorte SVM solver to obtan support vectors and weghts 5. At test tme: compute kernel values for your test example and each support vector, and combne them wth the learned weghts to get the value of the decson functon

48 What about mult-class SVMs? Unfortunately, there s no defntve mult-class SVM formulaton In practce, we have to obtan a mult-class SVM by combnng multple two-class SVMs One vs. others Tranng: learn an SVM for each class vs. the others Testng: apply each SVM to test example and assgn to t the class of the SVM that returns the hghest decson value One vs. one Tranng: learn an SVM for each par of classes Testng: each learned SVM votes for a class to assgn to the test example Slde: S. Lazebnk

49 SVMs: Pros and cons Pros Many publcly avalable SVM packages: Kernel-based framework s very powerful, flexble SVMs work very well n practce, even wth very small tranng sample szes Cons No drect mult-class SVM, must combne twoclass SVMs Computaton, memory Durng tranng tme, must compute matrx of kernel values for every par of examples Learnng can take a very long tme for large-scale problems Slde: S. Lazebnk

50 Summary: Dscrmnatve methods Nearest-neghbor and k-nearest-neghbor classfers L1 dstance, χ 2 dstance, quadratc dstance, Earth Mover s Dstance Support vector machnes Lnear classfers Margn maxmzaton The kernel trck Kernel functons: hstogram ntersecton, generalzed Gaussan, pyramd match Mult-class Of course, there are many other classfers out there Neural networks, boostng, decson trees,

51 Generatve learnng methods for bags of features Model the probablty of a bag of features gven a class

52 Generatve methods We wll cover two models, both nspred by text document analyss: Naïve Bayes Probablstc Latent Semantc Analyss

53 The Naïve Bayes model Assume that each feature s condtonally ndependent gven the class N,, f c) p( f c) 1 N 1 p( f f : th feature n the mage N: number of features n the mage Csurka et al. 2004

54 The Naïve Bayes model Assume that each feature s condtonally ndependent gven the class N M n( w 1,, f N c) p( f c) p( wj c) 1 j 1 p( f j ) f : th feature n the mage N: number of features n the mage w j : jth vsual word n the vocabulary M: sze of vsual vocabulary n(w j ): number of features of type w j n the mage Csurka et al. 2004

55 The Naïve Bayes model Assume that each feature s condtonally ndependent gven the class N M n( w 1,, f N c) p( f c) p( wj c) 1 j 1 p( f j ) p(w j c) = No. of features of type w j n tranng mages of class c Total no. of features n tranng mages of class c Csurka et al. 2004

56 The Naïve Bayes model Assume that each feature s condtonally ndependent gven the class N M n( w 1,, f N c) p( f c) p( wj c) 1 j 1 p( f j ) p(w j c) = No. of features of type w j n tranng mages of class c + 1 Total no. of features n tranng mages of class c + M (Laplace smoothng to avod zero counts) Csurka et al. 2004

57 M j j j c M j w n j c c w p w n c p c w p c p c j 1 1 ) ( ) ( )log ( ) ( log arg max ) ( ) ( arg max * The Naïve Bayes model Csurka et al Maxmum A Posteror decson: (you should compute the log of the lkelhood nstead of the lkelhood tself n order to avod underflow)

58 The Naïve Bayes model Graphcal model : c w N Csurka et al. 2004

59 Probablstc Latent Semantc Analyss = p 1 + p 2 + p 3 Image zebra grass tree vsual topcs T. Hofmann, Probablstc Latent Semantc Analyss, UAI 1999

60 Probablstc Latent Semantc Analyss Unsupervsed technque Two-level generatve model: a document s a mxture of topcs, and each topc has ts own characterstc word dstrbuton d z w document topc word P(z d) P(w z) T. Hofmann, Probablstc Latent Semantc Analyss, UAI 1999

61 Probablstc Latent Semantc Analyss Unsupervsed technque Two-level generatve model: a document s a mxture of topcs, and each topc has ts own characterstc word dstrbuton w d z T. Hofmann, Probablstc Latent Semantc Analyss, UAI 1999 K k j k k j d z p z w p d w p 1 ) ( ) ( ) (

62 The plsa model p( w d j ) k K 1 p( w z k ) p( z k d j ) Probablty of word n document j (known) Probablty of word gven topc k (unknown) Probablty of topc k gven document j (unknown)

63 words words topcs The plsa model p( w d j ) k K 1 p( w z k ) p( z k d j ) documents topcs documents p(z k d j ) = p(w d j ) p(w z k ) Observed codeword dstrbutons (M N) Codeword dstrbutons per topc (class) (M K) Class dstrbutons per mage (K N)

64 Learnng plsa parameters Maxmze lkelhood of data: Observed counts of word n document j M number of codewords N number of mages Slde credt: Josef Svc

65 Inference Fndng the most lkely topc (class) for an mage: z arg max z p( z d )

66 Inference ) ( max arg d z p z z Fndng the most lkely topc (class) for an mage: z z z d z p z w p d z p z w p w d z p z ) ( ) ( ) ( ) ( arg max ), ( arg max Fndng the most lkely topc (class) for a vsual word n a gven mage:

67 Topc dscovery n mages J. Svc, B. Russell, A. Efros, A. Zsserman, B. Freeman, Dscoverng Objects and ther Locaton n Images, ICCV 2005

68 Applcaton of plsa: Acton recognton Space-tme nterest ponts Juan Carlos Nebles, Hongcheng Wang and L Fe-Fe, Unsupervsed Learnng of Human Acton Categores Usng Spatal-Temporal Words, IJCV 2008.

69 Applcaton of plsa: Acton recognton Juan Carlos Nebles, Hongcheng Wang and L Fe-Fe, Unsupervsed Learnng of Human Acton Categores Usng Spatal-Temporal Words, IJCV 2008.

70 plsa model K p( w d j ) k 1 p( w z k ) p( z k d j ) Probablty of word n vdeo j (known) Probablty of word gven topc k (unknown) w = spatal-temporal word d j = vdeo n(w, d j ) = co-occurrence table (# of occurrences of word w n vdeo d j ) z = topc, correspondng to an acton Probablty of topc k gven vdeo j (unknown)

71 Acton recognton example

72 Multple Actons

73 Multple Actons

74 Summary: Generatve models Naïve Bayes Ungram models n document analyss Assumes condtonal ndependence of words gven class Parameter estmaton: frequency countng Probablstc Latent Semantc Analyss Unsupervsed technque Each document s a mxture of topcs (mage s a mxture of classes) Can be thought of as matrx decomposton Parameter estmaton: Expectaton-Maxmzaton

75 Addng spatal nformaton Computng bags of features on sub-wndows of the whole mage Usng codebooks to vote for object poston Generatve part-based models

76 Spatal pyramd representaton Extenson of a bag of features Locally orderless representaton at several levels of resoluton level 0 Lazebnk, Schmd & Ponce (CVPR 2006) Slde: S. Lazebnk

77 Spatal pyramd representaton Extenson of a bag of features Locally orderless representaton at several levels of resoluton level 0 level 1 Lazebnk, Schmd & Ponce (CVPR 2006) Slde: S. Lazebnk

78 Spatal pyramd representaton Extenson of a bag of features Locally orderless representaton at several levels of resoluton level 0 level 1 level 2 Lazebnk, Schmd & Ponce (CVPR 2006) Slde: S. Lazebnk

79 Scene category dataset Mult-class classfcaton results (100 tranng mages per class) Slde: S. Lazebnk

80 Caltech101 dataset Mult-class classfcaton results (30 tranng mages per class) Slde: S. Lazebnk

81 Examples from PASCAL VOC Challenge 2010