Local features and matching. Image classification & object localization

Transcription

1 Overview Instance level search Local features and matching Efficient visual recognition Image classification & object localization

2 Category recognition Image classification: assigning a class label to the image Car: present Cow: present Bike: not present Horse: not present

3 Category Tasks recognition Image classification: assigning a class label to the image Car: present Cow: present Bike: not present Horse: not present Object localization: define the location and the category Car Cow Location Category

4 Given Image classification Positive training images containing an object class Negative training images that don t Classify A test image as to whether it contains the object class or not?

5 Bag-of-features features for image classification Origin: texture recognition Texture is characterized by the repetition of basic elements or textons Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

6 Texture recognition histogram Universal texton dictionary Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

7 Bag-of-features features Origin: bag-of-words (text) Orderless document representation: frequencies of words from a dictionary Classification to determine document categories Bag-of-words Common o People Sculpture

8 Bag-of-features features for image classification SVM Extract regions Compute Find clusters Compute distance Classification descriptors and frequencies matrix [Nowak,Jurie&Triggs,ECCV 06], [Zhang,Marszalek,Lazebnik&Schmid,IJCV 07]

9 Bag-of-features features for image classification SVM Extract regions Compute Find clusters Compute distance Classification descriptors and frequencies matrix Step 1 Step 2 Step 3 [Nowak,Jurie&Triggs,ECCV 06], [Zhang,Marszalek,Lazebnik&Schmid,IJCV 07]

10 Step 1: feature extraction Scale-invariant image regions + SIFT Rotation invariance for many realistic collections too much invariance Dense descriptors Improve results in the context of categories (for most categories) Interest points do not necessarily capture all features Color-based descriptors Shape-based descriptors

11 Dense features - Multi-scale dense grid: extraction of small overlapping patches at multiple scales -Computation of the SIFT descriptor for each grid cells -Exp.: Horizontal/vertical step size 6 pixel, scaling factor of 1.2 per level

12 Bag-of-features features for image classification SVM Extract regions Compute Find clusters Compute distance Classification descriptors and frequencies matrix Step 1 Step 2 Step 3

13 Step 2: Quantization

14 Step 2:Quantization Clustering

15 Step 2: Quantization Visual vocabulary Clustering

16 Airplanes Motorbikes Faces Wild Cats Leaves People Bikes Examples for visual words

17 Step 2: Quantization Cluster descriptors K-means Gaussian mixture model Assign each visual word to a cluster Hard or soft assignment Build frequency histogram

18 Hard or soft assignment K-means hard assignment Assign to the closest cluster center Count number of descriptors assigned to a center Gaussian mixture model soft assignment Estimate distance to all centers Sum over number of descriptors Represent image by a frequency histogram

19 Image representation fr requenc cy.. codewords Each image is represented by a vector, typically dimension, normalization with L2 norm fine grained represent model instances coarse grained represent object categories

20 Bag-of-features features for image classification SVM Extract regions Compute Find clusters Compute distance Classification descriptors and frequencies matrix Step 1 Step 2 Step 3

21 Step 3: Classification Learn a decision rule (classifier) assigning bag-of- features representations of images to different classes Decision boundary Zebra Non-zebra

22 Training data Vectors are histograms, one from each training image positive negative Train classifier,e.g.svm

23 Classifiers K-nearest neighbor classifier Linear classifier Support Vector Machine Non-linear classifier Kernel trick Explicit lifting

24 Classification Assign input vector to one of two or more classes Any decision rule divides input space into decision regions separated by decision boundaries

25 Nearest Neighbor Classifier Assign label of nearest training data point to each test data point from Duda et al. Voronoi partitioning of feature space for 2-category 2-D and 3-D data

26 k-nearest Neighbors For a new point, find the k closest points from training data Labels of the k points vote to classify Works well provided there is lots of data and the distance function is good k =5

27 Linear classifiers Find linear function (hyperplane) to separate positive and negative examples x x i i positive : negative : x x i i w w b b 0 0 Which hyperplane is best?

28 Linear classifiers - margin x 2 (color) Generalization is not good in this case: x1(roundness) 1 Better if a margin is introduced: b/ w x 2 (color) x1(roundness) 1

29 Support vector machines Find hyperplane that maximizes the margin between the positive and negative examples x positive ( y 1) : x w b x i i negative( i y i 1) : x i i w b 1 1 For support, vectors, x i w b 1 i The margin is 2 / w Support vectors Margin

30 Nonlinear SVMs Datasets that are linearly separable work out great: 0 x But what if the dataset is just too hard? 0 x We can map it to a higher-dimensional space: x 2 0 x

31 Nonlinear SVMs General idea: the original input space can always be mapped to some higher-dimensional feature space where the training set is separable: Φ: x φ(x)

32 Nonlinear SVMs The kernel trick: instead of explicitly computing the lifting transformation φ(x), define a kernel function K such that K(x i,x j j) = φ(x i ) φ(x j ) This gives a nonlinear decision boundary in the original feature e space: i yik ( xi, x ) i b

33 Kernels for bags of features Kernels for bags of features Hellinger kernel N i h i h h h K ) ( ) ( ) ( Hellinger kernel i i h i h h h K ) ( ) ( ), ( Histogram intersection kernel N i i h i h h h I )) ( ), ( min( ), ( Generalized Gaussian kernel i 1 2 ) ( 1 exp ) ( h h D h h K Generalized Gaussian kernel D can be Euclidean distance, χ 2 distance etc ), ( exp ), ( h h D A h h K D can be Euclidean distance, χ distance etc. N i h i h h h D ) ( ) ( ), ( 2 i i h i h ) ( ) ( ), ( 2

34 Multi-class SVMs Various direct formulations exist, but they are not widely used in practice. It is more common to obtain multi-class SVMs by combining two-class SVMs in various ways. One versus all: Training: learn an SVM for each class versus the others Testing: apply each SVM to test example and assign to it the class of the SVM that returns the highest decision value One versus one: Training: learn an SVM for each pair of classes Testing: each learned SVM votes for a class to assign to the test example

35 Why does SVM learning work? Learns foreground and background visual words foreground words high weight background words low weight

36 Illustration Localization according to visual word probability Correct Image: 35 Correct Image: Correct Image: 38 Correct Image: foreground word more probable background word more probable

37 Illustration A linear SVM trained from positive and negative window descriptors A few of the highest weighed descriptor vector dimensions (= 'PAS + tile') + lie on object boundary (= local shape structures common to many training exemplars)

38 Bag-of-features features for image classification Excellent results in the presence of background clutter bikes books building cars people phones trees

39 Examples for misclassified images Books- misclassified into faces, faces, buildings Buildings- misclassified into faces, trees, trees Cars- misclassified into buildings, phones, phones

40 Bag of visual words summary Advantages: largely unaffected by position and orientation of object in image fixed length vector irrespective of number of detections very successful in classifying images according to the objects they contain Disadvantages: no explicit use of configuration of visual word positions poor at localizing objects within an image

41 Evaluation of image classification PASCAL VOC [05-10] datasets PASCAL VOC 2007 Training and test dataset available Used to report state-of-the-art the art results Collected January 2007 from Flickr images downloaded and random subset selected 20 classes Class labels per image + bounding boxes 5011 training i images, 4952 test t images Evaluation measure: average precision

42 PASCAL 2007 dataset

43 PASCAL 2007 dataset

44 Evaluation

45 Overview Instance level search Local features and matching Efficient visual recognition Image classification & object localization

46 Recognition Classification Object present/absent in an image Often presence of a significant amount of background clutter Localization / Detection Localize object within the frame Bounding box or pixellevel segmentation

47 Pixel-level level object classification

48 Difficulties Intra-class variations Scale and viewpoint change Multiple aspects of categories

49 Approaches Intra-class variation => Modeling of the variations, mainly by learning from a large dataset, for example by SVMs Scale + limited viewpoints changes => multi-scale approach or invariant local features Multiple aspects of categories => separate detectors for each aspect, front/profile face, build an approximate 3D category model

50 Approaches Localization (bounding box) Sliding window approach Localization (segmentation) Pixel-based +MRF

51 Training Localization with sliding window Positive examples Negative examples Description + Learn a classifier

52 Localization with sliding window Testing at multiple l locations and scales Find local maxima, non-maxima suppression

53 Haar Wavelet / SVM Human Detector Haar wavelet descriptors Training set (2k positive / 10k negative) training 1326-D descriptor descriptors test Support vector machine results Test image [Papageorgiou & Poggio, 1998] Multi-scale search 53

54 Which Descriptors s are Important? 32x32 descriptors 16x16 descriptors Mean response difference between positive & negative training examples Essentially just a coarse-scale human silhouette template!

55 Some Detection Results

56 The Viola/Jones Face Detector A seminal approach to real-time object detection Training is slow, but detection is very fast Key ideas Integral images for fast feature evaluation Boosting for feature selection Attentional cascade for fast rejection of non-face windows P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. CVPR P. Viola and M. Jones. Robust real-time face detection. IJCV 57(2), 2004.

57 Image Features Rectangle filters Value = (pixels in white area) (pixels in white area) (pixels in black area)

58 Fast computation with integral images The integral image computes a value at each pixel (x,y) that is the sum (x,y) of the pixel values above and to the left of (x,y), inclusive This can quickly be computed in one pass through the image

59 Feature selection For a 24x24 detection region, the number of possible rectangle features is ~160,000!

60 Feature selection For a 24x24 detection region, the number of possible rectangle features is ~160,000! At test time, it is impractical to evaluate the entire feature set Can we create a good classifier using just a small subset of all possible features? How to select such a subset?

61 Boosting Boosting is a classification scheme that works by combining weak learners into a more accurate ensemble classifier Training consists of multiple boosting rounds During each boosting round, we select a weak learner that does well on examples that were hard for the previous weak learners Hardness is captured by weights attached to training examples Y. Freund and R. Schapire, A short introduction to boosting, Journal of Japanese Society for Artificial Intelligence, 14(5): , September, 1999.

62 Boosting for face detection First two features selected by boosting: This feature combination can yield 100% s eatu e co b at o ca y e d 00% detection rate and 50% false positive rate

63 Attentional cascade We start with simple classifiers which reject many of the negative sub-windows while detecting almost all positive sub-windows Positive response from the first classifier triggers the evaluation of a second (more complex) classifier, and so on A negative outcome at any point leads to the immediate rejection of the sub-window IMAGE SUB-WINDOW Classifier 1 T Classifier 2 T Classifier 3 T FACE F F F NON-FACE NON-FACE NON-FACE

64 The implemented system Training Data 5000 faces All frontal, rescaled to 24x24 pixels 300 million non-faces 9500 non-face images Faces are normalized Scale, translation Many variations Across individuals Illumination Pose

65 Result of Face Detector on Test Images

66 Profile Detection

67 Profile Features

68 Summary: Viola/Jones detector Rectangle features Integral images for fast computation ti Boosting for feature selection Attentional cascade for fast rejection of negative windows Available in open CV

69 Histogram of Oriented Gradient Human Detector Descriptors are a grid of local Histograms of Oriented Gradients (HOG) Linear SVM for runtime efficiency Tolerates different poses, clothing, lighting and background Assumes upright fully visible people Importance weighted responses 69 [Dalal & Triggs, CVPR 2005]

70 Human detection