Image and Video Understanding

Transcription

1 Image and Video Understanding 2VO WS Christoph Feichtenhofer, Axel Pinz Slide credits: Many thanks to all the great computer vision researchers on which this presentation relies on. Most material is taken from tutorials at NIPS, CVPR and BMVC conferences.

2 Outline Convolutional Networs (ConvNets) for Image Classification Operations in each layer Architecture Training Visualizations Krizhevsky, A., Sutskever, I. and Hinton, G. E., ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012 M. Zeiler & R. Fergus, Visualizing and Understanding Convolutional Networks, ECCV, 2014 A. Mahendran and A. Vedaldi. Understanding Deep Image Representations by Inverting Them CVPR 2015 State of the art methods Applications of ConvNets K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR S. Ioffe, C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, ICML 2015 K. He, X. Zhang, S. Ren, & j. Sun. Deep Residual Learning for Image Recognition. arxiv 2015.

3 Goal: Scene Understanding Car Person Leave a car in the desert Scenes Objects Actions Activities Desert Temporal input sequence Get out of car Open door

4 One application: Image retrieval

5 Deep Learning - breakthrough in visual and speech recognition Credit: B. Ginzburg

6 ConvNet Detection Examples

7 Credit: R. Fergus Summary: Compare: SIFT Descriptor Image Pixels Apply Gabor filters Spatial pool (Sum) Normalize to unit length Feature Vector

8 What are the weakest links limiting performance? Replace each component of the deformable part model detector with humans Good Features (part detection) and accurate Localization (NMS) are most important [Parikh & Zitnick CVPR 10]

9 Typical visual recognition pipeline Select / develop features Add on top of this Machine Learning for multi-class recognition and train classifier Image/Video Pixels Feature Extractor e.g. SIFT, HoG... Trainable Classifier e.g. SVM Object Class

10 Intuition Behind Deep Neural Nets Build features automatically based on training data Combine feature extraction and classification All the way from pixels classifier Each box is a feature detector One layer extracts features from output of previous layer Layer 1 Layer 2 Layer 3 CAR

11 Some Key Ingredients for Convolutional Neural Networks

12 Neural networks trained via backpropagation B-Prop Back-propagate error signal to get derivatives for learning parameters θ Compare outputs with correct answer to get error signal L outputs F-Prop Training hidden layers F-Prop / B-Prop (features) Learning by stochastic gradient descent (SGD): A) Compute loss L on weights small mini-batch of data input vector B) Compute gradient w.r.t. θ C) Use gradient to update θ (make a step in the opposite direction) Non-linearity Credit: G. Hinton

13 Neural networks trained via backpropagation Cat Compare outputs with correct answer to get error signal Back-propagate error signal to get derivatives for learning parameters θ Training F-Prop / B-Prop Learning by SGD: A) Compute loss on small mini-batch B) Compute gradient w.r.t. θ C) Use gradient to update θ input vector Credit: G. Hinton

14 Neural networks trained via backpropagation Dog Compare outputs with correct answer to get error signal Back-propagate error signal to get derivatives for learning parameters θ Training F-Prop / B-Prop Learning by SGD: A) Compute loss on small mini-batch B) Compute gradient w.r.t. θ C) Use gradient to update θ input vector Credit: G. Hinton

15 Credit: G. Hinton Neural networks testing Cat F-Prop

16 Credit: G. Hinton Neural networks testing Dog F-Prop

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32 Summary: Modularity of backpropagation Credit: A. Vedaldi

33 Credit: M. A. Ranzato Motivation: Images as a composition of local parts Pixel-based representation Too many parameters!

34 Motivation: Images as a composition of local parts Patch-based representation Credit: M. A. Ranzato

35 Motivation: Images as a composition of local parts Patch-based representation Credit: M. A. Ranzato Still too many parameters!

36 Motivation: Images as a composition of local parts Convolution example Credit: M. A. Ranzato

37 Motivation: Images as a composition of local parts Convolution example Credit: M. A. Ranzato

38 Credit: R. Fergus Motivation: Images as a composition of local parts Filtering example Why translation equivariance? Input translation leads to a translation of features Fewer filters needed: no translated replications But still need to cover orientation/frequency Patch-based Patch-based Convolutional

39 Convolutional Neural Networks

40 Multistage HubelWiesel Architecture: An Old Idea for Local Shift Invariance [Hubel & Wiesel 1962] Simple cells detect local features Complex cells pool the outputs of simple cells within a retinotopic neighborhood. Simple cells Complex cells Multiple convolutions pooling & subsampling Convolutional Networks [LeCun 1988-present] Retinotopic Feature Maps Credit: Y. LeCun

41 Convolutional Neural Networks Neural network with specialized connectivity structure After a few convolution and subsampling stages, spatial resolution is very small Use fully connected layers up to classification at output layer [LeCun et al. 1989] [LeCun et al. 1989]

42 Training large ConvNets on Paradigm shift enabled by Large annotated dataset Strong regularization (dropout) GPU(s) for fast processing ~ 150 images/sec days weeks of training Imagenet database: 1K classes ~ 1K training images per class! ~ 1M training images [Russakovsky et al. ICCV 13]

43 Top-5 error rate % ImageNet Classification 2012 Krizhevsky et al % error (top-5) Next best (non-convnet) 26.2% error SuperVision ISI Oxford INRIA Amsterdam Credit: R. Fergus

44 Architecture of [Krizhevsky et al. NIPS 12] ImageNet Classification % error (top-5) next best (variant of SIFT + Fisher Vectors) 26.2% error Same idea as in [LeCun 98] but on a larger scale: more training images (10 6 vs 10 3 ) more hidden layers Overall: 60,000,000 parameters which are trained on 2 GPUs for a week with several tricks to reduce overfitting Data augmentation DropOut (new regularization)

45 ConvNet Architecture Feed-forward: Convolve input Non-linearity (rectified linear) Pooling (local max) Supervised Train convolutional filters by back-propagating classification error Feature maps Pooling Non-linearity Convolution (Learned) Input Image Credit: R. Fergus

46 Components of Each Layer Pixels / Features Filter with Dictionary (convolutional or tiled) + Non-linearity Spatial/Feature (Sum or Max) [Optional] Normalization between feature responses Output Features Credit: R. Fergus

47 Convolutional filtering Why convolution? Statistics of images look similar at different locations Dependencies are very local Filtering is an opteration with translation equivariance... Input Feature Map Credit: R. Fergus

48 Non-Linearity Non-linearity applied to each response Per-feature independent Tanh Sigmoid: 1/(1+exp(-x)) Rectified linear : max(0,x) Simplifies backprop Makes learning faster Avoids saturation issues (much higher dynamic range) Preferred option Credit: A. Vedaldi

49 Pooling Spatial Pooling Non-overlapping / overlapping regions (-0.3% in error) Sum or max Provides invariance to local transformations Max Sum Credit: R. Fergus

50 Normalization Contrast normalization The same response for different contrasts is desired

51 Normalization Contrast normalization (between/across feature maps) Local mean = 0, local std. = 1, Local 7x7 Gaussian Equalizes the features maps Feature Maps Feature Maps After Contrast Normalization Credit: R. Fergus

52 Local feature normalisation Credit: A. Vedaldi

53 Recap: Components of a CNN CNN - multi-layer NN architecture Convolutional + Non-Linear Layer Sub-sampling Layer Convolutional +Non-Linear Layer Fully connected layers Supervised Feature Extraction Classification Credit: B. Ginzburg

54 Summary: Components of Each Layer Credit: A. Vedaldi

55 Architecture of Krizhevsky et al. Softmax Output Layer 7: Full Trained via backprop by maximizing the log-prob. of the correct class-label 8 layers total Trained on ImageNet dataset Layer 6: Full Layer 5: Conv + Pool Layer 4: Conv Layer 3: Conv Layer 2: Conv + Pool Layer 1: Conv + Pool Input Image Credit: R. Fergus

56 Motivation: Improving Generalization: DropOut Random Forests generalize well due to averaging of many models Decision Trees are fast - ConvNets are slow many models are not feasible Similar to random forest bagging [Breiman 94], but differs in that parameters are shared For fully connected layers only: In training: Independently set each hidden unit activity to zero with 0.5 probability In testing multiply neuron output by 0.5 [Hinton et al. NIPS 12] Corresponds to averaging over exponentially many samples from different model architectures

57 [Chatfield et al. BMVC 14] Mean removal Centered (0- mean) RGB values. Input representation Further pre-processing tricks Centered (0-mean) RGB values. Data augmentation Train on 224x224 patches extracted randomly from images, and also their horizontal reflections An input image (256x256) Minus sign The mean input image [Krizhevsky et al. NIPS 12]

58 Test error (top-5) Credit: R. Fergus ImageNet Classification 2013/2014 Results Pre-2012: 26.2% error 2012: 16.5% error 2013: 11.2% error

59 ImageNet Sample classifications [Krizhevsky et al. NIPS 12]

60 Validation ImageNet Sample classifications [Krizhevsky et al. NIPS 12]

61 ImageNet Classification Progress from Credit: Y. LeCun

62 Credit: K. Simonyan

63 Better ( deeper) architectures exist now ILSVRC14 Winners: ~7.3% Top-5 error VGG: 16 layers of stacked 3x3 convolution with stride 1 Credit: K. Simonyan

64 Better ( deeper) architectures exist now ILSVRC14 Winners: ~7.3% Top-5 error VGG: 16 layers of stacked 3x3 convolution with stride 1 Credit: K. Simonyan

65 Better ( deeper) architectures exist now ILSVRC14 Winners: ~6.6% Top-5 error - GoogLeNet: composition of multi-scale dimensionreduced modules: Zeiler-Fergus Architecture (1 tower) - 1x1 convolutions serve as dimensionality reduction Convolution Pooling Softmax Other wire together what fires together Credit: C. Szegedy

66 In images, correlations tend to be local Credit: C. Szegedy

67 Cover very local clusters by 1x1 convolutions number of filters 1x1 Credit: C. Szegedy

68 Less spread out correlations number of filters 1x1 Credit: C. Szegedy

69 Cover more spread out clusters by 3x3 convolutions number of filters 1x1 3x3 Credit: C. Szegedy

70 Cover more spread out clusters by 5x5 convolutions number of filters 1x1 3x3 Credit: C. Szegedy

71 Cover more spread out clusters by 5x5 convolutions number of filters 1x1 3x3 5x5 Credit: C. Szegedy

72 A heterogeneous set of convolutions number of filters 1x1 3x3 5x5 Credit: C. Szegedy

73 Schematic view (naive version) number of filters 1x1 Filter concatenation 3x3 1x1 convolutions 3x3 convolutions 5x5 convolutions 5x5 Previous layer Credit: C. Szegedy

74 Naive idea Filter concatenation 1x1 convolutions 3x3 convolutions 5x5 convolutions Previous layer Credit: C. Szegedy

75 Naive idea (does not work!) - Too many parameters Filter concatenation 1x1 convolutions 3x3 convolutions 5x5 convolutions 3x3 max pooling Previous layer Credit: C. Szegedy

76 Inception module Filter concatenation 1x1 convolutions 3x3 convolutions 1x1 convolutions 5x5 convolutions 1x1 convolutions 1x1 convolutions 3x3 max pooling Previous layer 1x1 Convolution Only operates across channels Performs a linear projections of a single pixel s features Can be used to reduce dimensionality Increases depth

77 Inception 9 Inception modules Network in a network in a network... Convolution Pooling Softmax Other Credit: C. Szegedy

78 Classification failure cases Groundtruth: coffee mug GoogLeNet: table lamp lamp shade printer projector desktop computer Credit: C. Szegedy

79 Classification failure cases Groundtruth: hay GoogLeNet: sorrel (horse) hartebeest Arabian camel warthog gaselle Credit: C. Szegedy

80 Classification failure cases Groundtruth: Police car GoogLeNet: laptop hair drier binocular ATM machine seat belt Credit: C. Szegedy

81 Credit: K. Simonyan

82 Credit: K. Simonyan

83 Credit: K. Simonyan

84 Better ( deeper) architectures exist now ILSVRC15 Winners: ~3.57% Top-5 error Credit: K. He

85 Faster R-CNN + ResNet Examples Credit: K. He

86 Faster R-CNN + ResNet Examples Credit: K. He

87 Faster R-CNN + ResNet Examples Credit: K. He

88 Better ( deeper) architectures exist now ILSVRC15 Winners: Object detection revolution Credit: K. He

89 Better ( deeper) architectures exist now Ultra deep all 3x3 conv filters (except first: 7x7) spatial size /2 # filters x2 Uses Batch Normalisation during training Standard hyperparameters & data augmentation No hidden fc layer No max pooling (except after first layer) No dropout 2014 Credit: K. He

90 Simply stacking layers does not work Credit: K. He

91 Simply stacking layers does not work Credit: K. He

92 Solution: Residual learning Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arxiv Credit: K. He

93 Breaking CNNs Intriguing properties of neural networks [Szegedy ICLR 2014]

94 What is going on? x E x x x E x Adversarial examples: modify the image to increase classifier error Explaining and Harnessing Adversarial Examples [Goodfellow ICLR 2015]

95 What is learned? Visualizing CNNs M. Zeiler & R. Fergus, Visualizing and Understanding Convolutional Networks, ECCV, 2014 A. Mahendran and A. Vedaldi. Understanding Deep Image Representations by Inverting Them, CVPR 2015

96 Visualization using Deconvolutional Networks Provides way to map activations at high layers back to the input [Zeiler et al. CVPR 10, ICCV 11, ECCV 14] Feature maps Same operations as Convnet, but in reverse: Unpool feature maps Convolve unpooled maps Filters copied from Convnet Used here purely as a probe Originally proposed as unsupervised learning method No inference, no learning Unpooling Non-linearity Convolution (learned) Input Image Credit: R. Fergus

97 Credit: R. Fergus Unpooling Operation Switches record where the pooled activations came from to unpool the reconstructed layer above

98 Deconvnet Credit: R. Fergus Deconvnet Projection from Higher Layers [Zeiler and Fergus. ECCV 14] Filters Feature Map... Filters Layer 2 Reconstruction Layer 2: Feature maps Layer 1 Reconstruction Layer 1: Feature maps Convnet Visualization Input Image

99 Visualizations of Higher Layers Use ImageNet 2012 validation set (stack of images) Push each image through network and look for image with the strongest activation for each feature map Feature Map... Take max activation from feature map associated with each filter Lower Layers Filters Use Deconvnet to project back to pixel space Use pooling switches distinctive to that activation Input Image Validation Images Credit: R. Fergus

100 Layer 1 Filters Credit: R. Fergus

101 Layer 1: Top-9 Patches Credit: R. Fergus

102 Layer 2: Top-1 Credit: R. Fergus

103 Layer 2: Top-9 Parts of input image that give strong activation of this feature map

104 Layer 2: Top-9 Patches Patches from validation images that give maximal activation of a given feature map

105 Layer 3: Top-1

106 Layer 3: Top-9

107 Layer 3: Top-9 Layer 3: Top-9 Patches

108 Layer 4: Top-1

109 Layer 4: Top-9

110 Layer 4: Top-9 patches

111 Layer 5: Top-1

112 Layer 5: Top-9

113 Layer 5: Top-9 Patches

114 Evolution of Features During Training

115 Credit: R. Fergus Evolution of Features During Training Higher layers evolve later in later epochs during training.

116 Visualizations can be used to improve model Visualization of Krizhevsky et al. s architecture showed some problems with layers 1 and 2 Alter architecture: smaller stride & filter size Visualizations look better and Performance improves Blocking artifacts Smaller stride for convolution Layer 2 filters Credit: R. Fergus

117 Credit: R. Fergus Too specific for low-level + dead filters Restrict size (smaller) Layer 1 filters Visualizations can be used to improve model Visualization of Krizhevsky et al. s architecture showed some problems with layers 1 and 2 Alter architecture: smaller stride & filter size Visualizations look better and Performance improves 11x11 filters, stride 4 7x7 filters, stride 2

118 Credit: R. Fergus Occlusion Experiment Mask parts of input with occluding square Monitor output of classification network Perhaps network using scene context?

119 Input image p(true class) Most probable class Credit: R. Fergus

120 Input image p(true class) Most probable class Credit: R. Fergus

121 Input image p(true class) Most probable class Credit: R. Fergus

122 Finding a Pre-Image Credit: A. Vedaldi

123 Images are locally smooth Credit: A. Vedaldi

124 Finding a pre-image at intermediat layers Credit: A. Vedaldi