BIL Computer Vision Mar 5, 2014

Transcription

1 BIL Computer Vision Mar 5, 2014 Image pyramids Aykut Erdem Dept. of Computer Engineering Hacettepe University

2 Image Scaling This image is too big to fit on the screen. How can we reduce it? How to generate a half-sized version? Slide credit: S. Seitz!2

3 Image Sub-Sampling 1/8 1/4 Slide credit: S. Seitz Throw away every other row and column to create a 1/2 size image - called image subsampling

4 Slide credit: S. Seitz Image Sub-Sampling 1/2 1/4 (2x zoom) 1/8 (4x zoom)

5 Good and Bad Sampling Good sampling: Sample often or, Sample wisely Bad sampling: Aliasing! Slide credit: S. Narasimhan

6 Aliasing Slide credit: F. Durand!6

7 Aliasing Occurs when your sampling rate is not high enough to capture the amount of detail in your image Can give you the wrong signal/image an alias! To do sampling right, need to understand the structure of your signal/image Enter Monsieur Fourier! To avoid aliasing: - sampling rate 2 * max frequency in the image said another way: two samples per cycle - This minimum sampling rate is called the Nyquist rate!7 Slide credit: L. Zhang

8 Aliasing When downsampling by a factor of two!!! - Original image has frequencies that are too high How can we fix this? Slide credit: N. Snavely!8

9 Slide credit: S. Seitz Gaussian pre-filtering G 1/8 G 1/4 Gaussian 1/2 Solution: filter the image, then subsample

10 Slide credit: S. Seitz Subsampling with Gaussian pre-filtering Gaussian 1/2 G 1/4 G 1/8 Solution: filter the image, then subsample

11 Slide credit: S. Seitz Compare with... 1/2 1/4 (2x zoom) 1/8 (4x zoom)

12 Gaussian prefiltering Slide credit: N. Snavely Solution: filter the image, then subsample blur F 0 F 0 H * F 1 subsample blur subsample F 1 H * F 2

13 { Gaussian pyramid Slide credit: N. Snavely blur F 0 F 0 H * F 1 subsample blur subsample F 1 H * F 2

14 Image pyramids Image information occurs at all spatial scales! Gaussian pyramid Laplacian pyramid Wavelet/QMF pyramid Steerable pyramid Slide credit: B. Freeman and A. Torralba

15 Slide credit: B. Freeman and A. Torralba The Gaussian pyramid Smooth with gaussians, because a gaussian*gaussian=another gaussian Gaussians are low pass filters, so representation is redundant.

16 The computational advantage of pyramids [Burt and Adelson, 1983] Slide credit: B. Freeman and A. Torralba

17 The Gaussian Pyramid [Burt and Adelson, 1983] Slide credit: B. Freeman and A. Torralba

18 Slide credit: B. Freeman and A. Torralba

19 Convolution and subsampling as a matrix multiply (1D case)!! Slide credit: B. Freeman and A. Torralba (Normalization constant of 1/16 omitted for visual clarity.)

20 Next pyramid level!! Slide credit: B. Freeman and A. Torralba

21 Slide credit: B. Freeman and A. Torralba The combined effect of the two pyramid levels!!!

23 Gaussian pyramids used for up- or downsampling images. Multi-resolution image analysis Look for an object over various spatial scales Coarse-to-fine image processing: form blur estimate or the motion analysis on very low-resolution image, upsample and repeat. Often a successful strategy for avoiding local minima in complicated estimation tasks. Slide credit: B. Freeman and A. Torralba

24 Slide credit: B. Freeman and A. Torralba 1D Gaussian pyramid matrix, for [ ] low-pass filter full-band image, highest resolution lower-resolution image lowest resolution image

26 Slide credit: B. Freeman and A. Torralba The Laplacian Pyramid Synthesis Compute the difference between upsampled Gaussian pyramid level and Gaussian pyramid level. band pass filter - each level represents spatial frequencies (largely) unrepresented at other level.

27 The Laplacian Pyramid!27

28 Slide credit: B. Freeman and A. Torralba Upsampling Insert zeros between pixels, then apply a low-pass filter, [ ]

29 Showing, at full resolution, the information captured at each level of a Gaussian (top) and Laplacian (bottom) pyramid. Slide credit: B. Freeman and A. Torralba

30 Laplacian pyramid reconstruction algorithm: recover x 1 from L 1, L 2, L 3 and x 4 Slide credit: B. Freeman and A. Torralba G# is the blur-and-downsample operator at pyramid level # F# is the blur-and-upsample operator at pyramid level #! Laplacian pyramid elements: L1 = (I F1 G1) x1 L2 = (I F2 G2) x2 L3 = (I F3 G3) x3 x2 = G1 x1 x3 = G2 x2 x4 = G3 x3!! Reconstruction of original image (x1) from Laplacian pyramid elements: x3 = L3 + F3 x4 x2 = L2 + F2 x3 x1 = L1 + F1 x2

31 Laplacian pyramid reconstruction algorithm: recover x 1 from L 1, L 2, L 3 and g 3 Slide credit: B. Freeman and A. Torralba + + +

34 1D Laplacian pyramid matrix, for [ ] low-pass filter Slide credit: B. Freeman and A. Torralba high frequencies mid-band frequencies low frequencies

35 Laplacian pyramid applications Texture synthesis Image compression Noise removal Slide credit: B. Freeman and A. Torralba

36 Image blending Slide credit: B. Freeman and A. Torralba

37 Szeliski, Computer Vision, 2010 Slide credit: B. Freeman & A. Torralba

38 Image blending Build Laplacian pyramid for both images: LA, LB Build Gaussian pyramid for mask: G Build a combined Laplacian pyramid: L(j) = G(j) LA(j) + (1-G(j)) LB(j) Collapse L to obtain the blended image 33 Slide credit: B. Freeman and A. Torralba

40 Slide credit: A. Torralba Linear transforms transformed image! f = U 1! F Linear transform Vectorized image Note: not all important transforms need to have an inverse

41 Slide credit: A. Torralba Linear transforms Pixels U=

42 Slide credit: A. Torralba Linear transforms Pixels U= Derivative U=

43 Slide credit: A. Torralba Linear transforms Pixels U= Derivative U= Integration 0 1 U -1 = No locality for reconstruction - Needs boundary

44 Slide credit: A. Torralba Haar transform The simplest set of functions: U= U -1 =

45 Slide credit: A. Torralba Haar transform The simplest set of functions: U= U -1 = To code a signal, repeat at several locations: U= U -1 = ½

46 Haar transform Reordering rows Low pass High pass Apply the same decomposition to the Low pass component: = And repeat the same operation to the low pass component, until length 1. Note: each subband is sub-sampled and has aliased signal components. Slide credit: A. Torralba

47 Slide credit: A. Torralba Haar transform The entire process can be written as a single matrix: Average Multiscale derivatives

48 Slide credit: A. Torralba Haar transform U= U -1 = Properties: Orthogonal decomposition Perfect reconstruction Critically sampled

49 2D Haar transform Basic elements:

50 2D Haar transform Basic elements: = 2 Low pass 50

51 2D Haar transform Basic elements: = 2 Low pass 1 1 = 1-1 = 1-1 = 51

52 2D Haar transform Basic elements: = 2 Low pass 1 1 = = =

53 2D Haar transform Basic elements: = 2 Low pass 1 1 = 2 High pass vertical 1-1 = High pass horizontal 1-1 = High pass diagonal

54 Slide credit: B. Freeman and A. Torralba 2D Haar transform Sketch of the Fourier transform 2 2 Horizontal low pass, Vertical low- pass Horizontal high pass, vertical low- pass Horizontal low pass, vertical high- pass - 2 Horizontal high pass, vertical high pass

55 Pyramid cascade Simoncelli and Adelson, in Subband coding, Kluwer, Slide credit: B. Freeman and A. Torralba

56 Slide credit: B. Freeman and A. Torralba Wavelet/QMF representation Same number of pixels!

58 Steerable Pyramid 2 Level decomposition of white circle example: Low pass residual Subbands Images from: Slide credit: B. Freeman and A. Torralba

59 Steerable Pyramid We may combine Steerability with Pyramids to get a Steerable Laplacian Pyramid as shown below Decomposition Reconstruction Images from: Slide credit: B. Freeman and A. Torralba

60 Steerable Pyramid We may combine Steerability with Pyramids to get a Steerable Laplacian Pyramid as shown below Decomposition Reconstruction Images from: Slide credit: B. Freeman and A. Torralba

61 Steerable Pyramid But we need to get rid of the corner regions before starting the recursive circular filtering Simoncelli and Freeman, Slide credit: B. Freeman and A. Torralba

62 Reprinted from Shiftable MultiScale Transforms, by Simoncelli et al., IEEE Transactions on Information Theory, 1992, copyright 1992, IEEE There is also a high pass residual Slide credit: B. Freeman and A. Torralba

63 63

64 Monroe 64

65 Dog or cat? 65

66 Almost no dog information 66

67 Summary of pyramid representations 67

68 Image pyramids Gaussian!! Laplacian!! Wavelet/QMF!! Steerable pyramid Progressively blurred and subsampled versions of the image. Adds scale invariance to fixed-size algorithms. Shows the information added in Gaussian pyramid at each spatial scale. Useful for noise reduction & coding. Bandpassed representation, complete, but with aliasing and some non-oriented subbands. Shows components at each scale and orientation separately. Nonaliased subbands. Good for texture and feature analysis. But overcomplete and with HF residual. 68

69 Schematic pictures of each matrix Shown for 1-d images transform The matrices for 2-d images are the same idea, but more complicated, to account for vertical, as well as horizontal, neighbor relationships. transformed image Vectorized image Fourier transform, or Wavelet transform, or Steerable pyramid transform 69

70 Slide credit: B. Freeman and A. Torralba Gaussian pyramid = * Gaussian pyramid pixel image Overcomplete representation. Low-pass filters, sampled appropriately for their blur.

71 Slide credit: B. Freeman and A. Torralba Laplacian pyramid = * Laplacian pyramid pixel image Overcomplete representation. Transformed pixels represent bandpassed image information.

72 Slide credit: B. Freeman and A. Torralba Wavelet (QMF) transform Wavelet pyramid = * Ortho-normal transform (like Fourier transform), but with localized basis functions. pixel image

73 Slide credit: B. Freeman and A. Torralba Steerable pyramid Multiple orientations at = one scale * Steerable pyramid Multiple orientations at the next scale the next scale pixel image Over-complete representation, but nonaliased subbands.

74 Why use image pyramids? Handle real-world size variations with a constant-size vision algorithm. Remove noise Analyze texture Recognize objects Label image features Image priors can be specified naturally in terms of wavelet pyramids. Slide credit: B. Freeman and A. Torralba