Image Stitching. Richard Szeliski Microsoft Research IPAM Graduate Summer School: Computer Vision July 26, 2013

Transcription

1 Image Stitching Richard Szeliski Microsoft Research IPAM Graduate Summer School: Computer Vision July 26, 2013

2 Panoramic Image Mosaics Full screen panoramas (cubic): Mars: New Years Eve: Richard Szeliski Image Stitching 2

3 Gigapixel panoramas & images Mapping / Tourism / WWT Medical Imaging Richard Szeliski Image Stitching 3

4 Gigapixel panoramas & images Photosynth Richard Szeliski Image Stitching 4

5 Image Mosaics = Goal: Stitch together several images into a seamless composite Richard Szeliski Image Stitching 5

6 Today s lecture Image alignment and stitching motion models image warping point-based alignment complete mosaics (global alignment) compositing and blending ghost and parallax removal Richard Szeliski Image Stitching 6

7 Readings Szeliski, CVAA: Chapter 3.6: Image warping Chapter 6.1: Feature-based alignment Chapter 9.1: Motion models Chapter 9.2: Global alignment Chapter 9.3: Compositing Recognizing Panoramas, Brown & Lowe, ICCV 2003 Szeliski & Shum, SIGGRAPH'97 Richard Szeliski Image Stitching 7

8 Motion models

9 Motion models What happens when we take two images with a camera and try to align them? translation? rotation? scale? affine? perspective? see interactive demo (VideoMosaic) Richard Szeliski Image Stitching 9

10 Projective transformations (aka homographies) keystone distortions Richard Szeliski Image Stitching 10

11 Image Warping

12 Image Warping image filtering: change range of image g(x) = h(f(x)) f h g x image warping: change domain of image g(x) = f(h(x)) x f h g x Richard Szeliski Image Stitching 12 x

13 Image Warping image filtering: change range of image g(x) = h(f(x)) f image warping: change domain of image g(x) = f(h(x)) h g f h g Richard Szeliski Image Stitching 13

14 Parametric (global) warping Examples of parametric warps: translation rotation aspect perspective affine cylindrical Richard Szeliski Image Stitching 14

15 2D coordinate transformations translation: x = x + t x = (x,y) rotation: similarity: affine: x = R x + t x = s R x + t x = A x + t perspective: x H x x = (x,y,1) (x is a homogeneous coordinate) These all form a nested group (closed w/ inv.) Richard Szeliski Image Stitching 15

16 Image Warping Given a coordinate transform x = h(x) and a source image f(x), how do we compute a transformed image g(x ) = f(h(x))? h(x) x f(x) x g(x ) Richard Szeliski Image Stitching 16

17 Forward Warping Send each pixel f(x) to its corresponding location x = h(x) in g(x ) What if pixel lands between two pixels? h(x) x f(x) x g(x ) Richard Szeliski Image Stitching 17

18 Forward Warping Send each pixel f(x) to its corresponding location x = h(x) in g(x ) What if pixel lands between two pixels? Answer: add contribution to several pixels, normalize later (splatting) h(x) x f(x) x g(x ) Richard Szeliski Image Stitching 18

19 Inverse Warping Get each pixel g(x ) from its corresponding location x = h(x) in f(x) What if pixel comes from between two pixels? h(x) x f(x) x g(x ) Richard Szeliski Image Stitching 19

20 Inverse Warping Get each pixel g(x ) from its corresponding location x = h(x) in f(x) What if pixel comes from between two pixels? Answer: resample color value from interpolated (prefiltered) source image x f(x) x g(x ) Richard Szeliski Image Stitching 20

21 Interpolation Possible interpolation filters: nearest neighbor bilinear bicubic (interpolating) sinc / FIR Needed to prevent jaggies and texture crawl Richard Szeliski Image Stitching 21

22 Motion models (reprise)

23 Motion models Translation Affine Perspective 3D rotation 2 unknowns 6 unknowns 8 unknowns 3 unknowns Richard Szeliski Image Stitching 25

24 Finding the transformation Translation = 2 degrees of freedom Similarity = 4 degrees of freedom Affine = 6 degrees of freedom Homography = 8 degrees of freedom How many corresponding points do we need to solve? Richard Szeliski Image Stitching 26

25 Plane perspective mosaics 8-parameter generalization of affine motion works for pure rotation or planar surfaces Limitations: local minima slow convergence difficult to control interactively Richard Szeliski Image Stitching 27

26 Rotational mosaics Directly optimize rotation and focal length Advantages: ability to build full-view panoramas easier to control interactively more stable and accurate estimates Richard Szeliski Image Stitching 29

27 3D 2D Perspective Projection (X c,y c,z c ) f u c u Richard Szeliski Image Stitching 30

28 Rotational mosaic Projection equations 1. Project from image to 3D ray (x 0,y 0,z 0 ) = (u 0 -u c,v 0 -v c,f) 2. Rotate the ray by camera motion (x 1,y 1,z 1 ) = R 01 (x 0,y 0,z 0 ) 3. Project back into new (source) image (u 1,v 1 ) = (fx 1 /z 1 +u c,fy 1 /z 1 +v c ) Richard Szeliski Image Stitching 31

29 Image Mosaics (Stitching) [Szeliski & Shum, SIGGRAPH 97] [Szeliski, FnT CVCG, 2006]

30 Image Mosaics (stitching) Blend together several overlapping images into one seamless mosaic (composite) = Richard Szeliski Image Stitching 37

31 Mosaics for Video Coding Convert masked images into a background sprite for content-based coding = Richard Szeliski Image Stitching 38

32 Establishing correspondences 1. Direct method: Use generalization of affine motion model [Szeliski & Shum 97] 2. Feature-based method Extract features, match, find consisten inliers [Lowe ICCV 99; Schmid ICCV 98, Brown&Lowe ICCV 2003] Compute R from correspondences (absolute orientation) Richard Szeliski Image Stitching 39

33 Stitching demo Richard Szeliski Image Stitching 41

34 Panoramas What if you want a 360 field of view? mosaic Projection Cylinder Richard Szeliski Image Stitching 42

35 Cylindrical panoramas Steps Reproject each image onto a cylinder Blend Output the resulting mosaic Richard Szeliski Image Stitching 43

36 Cylindrical Panoramas Map image to cylindrical or spherical coordinates need known focal length Image 384x300 f = 180 (pixels) f = 280 f = 380 Richard Szeliski Image Stitching 44

37 Cylindrical projection Map 3D point (X,Y,Z) onto cylinder Y Z X unit cylinder Convert to cylindrical coordinates Convert to cylindrical image coordinates s defines size of the final image unwrapped cylinder cylindrical image Richard Szeliski Image Stitching 47

38 Cylindrical warping Given focal length f and image center (x c,y c ) (X,Y,Z) Y (sinq,h,cosq) Z X Richard Szeliski Image Stitching 48

39 Spherical warping Given focal length f and image center (x c,y c ) (x,y,z) φ cos φ Y Z X (sinθcosφ, sinφ, cosθcosφ) sin φ cos θ cos φ Richard Szeliski Image Stitching 49

40 3D rotation Rotate image before placing on unrolled sphere (x,y,z) (sinθcosφ, sinφ, cosθcosφ) p = R p φ cos φ sin φ cos θ cos φ Richard Szeliski Image Stitching 50

41 Radial distortion Correct for bending in wide field of view lenses Project to normalized image coordinates Apply radial distortion Apply focal length translate image center To model lens distortion Use above projection operation instead of standard projection matrix multiplication Richard Szeliski Image Stitching 51

42 Fisheye lens Extreme bending in ultra-wide fields of view Richard Szeliski Image Stitching 52

43 Matching features What do we do about the bad matches? Richard Szeliski Image Stitching 53

44 RAndom SAmple Consensus Select one match, count inliers Richard Szeliski Image Stitching 54

45 RAndom SAmple Consensus Select one match, count inliers Richard Szeliski Image Stitching 55

46 Least squares fit Find average translation vector Richard Szeliski Image Stitching 56

47 RANSAC for estimating homography RANSAC loop: 1. Select four feature pairs (at random) 2. Compute homography H (exact) 3. Compute inliers where p i, H p i < ε Keep largest set of inliers Re-compute least-squares H estimate using all of the inliers Richard Szeliski Image Stitching 57

48 Simple example: fit a line Rather than homography H (8 numbers) fit y=ax+b (2 numbers a, b) to 2D pairs 58

49 Simple example: fit a line Pick 2 points Fit line Count inliers 3 inliers 59

50 Simple example: fit a line Pick 2 points Fit line Count inliers 4 inliers 60

51 Simple example: fit a line Pick 2 points Fit line Count inliers 9 inliers 61

52 Simple example: fit a line Pick 2 points Fit line Count inliers 8 inliers 62

53 Simple example: fit a line Use biggest set of inliers Do least-square fit 63

54 RANSAC red: rejected by 2nd nearest neighbor criterion blue: Ransac outliers yellow: inliers Richard Szeliski Image Stitching 64

55 Image Stitching review 1. Align the images over each other camera pan translation on cylinder 2. Blend the images together (demo) Richard Szeliski Image Stitching 65

56 Full-view (360 spherical) panoramas

57 Full-view Panorama Richard Szeliski Image Stitching 70

58 Texture Mapped Model Richard Szeliski Image Stitching 71

59 Global alignment Register all pairwise overlapping images Use a 3D rotation model (one R per image) Use direct alignment (patch centers) or feature based Infer overlaps based on previous matches (incremental) Optionally discover which images overlap other images using feature selection (RANSAC) Richard Szeliski Image Stitching 72

60 Recognizing Panoramas Matthew Brown & David Lowe ICCV 2003

61 Recognizing Panoramas [Brown & Lowe, ICCV 03] Richard Szeliski Image Stitching 74

62 Finding the panoramas Richard Szeliski Image Stitching 75

63 Finding the panoramas Richard Szeliski Image Stitching 76

64 Finding the panoramas Richard Szeliski Image Stitching 77

65 Finding the panoramas Richard Szeliski Image Stitching 78

66 Fully automated 2D stitching Demo Richard Szeliski Image Stitching 79

67 Richard Szeliski Image Stitching 80

68 Rec.pano.: system components 1. Feature detection and description more uniform point density 2. Fast matching (hash table) 3. RANSAC filtering of matches 4. Intensity-based verification 5. Incremental bundle adjustment [M. Brown, R. Szeliski, and S. Winder. Multi-image matching using multi-scale oriented patches, CVPR'2005] Richard Szeliski Image Stitching 81

69 Probabilistic Feature Matching Richard Szeliski Image Stitching 85

70 RANSAC motion model Richard Szeliski Image Stitching 86

71 RANSAC motion model Richard Szeliski Image Stitching 87

72 RANSAC motion model Richard Szeliski Image Stitching 88

73 Probabilistic model for verification Richard Szeliski Image Stitching 89

74 How well does this work? Test on 100s of examples

75 How well does this work? Test on 100s of examples still too many failures (5-10%) for consumer application

76 Matching Mistakes: False Positive Richard Szeliski Image Stitching 92

77 Matching Mistakes: False Positive Richard Szeliski Image Stitching 93

78 Matching Mistake: False Negative Moving objects: large areas of disagreement Richard Szeliski Image Stitching 94

79 Matching Mistakes Accidental alignment repeated / similar regions Failed alignments moving objects / parallax low overlap feature-less regions (more variety?) No 100% reliable algorithm? Richard Szeliski Image Stitching 95

80 How can we fix these? Tune the feature detector Tune the feature matcher (cost metric) Tune the RANSAC stage (motion model) Tune the verification stage Use higher-level knowledge e.g., typical camera motions Sounds like a big learning problem Need a large training/test data set (panoramas) Richard Szeliski Image Stitching 96

81 Image Blending

82 Image feathering Weight each image proportional to its distance from the edge (distance map [Danielsson, CVGIP 1980] 1. Generate weight map for each image 2. Sum up all of the weights and divide by sum: weights sum up to 1: w i = w i / ( i w i ) Richard Szeliski Image Stitching 98

83 Image Feathering Richard Szeliski Image Stitching 99

84 Feathering = 1 0 Richard Szeliski Image Stitching 100

85 Effect of window size 1 left 1 0 right 0 Richard Szeliski Image Stitching 101

86 Effect of window size Richard Szeliski Image Stitching 102

87 Good window size 1 0 Optimal window: smooth but not ghosted Doesn t always work... Richard Szeliski Image Stitching 103

88 Pyramid Blending Burt, P. J. and Adelson, E. H., A multiresolution spline with applications to image Richard Szeliski Image Stitching 104 mosaics, ACM Transactions on Graphics, 42(4), October 1983,

89 Laplacian level 4 Laplacian level 2 Laplacian level 0 Richard Szeliski Image Stitching 105 left pyramid right pyramid blended pyramid

90 Laplacian image blend 1. Compute Laplacian pyramid 2. Compute Gaussian pyramid on weight image (can put this in A channel) 3. Blend Laplacians using Gaussian blurred weights 4. Reconstruct the final image Q: How do we compute the original weights? A: For horizontal panorama, use mid-lines Q: How about for a general 3D panorama? Richard Szeliski Image Stitching 106

91 Weight selection (3D panorama) Idea: use original feather weights to select strongest contributing image Can be implemented using L- norm: (p = 10) w i = [w ip / ( i w ip )] 1/p Richard Szeliski Image Stitching 107

92 Poisson Image Editing Blend the gradients of the two images, then integrate [Perez et al, SIGGRAPH 2003] Richard Szeliski Image Stitching 108

93 De-Ghosting

94 Local alignment (deghosting) Use local optic flow to compensate for small motions [Shum & Szeliski, ICCV 98] Richard Szeliski Image Stitching 110

95 Local alignment (deghosting) Use local optic flow to compensate for radial distortion [Shum & Szeliski, ICCV 98] Richard Szeliski Image Stitching 111

96 Region-based de-ghosting Select only one image in regions-of-difference using weighted vertex cover [Uyttendaele et al., CVPR 01] Richard Szeliski Image Stitching 112

97 Region-based de-ghosting Select only one image in regions-of-difference using weighted vertex cover [Uyttendaele et al., CVPR 01] Richard Szeliski Image Stitching 113

98 Cutout-based de-ghosting Select only one image per output pixel, using spatial continuity Blend across seams using gradient continuity ( Poisson blending ) [Agarwala et al., SG 2004] Richard Szeliski Image Stitching 114

99 Cutout-based compositing Photomontage [Agarwala et al., SG 2004] Interactively blend different images: group portraits Richard Szeliski Image Stitching 115

100 PhotoMontage Technical details: use Graph Cuts to optimize seam placement Demo: Windows Live Photo Gallery Photo Fuse Richard Szeliski Image Stitching 116

101 Cutout-based compositing Photomontage [Agarwala et al., SG 2004] Interactively blend different images: focus settings Richard Szeliski Image Stitching 117

102 Cutout-based compositing Photomontage [Agarwala et al., SG 2004] Interactively blend different images: people s faces Richard Szeliski Image Stitching 118

103 More stitching possibilities Video stitching High dynamic range image stitching Flash + Non-Flash Video-based rendering Related lecture: Computational Photography Richard Szeliski Image Stitching 119

104 Other types of mosaics Can mosaic onto any surface if you know the geometry See NASA s Visible Earth project for some stunning earth mosaics Richard Szeliski Image Stitching 120

105 Slit images y-t slices of the video volume are known as slit images take a single column of pixels from each input image Richard Szeliski Image Stitching 121

106 Slit images: cyclographs Richard Szeliski Image Stitching 122

107 Slit images: photofinish Richard Szeliski Image Stitching 123

108 Final thought: What is a panorama? Tracking a subject Repeated (best) shots Multiple exposures Infer what photographer wants? Richard Szeliski Image Stitching 124

109 ( Questions ) [ The End ]