What makes Big Visual Data hard?

Transcription

1 What makes Big Visual Data hard? Quint Buchholz Alexei (Alyosha) Efros Carnegie Mellon University

2 My Goals 1. To make you fall in love with Big Visual Data She is a fickle, coy mistress but holds the key to achieving real visual understanding 2. To ask for help in tackling this Big Interdisciplinary Problem

3 Driven by Visual Data Texture Synthesis Dating Historical Images Seeing Through Water Unsupervised Object Discovery Action Recognition Illumination Estimation Inferring 3D from 2D Geo-location

4 Texture: microcosm of Big Data radishes rocks yogurt

5 Texture Synthesis

6 Classical Texture Synthesis Synthesis Novel texture Parametric Texture Model This is hard! Analysis Sample texture

7 Throwing away too much too soon? input texture synthesized texture

8 Non-parametric Approach Synthesis Novel texture Analysis Sample texture

9 [Efros & Leung, 99, Efros & Freeman 01] p non-parametric sampling Input image

10 Texture Growing

11 Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Our algorithm

12 Two Kinds of Things in the World Navier-Stokes Equation + weather + location +

13 Lots of data available

14 Unreasonable Effectiveness of Data Parts of our world can be explained by elegant mathematics: physics, chemistry, astronomy, etc. But much cannot: [Halevy, Norvig, Pereira 2009] psychology, genetics, economics, visual understanding? Enter: The Magic of Data Great advances in several fields: e.g. speech recognition, machine translation, Google

15 The A.I. for the postmodern world

16 The Good News Really stupid algorithms + Lots of Data = Unreasonable Effectiveness

17

18 140 billion images 6 billion added monthly 6 billion images 1 billion images served daily 72 hours uploaded every minute 3.5 trillion photographs 90% of net traffic will be visual!

19 Physics Dating Drugs Scientific Experiments Psychology Social Graphs Collaborative Filters Genetics Web Text Medical Data Data Mining Search Disease Tracking Policy Economic Data Business Intelligence Business Data Marketing Visual Data?

20 Bad News Visual Data is difficult to handle text: clean, segmented, compact, 1D, indexable Visual data: Noisy, unsegmented, high entropy, 2D/3D

21 Computing distances is hard CLIME - CRIME = hamming distance of 1 letter y y x - x = Euclidian distance of 5 units - = Grayvalue distance of 50 values - =?

22 How similar are two pictures?? =

23

24 Medici Fountain, Paris

25

26

27 INDEXING VIA VISUAL WORDS

28 VISUAL WORD MATCHING [SIFT: Lowe, 2004]

29 letter VISUAL WORD MATCHING [SIFT: Lowe, 2004]

30 Medici Fountain, Paris (winter)

31

32

33

34

35

36 Visual Garbage Heap It irritated him that the dog of 3:14 in the afternoon, seen in profile, should be indicated by the same noun as the dog of 3:15, seen frontally My memory, sir, is like a garbage heap. -- from Funes the Memorious Jorge Luis Borges Organizing the Garbage Heap : Finding visual correspondences across data Mining Visual Data Connecting visual data to enable understanding (Visual Memex)

37 Improving Visual Correspondence

38 Improving Visual Correspondence

39 Lots of Tiny Images 80 million tiny images: a large dataset for nonparametric object and scene recognition Antonio Torralba, Rob Fergus and William T. Freeman. PAMI 2008.

40 Lots Of Images A. Torralba, R. Fergus, W.T.Freeman. PAMI 2008

41 Lots Of Images A. Torralba, R. Fergus, W.T.Freeman. PAMI 2008

42 Lots Of Images

43 Automatic Colorization Grayscale input High resolution Colorization of input using average A. Torralba, R. Fergus, W.T.Freeman. 2008

44 [Hays & Efros, SIGGRAPH 07]

45

46 Scene Descriptor

47 Scene Descriptor Scene Gist Descriptor (Oliva and Torralba 2001)

48 2 Million Flickr Images

49

50

51

52

53

54

55

56

57

58 200 scene matches

59

60

61 Improving Visual Correspondence

62 Improving Visual Correspondence

63 Visual Data has a Long Tail The rare is common!

64 LEARNING BETTER VISUAL CORRESPONDENCES ABHINAV SRIVASTAVA, TOMASZ MALISIEWICZ, ABHINAV GUPTA, ALEXEI EFROS SIGGRAPH ASIA 11

65

66 Input Query Top Matches

67 Input Query Top Matches

68 Input Query Top Matches

69 IMPORTANT PARTS? Input Query Important Parts

70 Input Query Top Matches

71 72

72 Way more efficient approaches: [Ramanan et al 2012, Durand et al 2012]

73 SEARCH USING PAINTINGS GIST Input Painting Bag-of-Words Tiny Images Our Approach HOG

74 SEARCH USING PAINTINGS Input Painting Top Matches

75 SEARCH USING PAINTINGS Input Painting Top Matches

76 SEARCH USING SKETCHES Tiny Images Input Sketch GIST Bag-of-Words Our Approach 81

77 SEARCH USING SKETCHES

78 APPLICATIONS

79 RE-PHOTOGRAPHY Computational Re-photography (Bae et al., 2010) Historical Image of Boston Station Re-photographed Image

80 RE-PHOTOGRAPHY Computational Re-photography (Bae et al., 2010) Historical Image of Boston Station Re-photographed Image Then & Now View

81 INTERNET RE-PHOTOGRAPHY Computational Re-photography (Bae et al., 2010) Historical Image of Boston Station Re-photographed Image Then & Now View Our Approach Search 10,000 Flickr Images of Boston Historical Image of Boston Station Top Match

82 INTERNET RE-PHOTOGRAPHY Computational Re-photography (Bae et al., 2010) Historical Image of Boston Station Re-photographed Image Our Approach Then & Now View Historical Image of Boston Station Top Match From 10,000 Flickr Images Then & Now View

83 WHERE WAS THE PAINTER STANDING? Input Painting

84 PAINTING2GPS Input Painting Retrieval set 10,000 Geo-tagged Flickr Images 100 top matches used to estimation

85 PAINTING2GPS Input Painting Estimated Geo-location Estimated using 100 top matches

86 VISUAL SCENE EXPLORATION

87 VISUAL SCENE EXPLORATION 96

88 Query image FINDING SIMILAR IMAGES

89 PAIRWISE SIMILARITY MATRIX

90 TRAVERSING THE GRAPH

91