CLUSTERING SEGMENTATION

Size: px

Start display at page:

Download "CLUSTERING SEGMENTATION"

Leo Fleming
7 years ago
Views:

1 CLUSTERING SEGMENTATION The slides are from several sources through James Hays (Brown); Silvio Savarese (U. of Michigan); Bill Freeman and Antonio Torralba (MIT), including their own slides.

2 Basic ideas of grouping in human vision Gestalt properties Figure-ground discrimination Emergence Currently in computer vision we don't really take into account the top-down (from the "memory") grouping only the bottom-up part.

3 Gestalt properties Berlin School early 20th century A series of factors affect whether elements should be grouped together. Difficult to translate into algorithms.

4 Gestalt properties

5 Grouping cues: Similarity (color, teture, proimity)

6 Grouping cues: Common fate Image credit: Arthus-Bertrand (via F. Durand)

7 Grouping by occlusions Gestalt properties

8 Gestalt properties Grouping by occlusions Segmentation is a global process!

9 Grouping by invisible completions Gestalt properties

10 Figure-ground discrimination Grouping can be seen in terms of allocating some elements to a figure, some to ground. Can be based on local bottom-up cues or high level recognition.

11 Figure-ground discrimination

12 Emergence 1970s: R. C. James

13 Emergence 2000s: Bev Doolittle

14 Feature Space Every token is identified by a set of salient visual characteristics. For eample: Position Color Teture Motion vector Size, orientation (if token is larger than a piel) Slide credit: Christopher Rasmussen

15 Feature Space Source: K. Grauman

16 Feature space: each token is represented by a point. b g r

17 Token similarity is thus measured by distance between points ( feature vectors ) in feature space. b g p q r

18 Cluster together tokens with high similarity. b g this is clustering... r

19 Image Segmentation as Clustering Goal: Break up the image into meaningful or perceptually similar regions.

20 Types of segmentations Oversegmentation Undersegmentation Multiple Segmentations

21 Why do we cluster? Summarizing large quantity of data in a few clusters with a few parameters. Each cluster can be used to predict if an unknown region becomes this cluster or not. Segmentation of an image (mainly those in 2D) can be achieved upto a decent estimate... usually.

22 Some Issues... How do we decide that two piels are likely to belong to the same region? How many regions are there?

23 Segmentation as clustering Cluster together tokens that share similar visual characteristics. Will eamine: K-mean Mean-shift Many other clustering methods eist, some using the graph structure of the image are also popular. But none of the clustering algorithm, as advanced as they are, can give a complete segmentation of the 3D reality in the 2D image.

24 K-nearest neighbor Take the closest K (say, L_2 norm) neighbors. o o + o o o o + o are the two clusters o + are two points...

25 with 1-nearest neighbor o o + o o o o + 2 o 1

26 with 3-nearest neighbor o o + o o o o + 2 o 1

27 with 5-nearest neighbor o o + o o o o + 2 o As the number of neighbors grow, the K-nearest points can come from both clusters too. 1

28 K-Means Clustering Initialization: Given K, number of clusters; N points in feature space. Pick K points randomly. These are initial cluster centers (means) 1,, K. Repeat the following: 1. Assign each of the N points, j, to clusters by nearest i. 2. Recompute mean i of each cluster from its member points 3. If no mean has changed more than some, stop. K objective function Slide credit: Christopher Rasmussen

29 K-means The number of clusters (three) in given by the user. Initialize cluster centers Assign points to clusters Recompute means Repeat previous two steps The clusters are Gaussian distributions with L_2. Illustration Source: wikipedia

30 the final result A simple clustering eample with three clusters.

31 There are many limitations of K-means. The density is different locally.

32 Non-gaussian shapes which will be not solved with many clusters......beside, the number of clusters have to be given be the user. Improvement eist, but nothing is significantly better.

33 K-Means pros and cons Pros Simple and fast. Converges to a local minimum of the error function. Cons Need to pick K. Sensitive to initialization. Only finds spherical clusters. Sensitive to outliers. adaptive?! Rarely used for segmentation.

34 Mean shift segmentation D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002, v.24, Versatile technique for clustering-based segmentation. Uses the L*u*v* color space which is also perceptually uniform. A nonlinear transformation from RGB.

35 Mean shift algorithm......try to find modes of this non-parametric density.

36 Kernel density estimation Kernel density estimation function K() > 0 only for <= 1 the bandwidth, h, has to be given by the user. The kernel is symmetric are depents on. The Epanachikov kernel ~(1 - ) and the truncated Gaussian kernel 2 2 are used.

37 Mean shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

38 Mean shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

39 Mean shift Region of interest Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

40 Mean shift Region of interest Center of mass Slide by Y. Ukrainitz & B. Sarel

41 Two synthetic Gaussian modes, but is not used in mean shift. The only parameter used is h. The two kernels converge to different modes in spite starting from overlapping regions.

42 Computing the Mean Shift 2 k( ) = K() g() = - k'() profile of the kernel gradient f() = 0 in each iteration Translate the kernel window by m() m( ) n i 1 n i 1 i g g - h h - i i 2 2 Slide by Y. Ukrainitz & B. Sarel g( ) k ( )

43 Modality Analysis Tessellate the space with windows. Merge windows that end up near the same mode (peak).

44 Attraction basin : the region for which all trajectories lead to the same mode all data points in the attraction basin of a mode Slide by Y. Ukrainitz & B. Sarel

45 Eample: attraction basins zero gradient, g()=0 but not a maimum stationary point eliminated by shifting a little bit the trajectory

46 A color piel is represented by a spatial and a range bandwidth. a two-dimensional spatial bandwidth: h a three-dimensional range bandwidth: h The user have to give only this two parameters. They are not very strict like in k-means. s r = 5 dimensions Gray level images have one-dimensional range bandwidth.

47 Eample of a window's convergence (8,4)

48 original (inverted) mean shift filtering segmentation

49 Mean shift clustering The mean shift algorithm seeks modes of the given set of points. 1. Choose kernel and two bandwidths. 2. For each point: a) Center a window on that point. b) Compute the mean of the data in the search window. c) Center the search window at the new mean location. d) Repeat (b,c) until convergence. 3. Assign points that lead to nearby modes to the same cluster. In segmentation, the means are both in spatial and range and the points always converge to the nearest mode.

50 Mean shift clustering are two phases: filtering, as was described before; segmentation, unify adjacent clusters if they are closer than h_s in the spatial domain and h_r in the range domain. (Step 3.) EDISON program can do additional things too, but we will not describe them here. Mean shift was also used for tracking of motions and for nonlinear spaces through Riemannian manifolds.

51 Mean shift segmentation results (16,7,40) Eliminate spatial regions containing less than M piels.

52 (8,7,20) The sky changes with location but is still segmented into one. Sometimes it is possible to take planar surfaces and represent it still as a constant surface.

53 parameters (8,7,20)

54 parameters (8,7,100)

55 Mean shift pros and cons Pros Does not assume spherical clusters. Just two parameters (window sizes). Finds variable number of modes, which are not given. Robust to outliers and weak nonconstant regions. Cons Output depends on window size. Efficient implementation uses on short cuts in the search. Does not scale well directly with dimension of feature space is above ten.

56 one of the best segmentation groups Berkeley Segmentation Engine

57 The goals of segmentation Group together similar-looking piels for efficiency of further processing. Bottom-up B process... at least most of it. Unsupervised... but humans have a memory too. superpiels Quasi-constant piels is not always enough... X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.

58 The human segmentation and image segmentation are strongly different. They are not the same... Separate image into coherent objects. Bottom-up or top-down process? Top-down is now very weak. Supervised or unsupervised? Supervised will increase top-down. image human segmentation Berkeley segmentation database: Segmentation without strong top-down ("memory") will not work in a lot of cases. This "memory" does not eist for the moment.

59 pb boundary detector pb = probability-of-boundary brightness: L color:a,b a lot of computations Martin, Fowlkes, Malik: Learning to Detect Natural Boundaries Using Local Brightness, Color, and Teture Cues. TPAMI Figures from Fowlkes

60 pb Boundary Detector intensity brightness color teture

61 Brightness Color Teture Combined Human

62 Using Contours to Detect and Localize Junctions in Natural Images" M. Maire, P. Arbelaez, C. Fowlkes, and J. Malik. CVPR D segmentation in probably not sufficient to do object recognition. probability of boundary global prob. boundary optimally thresholded

63 45 years of boundary detection TS/(TS+FS) (correct + unepected) human ground-truth boundaries True/False Segm./Ground TS/(TS+FG) (correct + missing) Source: Arbelaez, Maire, Fowlkes, and Malik. TPAMI 2011 (pdf)

64 This will be rather difficult to segment... with a machine.

Segmentation & Clustering

EECS 442 Computer vision Segmentation & Clustering Segmentation in human vision K-mean clustering Mean-shift Graph-cut Reading: Chapters 14 [FP] Some slides of this lectures are courtesy of prof F. Li,