Extracting Convex Groups Efficiently

Transcription

1 Chapter 7 Extracting Convex Groups Efficiently In the previous chapter, we mentioned that current contour extraction techniques are still unable to deal with unconstrained environments. We argued that the presence of clutter and texture can negatively affect the performance of grouping schemes to the point of rendering them impractical, and we also raised the issue of robustly evaluating the perceptual saliency of the groups extracted by a given algorithm. In this chapter, we present a framework that robustly and efficiently extract convex groups of line segments from images with significant clutter and texture. Our algorithm is based on a simple, Gestalt inspired measure of affinity between pairs of segments. We will show that while the absolute affinity values are difficult to work with, normalizing the affinity values locally for each segment greatly increases efficiency and robustness, and leads to consistent performance across line sets from different domains that have different levels of clutter and texture. We will demonstrate that our framework is capable of finding perceptually salient groups on cluttered images, while at the same time achieving a reduction of several orders of magnitude in the amount of search required to complete the task. We will also address the problem of evaluating the saliency of the extracted polygons, and we will show that the Qualitative Probabilities [48] framework can be used to obtain a robust measure of the quality of a group that does not rely on domain specific knowledge. Finally, 124

2 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 125 we will compare the results obtained with our algorithm against the results produced by an algorithm based on Jacobs boundary-coverage based search framework [47]. The search framework described here forms the basis of a general contour extraction algorithm that will be presented in the next chapter. The final goal of our perceptual grouping research is to demonstrate that a suitably constrained search-based grouping algorithm can go a long way toward achieving contour extraction (and hence, figure-ground segmentation) in unconstrained environments. 7.1 Clutter, Texture, and the Abundance of Convex Groups In his convex group extraction paper, Jacobs [47] demonstrated that convexity is a powerful constraint for perceptual grouping, and that coupled with proximity (in the form of a boundary coverage measure) it can lead to an efficient procedure for convex group detection. Jacobs work is based on the observation that on random line-sets, convexity is very rare. This means that a search procedure that explicitly looks for convex shapes will only have to examine a small subsets of all possible groups that can be formed on a random line set. Jacobs showed that as long as the number of convex groups in the image is small the search can be performed efficiently, and proposed the use of a threshold on boundary coverage as a means to limit the number of groups examined by the algorithm. While this works very well on images with little texture and clutter (and indeed, within particular domains, Jacobs algorithm has been successful as a component of a model-base object recognition framework), complex line-sets often become intractable. The main reason for this is that while convexity is rare in random line-sets, segments extracted from real world imagery are generally not random. In fact, it is often the case that most of the structure in the image is perversely aligned in such a way that the number of convex groups becomes combinatorially large. There are several reasons why the number of convex groups that can be formed in the

3 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 126 a) b) Figure 7.1: Common situations on images with clutter and texture. a) Texture aligned with object boundaries leads to large numbers of convex groups (over with boundary coverage > 95% for this image). b) Adjacent curves result in an even larger number of paths that lead to convex groups (over with coverage > 95% for this image). In either case, even a high threshold on boundary coverage is not sufficient to keep the search efficient. image can grow substantially: First, texture in artificial structures tends to be aligned with object boundaries (this is particularly problematic with text), which creates repetitive patterns in the line-set that result in a large number of paths that lead to closed, convex contours with a large boundary coverage. Secondly, adjacent curves in close proximity result in sequences of small, adjacent line segments. These segments can be grouped in combinatorially many ways to generate a contour, leading to an explosion in the number of groups a search algorithm would have to explore. Figure 7.1 shows examples of these situations. In order to maintain efficiency, a search method based on boundary coverage would have to use a threshold so large that it prunes out most of the possible groups before they re explored. This is impractical because it causes the algorithm to miss many perceptually salient groups due to small gaps along their boundary. The use of boundary coverage has one other disadvantage. Performance is highly dependent on the particular line-set being processed; a coverage threshold that works well on one image will fail in another, and there is no automatic way to estimate an appropriate threshold value. Borra and Sarkar [9] had already noted that

4 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 127 for Jacobs algorithm, small changes in boundary threshold result in large variations in performance, so threshold selection becomes a critical issue. We will show that on images like those shown on Figure 7.1, it is in fact not possible to select a threshold that will result in an efficient search and, at the same time, allow for most of the perceptually salient groups in the image to be extracted. With the above limitations in mind, we propose a framework that is capable of dealing with oriented texture, clutter, adjacent contours, and repetitive structure. Our framework is robust and performs consistently for a fixed set of input parameters, thus obviating the need for adjusting values manually for each image. We start by presenting the affinity measure we use to measure the quality of the junctions formed by pairs of line segments, then we will discuss why the absolute value of this affinity is not very informative, and propose a normalization scheme that determines locally which grouping choices look significantly better than the existing alternatives. We will explore the properties of the normalized affinity values, and show that a threshold on normalized affinity can be expected to work equally well for line-sets with reasonable variation. Using these normalized affinities, we propose a set of constraints that can be used to guide group formation, and introduce a search control technique intended to keep the algorithm from repeatedly extracting the same contour combinatorially many times on images such as the one depicted in Fig 7.1b. We present grouping results on several images, and compare our algorithm against a boundary coverage based method. This comparison is in terms of run-time, the amount of search performed, and a visual evaluation of the results for each algorithm. 7.2 Pairwise Affinity Measure Just like Jacobs algorithm, our convex group extraction method consists of a constrained search procedure. Contours are formed by growing an initial group (a single segment from the line-set) one segment at a time until a closed group is detected, subject to a set of con-

5 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 128 straints that are designed to reduce as much as possible the number of combinations that must be tested without getting rid of combinations that may lead to perceptually salient groups. The first step toward choosing which combinations of segments are worth exploring is to develop a measure of affinity between pairs of lines. The value of this affinity gives an estimate of the quality of the junction formed by the two segments in terms of general perceptual principles. For convex group extraction, we use a simple affinity measure based on proximity and the geometric configuration of the segments. Figure 7.2 shows typical junctions we expect to find in a line-set. The junction is characterized by the intersection point of the two segments, the distance between the intersection point and the closest endpoint of each segment (and whether this distance corresponds to a gap, or to a tail -which occurs when the intersection point splits the line in two parts-), and the estimated uncertainty about the true location of the intersection (shown as a gray shaded region in Fig. 7.2). In a perfect junction both segments meet exactly at the intersection point, so we design our affinity measure such that whenever there is a gap or tail, the affinity between the segments decreases. The effect of a gap on the affinity measure is calculated using a Gaussian function DF gap = e d2 /(2σ 2 gap ), (7.1) where d is the length of the gap, and σ gap is related to the size in pixels of the gaps we expect to occur due to imperfections in the line extraction process. We use a similar function to calculate the effect of tails; however, since we expect that segments that correspond to object boundaries will in general not be split, we make the affinity decrease faster for tails than it does for gaps DF tail = e d2 /(2σ 2 tail ), (7.2) where σ tail = σ gap /2. For each segment in a junction, we compute the appropriate distanceaffinity term DF gap or DF tail. We know that the line endpoints returned by the line-fitting process constitute a noisy estimate of the actual image location of the segment s endpoint. The line fitting process is affected

6 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 129 Figure 7.2: Common types of intersections between pairs of segments. A gap occurs when the intersection point occurs away from the line segments (see for example L1, and L2). A tail occurs when the intersection point cuts a line in two (see L3). The affinity measure for a pair of lines considers the length of the gap (or tail) from each segment to the intersection point, as well as the size of the shaded region surrounding the intersection (which indicates uncertainty about the location of the intersection, see text).

7 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 130 by noise, low contrast, and the choice of filter used to extract edges. For our purposes, this means that the segment s orientation, and hence the location of the intersection between two segments, are bound to be noisy as well. We account for this effect by noting that the location of the intersection point is more strongly affected if the endpoint of a segment is displaced a small distance β perpendicular to the segment s estimated direction (see Figure 7.3). If one endpoint is displaced while keeping the center of the segment fixed, we obtain a cone that has width 2β at the segment s endpoint. We expect the true orientation of the segment to lie anywhere within this cone, and use the width of the cone at the intersection point as a measure of uncertainty about the location of the intersection. The width of the cone grows faster for smaller segments, which reflects the notion that we are more confident about the orientation of longer segments, this is also illustrated in Figure 7.3. The effect of intersection point localization uncertainty is evaluated with another Gaussian UF = e w2 /(σ u) 2, where w is the width of the cone at the intersection point, and σ u is a suitable constant. In practice, σ u is set so that the affinity value for long segments is only affected weakly, while the affinities for smaller segments decrease fast as the distance to the intersection point grows. This means that everything else being equal, our affinity measure will favour grouping with longer segments. The geometric affinity between two segments l1 and l2 is then calculated using the distance and uncertainty terms for each segment G affinity(l1, l2) = (DF l1 UF l1 DF l2 UF l2 ). (7.3) Given the definitions above, this measure favours grouping longer segments that are close to each other in the image. We will make two important observations here: First, using the above affinity definition, the affinity value for the best junction that can be formed with a particular segment may range over several orders of magnitude. This means that a threshold on geometric affinity would be of little use. Second, for bad junctions, very small changes in geometric configuration cause large relative changes in the resulting affinity. This means that below a certain point, the magnitude

8 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 131 Figure 7.3: Segment orientation uncertainty, we assume that the true location of the line s endpoint is within a circle of radius β around the measured endpoint coordinates; the largest error occurs when the endpoint lies β pixels away, perpendicular to the line. The cone extending from the line represents the effect of displacing the endpoint β pixels in either direction, and the uncertainty w at a given distance d can be calculated given β, d, and the length of the segment. Uncertainty is greater for smaller lines.

9 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 132 of the affinity values becomes uninformative. To address the second problem, we add a uniform term to Eq.(7.3) and compute a final affinity value for a each junction T affinity(l1, l2) = G affinity(l1, l2) + κ, (7.4) where κ is a suitably chosen constant that puts a lower bound on the affinity score that can be assigned to any junction. This lower bound is important because it corresponds to the notion that all junctions whose geometric affinity is reasonably small are considered equally bad for grouping purposes. The uniform term has an important effect on the distribution of normalized affinities, we will return to this point in the next section. 7.3 Normalized Affinities We have defined a suitable measure of affinity between pairs of segments and we can use it to rank junctions during the search for convex groups. However, as mentioned above, the affinities themselves will vary over several orders of magnitude depending on the local characteristics of the line-set. To gain robustness against variations in the line-set, and against variations across line-sets from different images and different domains, we base our search algorithm on locally normalized affinity values. For each segment l1 in the line-set, we find the set K that contains the k segments with the largest affinity values toward l1 given by Eq. 7.4, and compute a normalized affinity for each junction in this set N affinity(l1, l2) = T affinity(l1, l2) for all l2 K. (7.5) T affinity(l1, l2), l2 K The normalized affinity values are constrained to be within (0, 1), and the distribution of normalized affinities over the image has the shape illustrated in Figure 7.4. The histogram shows a red line at the normalized affinity value of 1/k. This value is significant because of the way the original affinities were computed. Consider the situation

10 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 133 Normalized Affinity Histogram Frequency a) Normalized Affinity b) Figure 7.4: Typical distribution of normalized affinities. a) Original line-set, b) Histogram of normalized affinities. We used σ gap = 20, σ u = 15, κ =.25, and k = 20 (see the text for a discussion on the choice of these values). The red line corresponds to a normalized affinity of 1/k. illustrated in Figure 7.5, which shows a set of lines that may correspond to clutter, texture, or other image artifacts. The geometric affinities for junctions in this line-set will generally be low since they correspond to accidental intersections of otherwise unrelated segments. This means that the absolute affinity for most junctions will be close to κ, the uniform component of our affinity measure. Since the absolute affinities are similar, if we pick one segment and normalize its best k affinities, the resulting normalized affinities will be close to 1/k. This is an important property, and tells us that junctions with normalized affinity close to 1/k are indistinguishable from accidental junctions. The histogram in Figure 7.4 clearly shows that typically there will be a few junctions with affinity greater than 1/k while most junctions that can be formed are below the 1/k threshold. A similar reasoning indicates that junctions with affinity greater than 1/k are measurably better than the remaining alternatives, as illustrated in Figure 7.6. In this form, normalized affinities give us a robust ranking of the grouping choices available for each line, relative to the local characteristics of the line-set around that segment.

11 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 134 Figure 7.5: Random line-set, consider the segment shown in blue. There is no clear continuation for the segment since junctions formed with other lines are accidental. The geometric affinities for most of these junctions will be low, and the corresponding absolute affinities will be close to κ indicating that we have no reason to consider one of these junctions to be better than any other.

12 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 135 Figure 7.6: Typical neighborhood of a line segment (blue). Some lines offer good grouping choices relative to other possible junctions, and receive normalized affinities greater than 1/k (dark green). Other junctions are of dubious quality (they don t offer a clear advantage over other existing choices) and get affinities close to 1/k (red). Yet other junctions look bad relative to other available choices; these lines have normalized affinities smaller than 1/k (brown). Typically, few lines are above the 1/k threshold for any given segment.

13 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 136 Average Histogram for Normalized Affinities Frequency Normalized Affinity 0.2 Figure 7.7: Average histogram of normalized affinities computed over 20 test images with the same parameters used for the histogram shown in Fig This histogram indicates that the distribution of normalized affinities has the same shape regardless of variations between different line-sets. The red line corresponds to a normalized affinity of 1/k, with k = 20. Normalized affinities also have the appealing property of being robust to variations across line-sets from different images. Figure 7.7 shows an average histogram of normalized affinities computed using 20 test images from different domains. The images themselves vary significantly in complexity, but the resulting distributions of normalized affinities have the same typical shape. A final point worth making here is that the distribution of normalized affinities provides information about the discriminative power of the affinity measure. An affinity measure that provides ample separation between good and bad groups, and has a suitably chosen uniform component (κ in our formulation) will yield a histogram that peaks below 1/k and has a long tail beyond this value. If either of these conditions is not satisfied, the resulting normalized affinities will lack the properties mentioned above. Figure 7.8 shows normalized affinity his-

14 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 137 Normalized Affinity Histogram 4000 Normalized Affinity Histogram Frequency Frequency Normalized Affinity a) b) Normalized Affinity Figure 7.8: Normalized affinity histograms for a) Uninformative affinity function for which we used the same form of the affinity measure but set σ gap = σ tail = σ u = 200. This yields similar affinities for all junctions. b) Affinity measure without uniform component, we used the same values for σ gap and σ u described in Fig. 7.4 but used κ = 0. In either case, the resulting distribution has none of the desired properties, and provides little useful information for grouping purposes. tograms for these situations. As expected, the normalized affinity histograms show none of the characteristics described above. This result is important for two reasons, first it allows us to validate the choice of parameters used in the affinity function, and second, it tells us what properties alternative affinity functions must have so that the corresponding normalized affinities have the desired properties and can be used within our search framework. This will become important when we extend our framework to use other cues for evaluating pairwise affinities in the next chapter. Our search algorithm uses a threshold τ on normalized affinities, together with the convexity constraint, to efficiently extract salient groups of lines. To select an appropriate threshold, we use the knowledge that normalized affinities close to 1/k indicate junctions that are indistinguishable from accidental intersections. We wish to avoid any such junctions, so we must set a threshold that is greater that 1/k. Since the normalized affinities are bounded from above

15 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 138 at 1, we have a small range of possible values that the threshold can take. Furthermore, we note that most of the normalized affinity values greater than 1/k are within a small distance of the 1/k value. We have found experimentally that threshold values within [1.1/k, 1.5/k] result in an efficient search algorithm, and at the same time, keep most of the interesting groups from being pruned during search. Given that the distribution of normalized affinities has the same shape regardless of the characteristics of the line-set; we expect that a particular threshold τ will work equally well for different line-sets. We will show in our experimental results that for line-sets with reasonable amounts of variation this is indeed the case, and that the use of normalized affinities allows us to define a unique set of parameters that results in efficient convex group extraction on all of our test cases. With these considerations in mind, we selected appropriate values for the 4 parameters in our affinity measure; the values were selected to yield a distribution of normalized affinities with a long tail beyond 1/k and peaking well below this value. The parameter values we will be using from this point on are σ gap = 20 (which sets σ tail = σ gap /2 = 10), σ u = 15, and κ =.25. Additionally, we use k = 20 in all our tests, this is large enough to generate a set of candidate junctions that contains any interesting groups a given segment may create. 7.4 Constrained Search Using Normalized Affinities The search algorithm we propose takes the form of a depth-first search. It examines each segment in the line-set in turn, and tries to extract as many (distinct and plausible) convex groups as possible from that initial group. At each level of the search, the algorithm attempts to add one segment to the current group subject to the following conditions: The normalized affinity between the last segment added to the group and the new edge must be greater that the threshold τ, the resulting group must be convex, and the resulting shape must be simple (i.e. the contour must not be self-intersecting, and must not correspond to a spiral shape). Convexity is

16 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 139 Figure 7.9: Adjacent curves leading to combinatorially many convex groups. Assume that we have found chain a-b-c-d-e-f-g-h, and that it leads to a closed polygon. We want to prune variations of this chain such as a-i-c-d..., a-i-j-e..., a-b-c-k-f..., and so on. These chains lead to a similar shape, and the number of possible combinations grows exponentially with the number of segments along the adjacent curves. enforced by checking that when traveling along the contour clockwise from the initial segment, every edge in the contour makes a right-hand turn with respect to the preceding segment. The search continues adding edges to the group until a closed shape has been found, there are no more segments to try, or the resulting group is either non-convex, self-intersecting, or spiral. Upon encountering any of these conditions, the search algorithm backtracks to the previous level of the search tree. The condition for closure is that the first and last segments of the group must intersect, and their normalized affinity must be above the threshold τ. The above procedure is enough to yield an efficient search on most line-sets; however, we can improve performance on images such as the one shown in Figure 7.1b. On such line-sets, adjacent curves generate a combinatorially large number of paths that lead to the same shape (with slight variations), this is illustrated in Figure 7.9. To reduce this problem, we perform an additional check whenever a segment is being considered for addition to a group: If the new segment is covered by any closed contour found

17 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 140 starting from the same initial segment, the search procedure backtracks. A segment is covered if both of its endpoints are within a small distance (no more than a few pixels) from some edge in the closed contour. For example, in Figure 7.9, segments i, j, and k would be covered by the polygon a-b-c-d-e-f-g-h. Consequently, groups starting at a and including any of these segments would be pruned. This in effect forces partial chains of edges to introduce and subsequently maintain a reasonable difference with regard to previously extracted shapes in order to be explored. The assumption on which we base the above pruning condition is that if a partial chain is similar to some part of an already extracted polygon, a continued search from this chain is likely to lead to a small variation of the same convex group. This is supported by the observation that segments with similar length, orientation, and image location will have similar sets K of possible junctions, and the normalized affinities for these junctions will be comparable, leading to similar search paths. The above constraint on chain similarity yields a significant improvement on images with adjacent, curved contours. However, there are situations in which similar chains will lead to distinct shapes, and in such cases some of these shapes will be missed by the search started at a particular segment. We expect that any salient group missed in this way will be detected when the search procedure starts at one of the segments that makes the group distinct. There is one final addition to the algorithm to improve performance. Since we expect that most salient groups to be discovered early in the search (their junctions are likely to have a high normalized affinity), we would like to limit the amount of searching that can be carried out starting at a particular initial segment i. We accomplish this by adding a small amount ɛ tau = to the threshold τ each time a new combination of edges starting from i is examined. The increment is small enough that its effect is not significant until the number of combinations tried for a particular initial segment has grown considerably. The effect of this variable threshold is to slowly raise the bar on the relative goodness a junction must have in order to be explored, the larger the number of combinations explored

18 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 141 at some point in the search, the better a junction must be in order for the algorithm to try it. Intuitively this indicates that after a certain effort has been invested in finding groups that start at a particular segment, we are unwilling to try additional combinations unless they are clearly much better than other possible choices. The threshold is reset to its initial value τ 0 whenever a new initial segment is considered. Given all the conditions mentioned above, we can summarize the convex group search algorithm as follows: 1 - For each segment i in the line-set, generate an initial group containing i, set τ = τ Find the set K with the best k junctions for the last segment in the group (sorted by decreasing normalized affinity). 3 - For each segment j in K If N affinity(i, j) < τ end loop, otherwise add j to the group. Check whether j is covered by a previously found group, if so, try the next j. Test for convexity and simplicity, if test fails try the next j. Check for closure, if the group is closed, compute its saliency and report it. Increase τ = τ + ɛ τ. Go to step Report the extracted polygons sorted in order of decreasing saliency. Saliency estimation is actually done in two steps, a partial saliency estimate is computed when a new closed contour is identified, and this estimate is used to insert the new polygon into an intermediate, sorted list of contours found for a particular segment. Once search is completed for that segment, a full estimate of saliency for the polygons in the intermediate list is computed, and the polygons are inserted into a final, sorted list of extracted contours. The intermediate list is then deleted. We describe the saliency estimation procedure next.

19 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY Evaluating the Saliency of Convex Groups The above search algorithm will typically yield a large number of convex groups on any given image. Some of these polygons correspond to salient image structure, while others result from clutter and texture. It is also typical that the same contour (or small variations of it) will be detected many times starting from different segments. We need a robust way to estimate the quality of the extracted groups so that we can report first the groups with the highest quality, and so that, in the presence of multiple versions of the same shape, we can keep only the best one. To identify the most promising groups, we use the Qualitative Probabilities framework described by Jepson and Mann in [48]. Qualitative Probabilities estimate the log-unnormalized posterior of a model given the image data log q(m I) = log (p(i M)p(M)), (7.6) where p(m) is the prior probability for the model, and p(i M) is the likelihood of the image given that model. Jepson and Mann show that the prior probabilities for lines and objects composed of lines can be specified in terms of a small constant ɛ 1 that gets smaller as image resolution increases. In particular, the probability of finding a line endpoint or polygon vertex at a specified location in the image is of order O(ɛ 2 ). Since a line is specified by a pair of endpoints, the prior probability of specifying a particular segment is of order O(ɛ 2 ɛ 2 ) = O(ɛ 4 ), and the prior probability of specifying a polygon with t vertices is of order O(ɛ 2 t ). This corresponds to the term p(m) for a convex group in equation 7.6. In a similar way, the prior probability of specifying a set of n independent lines is of order O(ɛ 4 n ). Under QP, the prior probability for a complete image is given by the multiplication of the prior probabilities for all the hypothesized groups, and any remaining lines that are not part of these polygons. Since ɛ 1, the larger the order of epsilon (the more endpoints, polygon vertices, and lines that need be specified), the smaller the corresponding probability (at least in the limit as ɛ 0). A group increases its posterior probability by decreasing the total number of parameters

20 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 143 that are required to account for the observed image data. It does so by offering a cheaper explanation for the lines that compose its boundary. However, in general the match between the model and the observed lines is imperfect; any gaps or tails occurring along the boundary must also be accounted for. The term p(i M) in equation 7.6 incorporates these effects. Figure 7.10 shows part of a polygon s boundary under different situations of line-set support, and gives a detailed example of how the term p(i M) is evaluated. The epsilon order of the likelihood term depends on the number of free parameters that must be specified to account for the observed gaps and tails in a particular polygon. For our grouping method, the QP framework was extended so that the term p(i M) also accounts for lines that terminate at the boundary of a hypothetical group, as well as lines that are split, and lines that completely cross over the proposed polygon. This allows us to incorporate more evidence into the QP calculation. Figure 7.11 shows several possible situations, and the associated value of p(i M) in terms of ɛ. Intuitively, p(i M) is large for a polygon that has few gaps along the boundary, receives evidential support from lines terminating at its boundary, and doesn t split segments. The above considerations can be used to efficiently estimate q(m I) for any polygon. With this framework, we can robustly evaluate the quality of any extracted groups. For each initial segment i, we keep a temporary list of extracted polygons starting at i, and use it to prune similar chains. The list is sorted in decreasing order of QP score q(m I). When a new group is found, its QP score is computed, and the polygon is compared for similarity against the members of the temporary list. Two polygons P a and P b are said to be similar if all the vertices of P a are within a small distance of an edge in P b, and vice-versa. If an equivalence is detected, the group with the largest estimated q(m I) is kept. Notice, however, that the full evaluation of p(i M) requires computing line intersections and performing some geometric processing. This would be too expensive if it were to be performed on every polygon that is found, instead, for the intermediate group list we consider in the likelihood term p(i M) only lines that lie along the polygon s boundary, as shown in

21 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 144 Figure This is no more expensive than checking the boundary for coverage. Once all possible distinct groups have been found for a particular initial segment, the contents of the intermediate group list are inserted into a final list of convex polygons, sorted by their asymptotic posterior probability. This time, however, the complete likelihood term including all interactions between the polygon and the line-set is calculated. Any polygon whose posterior is worse than O(ɛ 4n ) is discarded since at least as good an explanation for the same image is provided by a model that consists of n independent lines. The end result of the above procedure is that the final polygon list is significantly smaller than the original number of convex groups detected in the image. The list will contain only groups that are reasonably different from each other, and the polygons at the top of the list will be those that do a better job of explaining the observed line-set. We will show in the experimental results that the top polygons also tend to agree with perceptually salient structure. 7.6 Experimental Results In this section we show the results of convex group extraction using the search algorithm we have presented in this chapter. We compare the results obtained on each of our test images with the groups that are extracted by an algorithm based on Jacobs formulation [47]. We use the same search engine for both algorithms and only change the pruning conditions outlined above; namely, our pruning constraints are replaced by Jacobs threshold on boundary coverage. The convexity test and geometric consistency tests are identical, but for the boundary based method, saliency is given by the coverage fraction (as in Jacobs [47]) instead of Qualitative Probabilities. For the boundary coverage algorithm, polygons are sorted by decreasing order of boundary coverage as specified in [47], but there is one difference between Jacobs original formulation and the implementation we use here. The original algorithm only allows for convex turns of 90 degrees or less. We have no such constraint as we have found that on many common

22 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 145 situations objects of interest will have sharper bends (consider for example the corners of a rectangle under perspective projection). Since our algorithm doesn t use smooth continuation as a grouping constraint, we do not believe this biases the search results in our favour, it only means that both algorithms will in general carry out more searching than they would if the small turn constraint was enforced. For all our test images, we used the following set of parameters: σ gap = 20, σ tail = σ gap /2 = 10, σ u = 15, κ =.25, k = 20, and the threshold on normalized affinity τ 0 = 1.5/k =.075. The results will show that this set of parameters consistently yields an efficient search while recovering most of the salient convex groups in each image. For the boundary-coverage based method, we varied the coverage threshold so as to reduce as much as possible the amount of search performed while at the same time capturing at least some of the groups extracted by our algorithm. It will become clear from our experimental results that on several of our line-sets there is no boundary-coverage threshold that will result in an efficient search without missing many of the perceptually salient groups in the image (see for example Figure 7.17). Another observation that can be made is that small changes in this threshold can cause large variations in the amount of search performed (see Figure 7.15c, and 7.15d). This agrees with the observations made by Borra and Sarkar [9] about the behavior of the coverage based algorithm. Figures show the convex groups extracted from several different images using our algorithm, as well as the polygons extracted with the boundary-coverage method. Each figure indicates the number of nodes expanded by each algorithm, the number of extracted polygons (before removal of repeated shapes), the total number of distinct shapes identified, and the run-time of each algorithm on a 1.9GHz Pentium IV machine. Some of these results merit further discussion. Notice the extended run times for the coverage based algorithm on the images shown in Figures 7.12, 7.15, and Notice as well that on each image this algorithm extracts several million convex groups, but only a handful of these correspond to distinct shapes. The large number of extracted groups is mostly the re-

23 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 146 Image nlines searched found distinct runtime (hh:mm:ss) Fruits :00:56 Boxes , :00:13 Boxes :00:13 Cells 1, , :02:20 Eggs , :00:44 Bowls , :02:25 Table 7.1: Summarized search results for the images shown in Figs using our search framework. The columns correspond to image name, number of input lines, number of nodes searched, number of models found, number of distinct shapes found, and run-time on a 1.9GHz Pentium IV machine. sult of the adjacent curves that make up the contours of several of the objects in these images. Without a way to determine that a partial chain has been encountered before, an algorithm will find the same shape an exponentially large number of times. This, together with the abundance of suitable junctions with high coverage (especially in Fig. 7.15) results in unacceptably slow search performance. Tables 7.1 and 7.2 summarize the results for each algorithm. In contrast, our algorithm finds a significantly smaller number of convex groups, but the actual number of distinct shapes detected is significantly larger. More importantly, it recovers many perceptually salient groups that are missed by the coverage based algorithm. This indicates that our technique for pruning similar chains achieves its goal of substantially speedingup the search without any noticeable loss in salient structure detection. All of this is achieved together with a large and consistent reduction in search complexity and run time (up to 4 orders of magnitude on the image shown in Fig which to a human observer looks deceptively simple). Finally, we can see that the same set of parameters will result in good search performance even though the line-sets show significant variability. In particular, the same threshold on normalized affinities τ 0 works well on all images, and doesn t

24 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 147 Image nlines searched found distinct runtime (hh:mm:ss) Fruits :42:59 Boxes , :13:38 Boxes , :58:24 Cells 1, :39:35 Eggs :17:10 Bowls :00:42 Table 7.2: Summarized search results for the images shown in Figs using the boundary-coverage based method. The columns correspond to image name, number of input lines, number of nodes searched, number of models found, number of distinct shapes found, and run-time on a 1.9GHz Pentium IV machine. need to be adjusted by a human observer. Figures , and 7.19 illustrate the robustness of the Qualitative Probabilities framework for convex group ranking. Each figure shows all the groups reported for the image, as well as individual groups that correspond to interesting image structure along with their final rank. This shows that the QP framework is indeed capable of robustly evaluating the quality of a particular polygon, in all of our test images, we find that groups that correspond to interesting image structure are usually found at the top of the list (within the first 15 polygons reported). We conclude from this that the normalized affinities framework is robust, and allows our algorithm to select the best grouping choices with regard to the local context of a particular segment. Our search algorithm based on normalized affinities, and enhanced with a technique for detecting and pruning similar chains, achieves the goal of extracting convex groups that correspond to salient image structure efficiently, even for images that could not be processed with previous algorithms in a practical manner. At the same time, we have shown that Qualitative Probabilities can be used to obtain a robust estimate of the quality of any group generated 1 Original image courtesy of Prof. Darl R. Swartz, Indiana University School of Medicine Histology Collection.

25 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 148 by the algorithm. The constrained search framework we have presented here forms the basis of a general contour extraction procedure we will describe in the next chapter. We will show that the convexity constraint can be replaced by other Gestalt principles while maintaining search performance, and examine the nature of the contours extracted. We will then propose extensions to the search framework that allow for the incorporation of different image cues, and re-visit the topic of evaluating and ranking of extracted contours in light of the availability of additional image information.

26 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 149 Figure 7.10: Fragment of a polygon boundary (gray) and different coverage situations: a) The polygon s vertices explain the location of both line endpoints, no extra parameters are required, so the likelihood is O(1). b) There is one break, its location can be explained with one free parameter (it s distance along the edge), so its likelihood is O(ɛ 1 ). c) Three breaks, the likelihood is O(ɛ 3 ). d) There are two breaks (O(ɛ 2 )), and one tail. The tail is nothing more than a segment which happens to be aligned with the polygon boundary, it requires 3 parameters: 2 for the endpoint we can see, and 1 for the other endpoint which lies somewhere (its exact location is unknown) along the polygon boundary. The second endpoint requires only one parameter since we we know its direction. The combined likelihood in this case is O(ɛ 2 ɛ 3 ) = O(ɛ 5 ).

27 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 150 Figure 7.11: Typical polygon interactions with the line-set. The polygon is treated as an opaque surface. L1, L2, and L3 are not affected by the polygon, each requires 4 parameters. L4 and L7 are treated as lines that are painted onto the polygon, and terminate at the boundary (within some tolerance), whereas L5 and L6 are outside segments obscured by the polygon (the location of their second endpoint is unknown). In both cases, we require 3 parameters to specify the visible part of each line: 2 for the free endpoint, and 1 to specify the location where the segment meets the polygon boundary (as a length along the appropriate polygon edge), hence the likelihood is O(ɛ 3 ). L8 is split, it is treated as 2 lines terminating at the boundary, one inside, and one outside the polygon, so the split requires 6 parameters and the likelihood is O(ɛ 6 ). L9 is split in 3 parts, each outside segment is treated as a line obscured by the polygon, and requires 3 parameters. We also need 2 parameters to specify the inside fragment (2 distances, one along each polygon edge), thus L9 requires 8 parameters given this particular polygon, and the likelihood is O(ɛ 8 ).

28 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 151 a) b) c) Figure 7.12: a) Fruits line-set, 797 line-segments. b) Best 25 polygons extracted by our algorithm, the search took 56 sec., and the algorithm expanded nodes. It found 852 polygons out of which 78 were distinct. c) All polygons extracted by the coverage based method with a threshold of.95, the search took over 42 hours, and the algorithm expanded over nodes. It found over 29 million polygons out of which 19 were distinct.

29 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 152 a) b) c) Figure 7.13: a) Boxes1 line-set, 670 line-segments. b) Best 30 polygons extracted by our algorithm, the search took 13 sec., and the algorithm expanded nodes. It found 1,027 polygons out of which 170 were distinct. c) Best 30 polygons extracted by the coverage based method with a threshold of.9, the search took 13 min, 38 sec, and the algorithm expanded nodes. It found 50,669 polygons out of which 70 were distinct.

30 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 153 a) b) c) Figure 7.14: a) Boxes2 line-set, 835 line-segments. b) Best 20 polygons extracted by our algorithm, the search took 13 sec., and the algorithm expanded nodes. It found 849 polygons out of which 130 were distinct. c) Best 30 polygons extracted by the coverage based method with a threshold of.9, the search took 58 min, 24 sec, and the algorithm expanded over nodes. It found 89,553 polygons out of which 90 were distinct.

31 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 154 a) b) c) d) Figure 7.15: a) Cells line-set, 1,621 line-segments. b) Best 25 polygons extracted by our algorithm, the search took 2 min, 20 sec., and the algorithm expanded nodes. It found 2,735 polygons, more than 200 of these were distinct shapes. c) All polygons extracted by the coverage based method with a threshold of.875, the search took over 23 hours, and the algorithm expanded over nodes. It found over 19 million polygons out of which 12 were distinct. d) All polygons extracted by the coverage based method with a threshold of.9, run time was 1 hour, 17 min., and the algorithm expanded over nodes. It found 198,018 polygons out of which 7 correspond to distinct shapes.

32 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 155 a) b) c) Figure 7.16: a) Eggs line-set, 863 line-segments. b) Best 25 polygons extracted by our algorithm, the search took 44 sec., and the algorithm expanded nodes. It found 1,346 polygons out of which over 200 correspond to different shapes. c) All polygons extracted by the coverage based method with a threshold of.85, the search took 1 hour, 18 min., and the algorithm expanded over nodes. It found over 1 million polygons out of which 23 were distinct.

33 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 156 a) b) c) d) Figure 7.17: a) Bowls line-set, 499 line-segments. b) Best 15 polygons extracted by our algorithm, the search took 2 min, 25 sec., and the algorithm expanded nodes. It found 1,118 polygons, 23 of these were distinct shapes. c) All polygons extracted by the coverage based method with a threshold of.875, the search took over 143 hours, and the algorithm expanded over nodes. It found over 240 million polygons out of which 12 were distinct. d) All polygons extracted by the coverage based method with a threshold of.925, run time was 19 hours, 53 min., and the algorithm expanded over nodes. It found over 19 million polygons out of which 5 correspond to distinct shapes.

34 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 157 Original Image All groups Rank 1 Rank 2 Rank 4 Rank 5 Figure 7.18: Robustness of Qualitative Probabilities raking. The original image is shown along with all the groups extracted by the algorithm, notice the abundance of convex structure. Individual groups corresponding to a few of the salient structures in the image are shown along with their QP rank.

35 CHAPTER 7. EXTRACTING CONVEX GROUPS EFFICIENTLY 158 Original Image All groups Rank 1 Rank 2 Rank 3 Rank 4 Figure 7.19: Robustness of Qualitative Probabilities raking. The original image is shown along with all the groups extracted by the algorithm, again, there is abundant convex structure generated by texture aligned with object boundaries. Individual groups corresponding to a few of the salient structures in the image are shown along with their QP rank.