GLOBAL AND LOCAL HIGHLIGHT ANALYSIS IN COLOR IMAGES

Transcription

1 GLOBAL AND LOCAL HIGHLIGHT ANALYSIS IN COLOR IMAGES Karsten SCHLÜNS 1 and Andreas KOSCHAN 2 1: Humboldt University Berlin University Hospital Charité Institute of Pathology Berlin GERMANY karsten.schluens@charite.de 2: Technical University Berlin Institute for Technical Informatics FR 3-3, Franklinstr Berlin GERMANY koschan@cs.tu-berlin.de ABSTRACT In this paper we present two approaches to highlight removal in color images. The first approach is based on a global analysis of a single color image. Threedimensional color descriptors are transformed to a pair of two-dimensional descriptors. The second approach is based on a local analysis of three color images. The images are taken in analogy to photometritereo tasks. For every surface point in the scene there exist three color vectors which are stored in the three images. The matte reflection component is calculated by analyzing exclusively these three vectors. Both approaches in this paper are based on the dichromatic reflection model for inhomogeneous dielectric materials. Keywords: Highlight detection, highlight elimination, color image processing. 1. INTRODUCTION A common constraint in digital image processing is that a scene consists only of matte (Lambertian) surfaces. Employing this constraint reduces the applicability of the algorithms because our environment consists also of specular surfaces. Often highlights are misinterpreted as light objects or light patterns on surfaces. Therefore, the results of an image segmentation or the results of a correspondence analysis in stereo images may be falsified significantly. Digital color image processing provides the new chance to detect and to remove highlights in images. In this paper we present two approaches to highlight removal in color images. The first approach is based on a global analysis of a single color image. Threedimensional color descriptors are transformed to a pair of two-dimensional descriptors. This representation allows to perform the reflection analysis for multi-colored objects in a fast and robust manner. The second approach is based on a local analysis of three color images. The images are taken in analogy to photometric stereo tasks. For every surface point in the scene there exist three color vectors which are stored in the three images. The matte reflection component is calculated by analyzing exclusively these three vectors. Furthermore, we discuss three other approaches to highlight analysis and we compare these techniques to our techniques. In the following section we discuss global highlight analysis using a single image. Later we discuss local highlight analysis using three images. All approaches to highlight analysis that are discussed in this paper are based on the dichromatic reflection model for inhomogeneous dielectric materials [2]. 2. GLOBAL HIGHLIGHT ANALYSIS USING A SINGLE IMAGE Several techniques for the analysis of highlights in a single image have been published (see for example [2, 5]). These techniques analyze color clusters in a threedimensional color space and are time consuming. We call these techniques "global analysis techniques" because the entire image has to be considered for finding the color clusters. 2.1 Highlight analysis according to Klinker, Shafer, and Kanade Using the dichromatic reflection model for inhomogeneous dielectric materials Klinker, Shafer and Kanade [2] classify the physical events with the measured color variation in the image. Following the dichromatic reflection model the color signal of a pixel can be separated in C( x, = ms ( x, cs + mb( x, cb in RGB-space. The variables and c b denote the interface reflection color and the body reflection color, respectively. The scaling factors m s ( x, and m b ( x, which depend on the object surface normal n, the illumination direction s and the viewer direction v. All pixels corresponding to a common untextured surface material lie in the so-called dichromatic plane which is spanned by the two vectors and c b. The specfic shape of the cluster is due to the physical event and the

2 interaction between the interface and body reflection. A color cluster which for exa mple contains pixels belonging to a matte and a specular region looks like a skewed T or L in the color space. FIG. 1 shows a schematic plot of a T- shaped color cluster in RGB-space whereas FIG. 2 illustrates the real color cluster of an orange watering can under white illumination (see FIG. 5). blue c b green red FIG 1: Dichromatic plane of a curved, hybridly reflecting object showing one body color. blue red green FIG. 2: An L-shaped color cluster of a real, orange watering can (see FIG. 5) represented as color histogram (from [1]). By determining the position of a color in the color cluster with respect to the two base vectors and c b it is possible to smoothly classify a pixel to a matte or a specular region. For the generation of a matte image all color values associated with a specular region have to be projected onto the vector c b along the vector. The orienation of the vectors and cb in the original color space must be determined beforehand. It is important to note that for the highlight analysis only surfaces which show the same reflection properties (especially the same body reflection color) can be included in the highlight removal process at a time. If the object scence contains differently colored objects or object parts then their color clusters overlap in RGB-space. Therefore, a physics based segmentation method must be performed in advance. On the other hand, the segmentation results are also influenced by highlights and interreflections. As a conclusion both processes are highly interweaved and should control each other. 2.2 Highlight analysis according to Tong und Funt The physics based segmentation of the color image is one of the main difficulties in the highlight analysis according to Klinker, Shafer, and Kanade [2]. Errors or inaccuracies in the results of the segmentation have a direct influence on the highlight analysis. Tong and Funt [5] suggest a modification of this approach to highlight analysis. Instead of computing the vectors and c b and the scaling factors m s ( x, and m b ( x, for every segmented region, they combine the information about the regions and they initially estimate the vector. Due to the dichromatic reflection model the color of the interface reflection is identical to the color of illumination. Thus, it is identical for all regions. After a coarse segmentation of the image the corresponding dichromatic planes are computed for all single pixels. The vector is given by the intersection of all computed dichromatic planes. Instead of intersecting the planes directly, Tong and Funt find the line which is most parallel to all the planes. To find this line, the normal to each plane is calculated and then a least-squares fit is used to find the line closest to the perpendicular to all the plane normals. The basic idea of computing the vector for the entire image only once is rather efficient. However, the single dichromatic planes have to be separated and the corresponding normals have to be calculated when this technique is applied to a color image. Tong and Funt presented test images showing exclusively red, blue, yellow, and green objects which have sufficiently different colors. If a color image contained light red, dark red, and violet objects, the dichromatic planes could not be separated easily. Thus, no meaningful calculation of the normals is guaranteed in this case. 2.3 Highlight analysis based on twodimensional diagrams Both techniques for highlight analysis that are mentioned above need a costly analysis of the three-dimensional color space. We reduced the complexity of the task to the analysis of two-dimensional chromaticity diagrams of the image. We use the normalized u and v values of an (YUV) - space, the uv-chromaticity [4]. The (YUV) -space is given by

3 ( Y, U, V )' = ( R, G, B) Surfaces of ideal matte materials correspond to exact one point in the uv-space. Therefore, the dichromatic matte cluster corresponds to one point, too. As opposed to this, dichromatic highlight clusters form line segments in the uv-space. FIG. 3 illustrates the structure of dichromatic clusters in the uv-space and shows the clusters of one matte and of three hybrid (matte and specular) surfaces. Furthermore, we use an one-dimensional hue space (hspace) containing the number of pixels per angle α, where α is an angle in the uv-space between two lines. The h- space is transformed with a morphological filter. This data set is searched for maxima. For each maximum found in the h-space, a maximum in the uv-space is searched along the line corresponding to the actual angle. To reduce noise influence we combine this by seeking a frequency maximum. c b 1 c b 2 v c b 3 c b 4 FIG. 3: Representation of one matte and three hybrid surfaces in the uv-space. With the procedure described above we have determined the body reflection components. The next processing step is to segment the image physically to choose the right body reflection color for each pixel. To reach this, we only need to segment the h-space. This is straightforward because the h-space is onedimensional. Highlights are removed by setting the geometrical scaling factor of the interface reflection color to zero. FIG. 5 shows a watering can with a highlight (left) and the result of the highlight removal after employing our global analysis technique (right). u 3. HIGHLIGHT ANALYSIS USING THREE COLOR IMAGES The analysis of highlights can be simplified if more than one color image is utilized. Lee and Bajcsy [3] presented a technique for three trinocular stereo images. Our technique is applied to three photometric images. Both approaches to highlight analysis are detailed below. 3.1 "Spectral Differencing" Lee and Bajcsy [3] suggest a technique called "spectral differencing" for highlight analysis. They use three color images taken in analogy to a trinocular stereo approach. The objective of this technique is to detect highlights but not to remove them. For two color images taken under different viewer directions but with constant illumination direction spectral differencing denotes an algorithm which searches for a group of highlight pixels. The colors of these pixels share the property that they do not overlap with any other pixel in another image with respect to a three-dimensional color space (e.g., the RGB-space). The idea of the method is to detect viewer inconsistent object points. Therefore, images of so-called minimal spectral differences (MSD) are calculated. It is assumed that the illumination color and the color of the objects are different. The calculation of a MSD-image can be explained with help of an example. Let C α and C β be two color images of an object scence taken under different viewer directions. Let MSD ( Cα Cβ) denote the MSD-images calculated from C β to C α. The color value of a pixel in the MSD-image MSD ( Cα Cβ) is defined as the minimal value of all spectral differences between a pixel in image C α and all pixels in image C β. The spectral distance, i.e. the color distance, is defined as the distance of the colors in the three-dimensional color space. Every MSD-value which exceeds a predefined threshold belongs to a potential specular reflection pixel. The threshold only depends on the noise introduced by the imaging sensor. Its value can be set independently of the viewer direction. Pixels representing an object point which is invisible (occluded) in some of the images are also classified as highlights if the MSD-value is larger than the predefined threshold. The determination of an optimal threshold is a known problem. The advantage of this method is that no image segmentation is necessary. There are no assumptions on the number and properties of the light sources and the camera geometry. Lee and Bajcsy [3] could achieve good results for metal and inhomogeneous dielectriurfaces. Note, that this

4 method is only able to detect specular reflection but the generation of a matte image is not possible. 3.2 Local photometric highlight analysis We present an approach to remove highlights utilizing three color images taken in analogy to a photometric stereo approach. The three images are captured from a fixed camera position with three different illumination directions. We call these technique "local analysis technique" because it works locally on three pixels from three color images. c 1 image C 1 c 2 image C 2 c 1 c 2 c 3 image C 3 c 3 = c b FIG. 4: Principle of the local multiple image approach: Three images showing an object illuminated from different directions and the positions of the color vectors in the dichromatic plane when the color vector in image C 3 is not affected by the highlight. Our approach is based on the fact that for every surface point in the scene there exist three measured color values in the three images. For every pixel at location (x, there are three color vectors given: c 1 from image C 1, c 2 from image C 2 and c 3 from image C 3 (see FIG. 4). We assume that the illumination color is known and identical in all three images. Due to the dichromatic reflection model for inhomogeneous dielectric materials all three vectors c 1, c 2 and c 3 lie in the same dichromatic plane. The body reflection vector cb and the interface reflection vector have to be known to generate a matte image (analogous to [2]). We assume that the illumination directions differ sufficiently. Thus, at least one of the color vectors contains no specular reflection component. This vector is identical with the body reflection vector c b. The vector c i showing the largest angle to is the vector identical to c b because all three color vectors lie in the same dichromatic plane. We compute the three angles between and c i, i = 1,2,3 and search for the one with the largest value. In the example in FIG. 4 c 3 is the vector to be found. The hybrid vectors ( c 3 = cb in FIG. 4) are projected on the vector containing no specular reflection component ( c 3 = cb in FIG. 4) to generate matte images that can for example be used for surface recovery. This procedure enables a robust highlight removal process that can easily be implemented in parallel. FIG. 6 shows the results of this local approach for a softener bottle. 4. SUMMARY We have given an overview on three "classical" approaches to highlight analysis in color images. Furthermore, we presented an efficient global technique for the generation of matte images. This technique is limited to a two-dimensional analysis of color vectors of the image. In addition, a technique for a local photometric highlight analysis was shown. A robust elimination of highlights can be reached when this technique is applied to three color images. Both new approaches that have been presented in this paper have been applied successfully to synthetic and to real images. Acknowledgments We thank M. Teschner for implementing the global approach to highlight analysis. Furthermore, we thank the Technical University Berlin for providing the technical equipment. 5. REFERENCES [1] R. Klette, K. Schlüns, and A. Koschan, Computer Vision - Three-Dimensional Data from Images. Springer, Singapore, June [2] G.J. Klinker, S.A. Shafer, and T. Kanade, "A physical approach to color image understanding," International Journal of Computer Vision, Vol. 4, pp. 7-38, [3] S.W. Lee and R. Bajcsy, "Detection of specularity using color and multiple views," Proceedings of the 2nd European Conference on Computer Vision, pp , Santa Margherita Ligure, Italy, May [4] K. Schlüns and M. Teschner, "Analysis of 2d color spaces for highlight elimination in 3d shape reconstruction," Proceedings of the Asian Conference on Computer Vision, Vo l. II, pp , Singapore, December [5] F. Tong and B.V. Funt, "Specularity removal for shape from shading," Proceedings of the Vision Interface Conference 1988, pp , Edmonton, AB, Canada, 1988.

5 Appendix FIG. 5: A watering can with a highlight (left) and an image of the the same watering can after the highlight has been removed using our global approach (right). FIG. 6: A softener bottle with a highlight (left) and an image of the the same softener bottle after the highlight has been removed using our local approach (right).