An adaptive technique for shadow segmentation in high-resolution omnidirectional images

An adaptive technique for shadow segmentation in high-resolution omnidirectional images Vander Luis de Souza Freitas Univ Estadual Paulista - UNESP Undergraduate Student Scholarship PIBIC/CNPq Presidente Prudente, Brasil vandercomp@gmail.com Antonio Maria Garcia Tommaselli Univ Estadual Paulista - UNESP Department of Cartography Presidente Prudente, Brasil tomaseli@fct.unesp.br Abstract Segmentation of shadowed areas in digital images is an important step of several algorithms. One important drawback of existing algorithms is that shadows can be confused with parts of objects of a scene or even be considered as spurious elements. Dark objects are sometimes labeled as shadows, because of their similar radiometric properties. Both situations may reduce the outcomes of object detection algorithms, image matching and segmentation in general. The aim of this paper is to present an adaptive version of an existing algorithm, applying it to high-resolution omnidirectional terrestrial images. The efficacy was assessed by comparing the results with other techniques and with reference images. Keywords shadow segmentation, adaptive thresholding, omnidirectional images. I. INTRODUCTION Image processing algorithms are nowadays embedded in several automatic processing chains, like object detection, surveillance, and other problems. A common issue in object detection and recognition, and image matching, is the presence of shadowed areas, which can change the radiometric properties of an image or be confused as part of objects of a scene. There are two types of shadows: cast shadows and self shadows. Cast shadows are projected by objects of the scene. Self shadows are those projected on the object by itself, in areas where there are deficient illumination. Self shadows usually have higher intensity values than cast shadows, because they receive more secondary light from near objects. Umbra and penumbra are other two sub categories of shadows which must also be considered. Umbra represents a shadow region where the primary source of light is completely obscured. Penumbra is the region around the edge of a shadow where the light source is only partially obscured, or where secondary sources of light illuminate it with low intensity [1]. There are many techniques for shadow detection and removal, based on image characteristics like intensity, chromaticity, geometry and texture. The goal of this paper is to present an adaptive version of the technique proposed by Polidorio et al [2], using a modified co-occurrence matrix [3] and the thresholding technique of Rosenfeld and De La Torre [4], with some adaptations. Experiments were performed with high-resolution omnidirectional terrestrial images, taken by a digital camera with fisheye lens. The shadow labeling and enhancement on these images are part of a project for automatic detection of ground control points in aerial images [5]. The motivation is the problem of matching between terrestrial and aerial images, which is affected by the presence of shadows. Detection and segmentation of shadow areas are also of great value in many other problems of image processing, like motion detection, traffic monitoring and video surveillance [6]. In order to assess the results, comparisons will be done using the techniques developed by Santos et al [7] and Centeno and Pacheco [3]. II. RELATED PAPERS Polidorio et al [2] developed a technique to detect shadows in orbital and aerial images. A shadow quantifier index, which is computed using the intensity and saturation components in the HSI color space, was proposed and tested. The algorithm takes into account the low intensity and high saturation of shadow pixels, caused by a physical phenomenon of atmospheric dispersion of the sunlight, most well known as the Rayleigh scattering effect. This technique proved to be efficient with aerial and orbital images, but there are no mention to its use with terrestrial images. Dare [1] presented a technique for shadow removal from high-resolution satellite images from urban areas and performed experiments with Ikonos and Quickbird images. The algorithm starts with a thresholding, to separate the image in two groups of pixels. Then it is performed a region encoding, in order to identify each region in the image and calculate their attributes (size and location in image space). Finally, a region filtering is applied to separate shadow from falsely detected non-shadow regions, like water regions or other features that are not covered by shadows. Santos et al [7] created a mask to detect shadows in high-resolution aerial images, choosing pixels with less radiometric response as shadow pixels. Drew and Joze [8] and Finlayson et al [9] used the image formation theory to remove shadows from images. The first paper describes a technique that uses human interaction to identify regions in the image that corresponds to the same surface in and out shadows. Then they find the sharpening matrix to transform the original image, from which an

illumination invariant image is created by an entropy-minimization method. Tian et al [10] used an RGB attenuation model (Tricolor Attenuation Model TAM), considering the image formation theory. The process is entirely automatic and does not depend on additional information. Centeno and Pacheco [3] used a modified co-occurrence matrix, and an adaptive threshold, which is obtained through the frequency histogram of the principal axis of co-occurrence matrix. The technique was also developed to detect shadows in high-resolution satellite images. As Centeno and Pacheco [3], Devi et al [11] developed a method for shadow detection and removal from high-resolution satellite images. They performed experiments with CARTOSAT-2 images and panchromatic IKONOS satellite images, all from Indian territory. For shadow labeling, spectral and geometric property based techniques, region growing and filtering were used. The shadow removal step was done by luminosity recovering, gamma correction and histograms correspondence. Sanin et al [12] performed a comparative study among shadow detection techniques for video images. Intensity, chromaticity, geometry and texture based techniques have been analyzed. Experiments were performed using a database of indoor and outdoor images. A. Shadow segmentation considering the Rayleigh scattering effect When the sun rays reach the atmospheric layer of Earth, they suffer a dispersion effect called Rayleigh, which causes the blue effect on the sky. A direct consequence observed on images is the loss of contrast. Polidorio et al [2] developed a technique that takes into account this phenomenon to detect shadows in orbital and aerial images, considering two assumptions: Shadow areas have low luminance, because they do not receive a direct flow of luminous energy; Violet and blue wavelengths are more affected by atmospheric dispersion, therefore these colors are saturated; They used HSI color space to access intensity and saturation information, and then defined a shadow quantifier index (1). SD ij =I ij S ij (1) with SD ij being the matrix with the shadow quantifier indexes, I ij the intensity component of HSI color space and S ij the saturation one. After calculating the indexes, the thresholding process is applied, with the threshold value (k) depending on the sort of the input image (2). 0, aerial k={ (2) 0.2, orbital image} The threshold value is defined considering that the farther the sensor is from the Earth surface the larger is Rayleight scattering effect, and same way the saturation (S). In this case, (1) will result in lower values for farther sensors and larger values for closer ones. So k is inversely proportional to the distance between sensor and scene. Shadow areas are defined by (3). SDW (ij, k ) ={ 1, if SD ij k (3) 0, otherwise} An advantage of this technique is that it does not require additional data (DTM, Sun's position, background model, etc). However, it is limited to orbital and aerial cases, if the predefined thresholds were used. III. A TECHNIQUE FOR TERRESTRIAL IMAGES An adaptive version of the Polidorio et al [2] technique can be proposed to be used in high-resolution terrestrial images. In orbital and aerial cases, predefined thresholds were already suggested by the authors, but they are not suitable to terrestrial images. Experiments were performed with five high-resolution omnidirectional nadiral terrestrial images (3030 x 2036 pixels), obtained by a Fuji S3 Pro camera, equipped with Bower Fisheye lens and lifted up to 5 meters high with a telescopic rod. Public datasets of images were not used because this paper aims to segment shadows from omnidirectional images and these type of dataset were not available to the knowledge of the authors. The reference shadow maps of these five images were defined by digitizing them interactively on the screen, to be used as reference for the experiments. These shadow maps will be used to evaluate rates of success, false positives and false negatives, after shadows automatic detection. Firstly the input image was converted from RGB to HSI color space. The next step was the computation of the shadow quantifier indexes through (1), using the intensity (I) and saturation (S) components. These indexes contain negative values, so they were normalized to the interval [0, 255] in order to be stored as an 8-bit image (Fig. 2). After normalization the image texture was analyzed by calculating the co-occurrence matrix [13], and verifying the occurrences of values b just right placed from value a (both varying from 0 to 255). Considering that shadows have similar radiometric properties, just the primary diagonal of the co-occurrence matrix is used to estimate an adaptive threshold to be used with the technique proposed by Polidorio et al [2]. Then, a histogram of the values of this primary diagonal was created and an adapted version of the algorithm proposed by Rosenfeld and De La Torre [4] was used to find the threshold value, based on the histogram concavities (local minimum). This technique originally finds out a concavity in the histogram that splits it into two distinct regions, so that the first one, placed leftmost, belongs to shadow regions. The problem is that it just finds out the cast shadows areas with lowest intensities, and self shadows are not detected. This problem was partially solved by choosing the concavity amidst the second group of values in histogram, as can be seen in Fig. 1a. To achieve these values, the concavity that represents the beginning of this group must

be found. It is performed by looking for the concavity with ¼ of the size of that one related with the threshold computed by the original algorithm. Using the example of Fig. 1a, from the concavity that represents the value 65, it is performed a search for the next concavity with ¼ of its size. The algorithm runs normally from this point, in order to find out the new threshold value. The value ¼ was defined empirically, after analyzing the histogram behavior of the five images of the experiment. Fig. 1b and Fig. 1c show the results with the first and second threshold values. (a) (b) (c) 2. Computation of shadow quantifier indexes; 3. Normalize indexes into the interval [0, 255]; 4. Calculate the co-occurrence matrix; 5. Calculate the threshold value based on the frequency histogram of the primary diagonal of the co-occurrence matrix, using the adapted version of Rosenfeld and De La Torre algorithm [4]; 6. Compute (3) to determine which pixels belong to shadow regions. IV. EXPERIMENTS AND RESULTS Experiments were performed with the five images previously mentioned, with the sensor positioned about 4.6m above the terrain. Despite this distance being quite short, when compared to the original flight heights of the images used in the algorithm proposed by Polidorio et al [2] (orbital and aerial cases), the results are similar to those obtained with other techniques. In order to evaluate the results, some shadow maps were interactively draft on the screen. They were also used to assess two further techniques of shadow detection and to compare with that one proposed in this paper. Table I presents the threshold values obtained with the adaptive technique. TABLE I. Thresholds of the adaptive version of Polidorio et al [2] algorithm Images Threshold Image 1 0.032 Image 2 0.072 Image 3 0.015 Image 4 0.125 Image 5 0.243 After applying the algorithm developed by Polidorio et al [2], using these threshold values, the resulting images were compared with their reference shadow maps (Fig. 3). As it can be seen, there are many false positives located near the boundaries of shadow regions. This occurs because of the transition of illumination between shadow and non-shadow areas. Success rate (S) and false positives rate (FP) were calculated using (4) and (5) respectively. These equations were applied in the output images in order to assess the adaptive approach proposed in this paper and to compare its results with the techniques developed by Santos et al [7] and Centeno and Pacheco [3] (Table II). Fig. 1. (a) First threshold value (65) computed by the original algorithm [11]. Second threshold value (77) calculated by its adapted version; (b) Results of the Thresholding using 65 as threshold value (white pixels represents shadow areas); (c) Results of the Thresholding using 77 as threshold value. Having this value (k), (3) is applied to define the shadowed areas. The steps of the complete algorithm can be stated as: 1. RGB HSI color space conversion; S= DSp Sp with Dsp being the number of detected shadow pixels and Sp the real number of shadow pixels in the image. FP= ndsp nsp (4) (5)

with ndsp being the number of pixels labeled as non-shadow pixels and nsp the real number of non-shadow pixels in the image. The results achieved with the application of the adaptive technique are very similar of those obtained with Santos et al [7] algorithm. The technique developed by Centeno and Pacheco [3] detected less shadow pixels and the same way the false positives were lower. Fig. 2. Input images, shadow quantifier indexes normalized images, and detected shadows (white pixels). Images 1, 2, 3, 4 and 5 respectively.

Fig. 3. Outcomes of Images 1, 2 and 3 respectively: Green represents success; Red represents false positives; Blue represents false negatives. TABLE II. COMPARISON OF SUCCESS RATE (S) AND FALSE POSITIVE RATE (FP) OF THE SHADOW DETECTION ALGORITHMS Images Adaptive technique Santos et al [7] Centeno and Pacheco [3] S FP S FP S FP Image 1 0.89 0.06 0.93 0.16 0.65 0.00 Image 2 0.88 0.04 0.79 0.02 0.71 0.01 Image 3 0.77 0.13 0.82 0.08 0.59 0.00 Image 4 0.91 0.02 0.61 0.00 0.71 0.00 Image 5 0.99 0.09 0.45 0.00 0.85 0.01 V. CONCLUSIONS An adaptive version of technique developed by Polidorio et al [2] for shadow detection in high-resolution images was created successfully. This algorithm does not depend on additional information about the image acquisition process like Sun's position, distance between sensor and scene, etc. Although the technique considers Rayleigh scattering effect, that is most noticeable on orbital and aerial images, results have shown that it can be also used to detect shadows in terrestrial images. It is important to note that the images used in this paper were acquired with fish-eye lenses and the effects of the geometric distortion in the radiometry were not assessed yet.

More experiments will be performed using larger samples of images. ACKNOWLEDGMENTS This work was funded by PIBIC/CNPq (The National Council for Scientific and Technological Development, grant n. 800124/2012-0). REFERENCES [1] P. M. Dare, Shadow analysis in high-resolution satellite imagery of urban areas, Photogramm. Eng. Remote Sens., vol. 71, pp. 169 177, Feb. 2005. [2] A. M. Polidorio, F. C. Flores, N. N. Imai, A. M. G. Tommaselli, and C. Franco, Automatic shadow segmentation in aerial color images, in Computer Graphics and Image Processing, 2003. SIBGRAPI 2003. XVI Brazilian Symposium on, 2003, pp. 270 277. [3] A. Berveglieri, J. Marcato Junior, M. V. A. Moraes, and A. M. G. Tommaselli, Automatic location and measurement of ground control points based on vertical terrestrial images, in RSPSOC 2012, London, 2012, vol. 1. [4] R. Cucchiara, C. Grana, M. Piccardi, and A. Prati, Detecting moving objects, ghosts, and shadows in video streams, Pattern Anal. Mach. Intell. Ieee Trans., vol. 25, no. 10, pp. 1337 1342, 2003. [5] D. R. Santos, Q. Dalmolin, and M. A. Basso, Detecção automática de sombras em imagens de alta resolução, Bol. Ciências Geodésicas, vol. 12, pp. 87 99, 2006. [6] J. A. S. Centeno and A. da P. Pacheco, Classificação de sombra em imagens orbitais e aéreas utilizando uma matriz de co-ocorrência modificada, Bol. Ciências Geodésicas, vol. 17, no. 1, 2011. [7] G. D. Finlayson, S. D. Hordley, C. Lu, and M. S. Drew, On the removal of shadows from images, Pattern Anal. Mach. Intell. Ieee Trans., vol. 28, no. 1, pp. 59 68, 2006. [8] M. S. Drew and H. R. V. Joze, Sharpening from Shadows: Sensor Transforms for Removing Shadows using a Single Image, in 17th CIC, Albuquerque, New Mexico, 2009, vol. 17, pp. 267 271. [9] J. Tian, J. Sun, and Y. Tang, Tricolor Attenuation Model for Shadow Detection, Image Process. Ieee Trans., vol. 18, no. 10, pp. 2355 2363, 2009. [10] G. Devi, S. Kaliappan, and Malini, Stochastic Method for De-shadowing and Objects Retrieval from High Resolution Images, Int. J. Comput. Sci. Eng. Ijcse, vol. 4, no. 3, pp. 1468 1478, Apr. 2011. [11] A. Sanin, C. Sanderson, and B. C. Lovell, Shadow detection: A survey and comparative evaluation of recent methods, Pattern Recognit., vol. 45, no. 4, pp. 1684 1695, 2012. [12] R. C. Gonzalez and R. E. Woods, Processamento Digital de Imagens, 3rd ed. São Paulo: Pearson Pretience Hall, 2010. [13] A. Rosenfeld and P. de la Torre, Histogram concavity analysis as an aid in threshold selection, Syst. Man Cybern. Ieee Trans., vol. SMC-13, no. 2, pp. 231 235, 1983.