Potential of face area data for predicting sharpness of natural images

Transcription

1 Potential of face area data for predicting sharpness of natural images Mikko Nuutinen a, Olli Orenius b, Timo Säämänen b, Pirkko Oittinen a a Dept. of Media Technology, Aalto University School of Science and Technology, Espoo, Finland b Dept. of Psychology, University of Helsinki, Helsinki, Finland ABSTRACT Face detection techniques are used for many different applications. For example, face detection is a basic component in many consumer still and video cameras. In this study, we compare the performance of face area data and freely selected local area data for predicting the sharpness of photographs. The local values were collected systematically from images, and for the analyses we selected only the values with the highest performance. The objective sharpness metric was based on the statistics of the wavelet coefficients for the selected areas. We used three image contents whose subjective sharpness values had been measured. The image contents were captured by 13 cameras, and the images were evaluated by 25 subjects. The quality of the cameras ranged from low-end mobile phone cameras to low-end compact cameras. The image contents simulated typical photos that consumers take with their mobile phones. The face area sizes on the images were approximately 0.4, 1.0 or 4.0 %. Based on the results, the face area data proved to be valuable for measuring the sharpness of the photographs if the face size was large enough. When the face area size was 1.0 or 4.0 %, the performance of the measured sharpness values was equal to or better than the sharpness values measured from the best local areas. When the face area was too small (0.4 %), the performance was low compared with the best local areas. Keywords: sharpness, image quality, face area, digital cameras 1. INTRODUCTION The reproduction of human faces is an important quality factor for photographs [1,2]. Face detection and recognition is a wide and active area of research [3-5] and is a sub-field of image analysis and pattern recognition research. Face detection methods can be used for numerous applications. For example, face detection has been used in many consumer still and video cameras. Face area data can be used for tuning exposure, focus or color [6-9]. Face area data can also be utilized for image enhancement processing in a camera or post-processing in a computer [10,11]. In this study, the values of face area data are evaluated for image quality calculation. We compared the performance of face area data and the best local area data for predicting the subjective sharpness of photographs. The attributes used to characterize camera image quality have often been measured using specific test targets under laboratory conditions. Differences between the lighting conditions of a laboratory and natural scenes can result in different image processing settings. The level of sharpening and noise removal, for example, can change when illumination level changes. Of course, different exposure times and color channel gains also have an effect on image quality. Problems related to lighting conditions can be avoided if quality is measured directly from natural photographs. However, difficulties arise from the unreliability of computational image quality methods when applied to images produced by digital cameras. For example, many no-reference sharpness metrics interpret graininess or noise as edges or some other image structure because they do not recognize image content or they cannot find the appropriate local areas from where the metric should be calculated. The reflectance and shape characteristics of human faces are fairly well known. The colors in faces fall within a specific area in the color space. In addition, the global statistics can be defined analytically. The idea behind this study is the analogy between the test target data and face area data in photographs. Because of known properties, test target images can be used to characterize the function of cameras. In addition, the sampling patches are easy to align with known shapes of targets. The same approach is also applicable for the face areas in photographs. We can think of the face area in a photograph as a natural test target because its characteristics are known. In this study, we compare the objective sharpness values measured from face area data to values measured from the best local areas. The local values were collected systematically from images, and for the analyses we selected only the values with the highest performance. The objective sharpness metric was based on the statistics of the wavelet coefficients of the selected area. The face areas were selected manually. The performance of the face detection algorithms was

2 disregarded because it was out of the scope of the study; we wanted to evaluate the information content of face data for sharpness measurements, not the performance of face detector algorithms. The first research question is: How does the performance of face area data for predicting sharpness compare to the performance of freely selected local area data? The second research question is: How does face area size affect performance? This paper is organized as follows. After an introduction about the background and motivations of this work, Section 2 presents the test images, the subjective data and the sharpness metric. Section 3 presents the results, and Section 4 offers conclusions and suggestions for future work. 2. METHODS 2.1 Test images The test materials used in this study were prepared from the three views (image contents) shown in Figure 1. The images were captured by 13 cameras (Table 1). One camera was a digital still camera (DSC), and the other 12 cameras were mobile device cameras. The pixel count of the cameras was between 3 and 12 Mpix. Eleven cameras were equipped with an LED or powerful xenon flash. Each view was captured several times, and only the best image was selected for each camera. The only allowable limiting factor of a shoot was related to the performance of the camera. The image had to be in focus, and both the white balance and brightness had to be accurately adjusted. The views were also captured by a Reference camera. The images from Reference camera were used as a high-quality reference for observers in the subjective tests. Table 1. Camera types used in the study. Camera Pixel count Type Flash type Mobile Mobile Mobile Dual LED Mobile LED Mobile Dual LED Mobile Dual LED Mobile Dual LED Mobile LED Mobile LED Mobile LED Mobile Xenon Mobile Xenon DSC Xenon Example images of content are shown in Figure 1. Content 1 simulates a bar or restaurant photo; its illuminance was very low (2 lux). The exposure was based on camera flash or LED and for the analyses of Content 1, we used only the images that were captured by cameras with a flash or LED. Thus, the images of Camera 1 and Camera 2 were rejected out of the analyses. Content 2 simulates a living room photo, and its illuminance was 100 lux. Content 3 simulates a tourist photo; it was captured outdoors, and its illuminance was high (15 klux). The relevant variable for this study was the face size in relation to the image size. For Content 1, face size was 4 %; for Content 2, the face size was 1 %; and for Content 3, the face size was 0.4 %.

3 Content 1 Content 2 Content 3 Figure 1. Image contents used in the study 2.2 Subjective data In this study, we utilized sharpness data from a large-scale subjective test set. For the subjective test, the images were scaled to a 1600 x 1200 pixel size. Black borders were added to the images to match the image file resolution to the display resolution (1920 x 1200). The test setup included two Eizo ColorEdge CG241W displays and one small display. The test image was shown on one display, and the reference image was shown on the other. The input of the observer was shown on the small display. The observers evaluated the overall quality value and the attribute values of an image. In this study, we utilized only the sharpness attribute data. The images from each content were shown one at a time. The order of images and contents were randomized for each observer. The viewing distance was about 80 cm, and the ambient illuminance was 20 lux. University students were used as observers (n=25). They were all naïve regarding image quality. Figure 2 shows the subjective sharpness values for different contents sorted in ascending order. The 95% confidence intervals (vertical lines) and a sharpness value of 50 (horizontal lines) on a scale of were added to the figures. The content specific scales were easier to evaluate with the aid of the added horizontal lines. Based on Figure 2, there are clear differences in the scales among the contents. Content 1 had low illuminance, and the sharpness scale was wide and increased smoothly. The high-end cameras with powerful xenon flashes had high sharpness values, and the low-end cameras with low-power LED flashes had low sharpness values. The distribution shapes for Content 2 and Content 3 differed. Content 2 had been captured under low indoor illuminance, and Content 3 had been captured under high outdoor illuminance. For Content 2, there is a group for the unsharp images without statistical differences between the values, and there is a group for the two or three sharper images. For Content 3, there are few groups for the sharp images without statistical differences between the values and the single unsharp image. Content 1 Content 2 Content 3 Figure 2. Subjective sharpness values on the vertical axis (scale 0-100) with 95 % confidence intervals sorted in ascending order (on the horizontal axis) for Content 1 with 11 cameras and for Content 2 and Content 3 with 13 cameras

4 2.3 Sharpness metric The objective sharpness metric, S, is calculated by Equations (1) and (2). Equation (1) calculates the standard deviation σ k of the first scale wavelet coefficient energy for the sub-band of direction k. Equation (2) combines the standard deviation values of the different directions for overall sharpness. Wavelet decomposition was performed using the Matlab Wavelet toolbox and its Haar wavelets. The standard deviation was calculated within a segment of size MxN pixels, where μ k is the mean wavelet coefficient energy, and k is the vertical, horizontal or diagonal (v, h, d) direction. A segment was a cropped face area or a freely selected pixel block. σ k = M N 1 ( w ijk k ) MN 1 1 i= j= μ (1) S = σ v + σ h + σ d 3 When we measured the face area performance, the face areas were cropped manually from the images. In the pretest, we tested a face detection algorithm for face area detection and cropping. The performance of the algorithm depended on the image quality and image content. The algorithm found the face areas but also made false-positive detections. The goal of this study was to evaluate the face area data for predicting image sharpness. With the manual cropping, the face area blocks were equal regardless of image quality or image content. Figure 3 shows the cropped face area images for Content 3. (2) Figure 3. Face areas cropped from the images of Content 3 for Cameras 1-13 Figure 4 shows the first scale approximation coefficients and wavelet coefficients in the vertical, horizontal and diagonal directions for the face area of an image from Content 3. The wavelet coefficients have been scaled [-50 50] for visualization purposes. It can be seen from Figure 4 that the mouth, eyebrows and nose areas have high coefficient values for the vertical and horizontal directions. It can be expected that their energy contributes the value of sharpness metric.

5 Figure 4. Wavelet decomposition for Content 3 image The local area blocks we compared to the face areas were selected by searching the highest correlation values between Equation (2) and the subjective data. Equation (2) was applied to the corresponding square block areas of the images. The block size was constant (125 x 125 pixels), and the image size was 1600 x 1200 pixels in all cases. The candidate blocks included structure energy such as edges and textures. The candidate blocks for Content 1 are shown in Figure 5 as an example. Finally, the three best local areas (blocks) for the different contents were selected for the next analyses. Figure 6 shows the three selected blocks and the face areas for the different contents. The selected blocks include both coarse textures and strong edges. Figure 5. Candidate blocks for Content 1 Content 1 Content 2 Content 3 Figure 6. Local block and face areas used for sharpness calculations

6 3. RESULTS Table 2 shows the Pearson linear correlation coefficients for the face area and the three best local areas with subjective sharpness. The Pearson linear correlation coefficients are also shown for the global areas. The global area metric was calculated using all the pixel values of an image. Based on the results, the face area performance of Content 1 and Content 2 was equal to or higher than the performance of the local or global areas. The face area performance of Content 3 was low compared to the best local area. The performance of the global area was high for Content 1, moderate for Content 2 and low for Content 3. Table 2. Pearson linear correlation coefficients for the face areas, the three best local areas and the global area Face area Local area 1 Local area 2 Local area 3 Global Content Content Content (left), (right) The performance of the face area for Content 1 and 2 was notably higher than the performance of Content 3. The reason for this result could be the larger face area. The larger face areas have more information related to sharpness than the smaller face areas. The size of the face area of Content 3 was only 0.4 %. The size of the face area was 4.0 % for Content 1 and 1.0 % for content 2. However, based on the data shown in Figure 2, Content 3 was difficult for the subjective observers, lowering its usefulness for objective metric validation. The illuminance level of Content 3 was high enough for all cameras, and thus the quality differences among the captured images were low. Content 1 was easier to evaluate. The illuminance level of Content 1 was low, and thus the quality differences between the captured images were high. For Content 1, the quality of the images captured by the cameras equipped with xenon flash was high, and the quality of the images captured by the cameras with low-power LEDs was low. In addition, Content 1 only included a single object, which further simplified the evaluation task. Content 2 and Content 3 included numerous objects that could divert the observers attention. The performance of the global area was high for Content 1. As with the subjective test, the reason for this result could be the low illuminance level and/or simple image composition. For Content 1, the global metric estimated only the reproduction of the person in the view. The peripheral energy did not affect the metric as much as it affected the other contents. With Content 2, for example, the view was complex, and there were many objects and textures that had an effect on the global values, but no significant effect on subjective perception. 4. CONCLUSIONS Based on the results, face area data are useful for measuring the sharpness of photographs if the face size is large enough. If the face area is too small, the performance can be low compared with the best local areas. It is concluded that face areas include information that no-reference or reduced-reference metrics can utilize. A metric could recognize the faces automatically or semi-automatically and use the data if the face area size is large enough. If faces cannot be found or their sizes are too small, the metric could employ traditional methods, such as edges, for calculations. There are certain factors that should be taken into account when the reliability of the study is analyzed and further studies are proposed. For example, the persons in Content 1 and Content 2 had eyeglasses. The frequency energy of eyeglasses can be a strong component of the sharpness metric. It could be argued that eyeglass data belong to the face area data. The face area performance of Content 1 and Content 2 was high compared to Content 3. However, the face area sizes of Content 1 and Content 2 were large compared to Content 3. The face areas of Content 1 and Content 2 provided the metric with more information. A comparison of the performances of the left and right face areas for Content 3 show that both were low, although the right face area had eyeglasses. It is clear that the eyeglass factor needs to be considered in further studies. The different contents also had different illuminance levels. A useful constraint for further studies would be to restrict the measurements to an environment in which the only variable would be the face area size. The validation measurements for the metric should be done under laboratory conditions. The only variable parameter would be the distance between the camera and the person.

7 In addition to the illuminance levels between contents, the persons can change between the contents. It would be useful to measure how the face area data of different persons affect the results (e.g., how a person affects the scale of objective values or what would be the most robust and person-independent statistical parameter for describing the face area data). ACKNOWLEDGEMENTS This work was partially financed by Nokia Mobile Solutions / Symbian Smartphones. The authors thank Fredrik Hollsten and Jussi Tarvainen for the test images. REFERENCES [1] Tong, Y., Konik, H., Cheikh, F. A., Tremeau, A., Full Reference Image Quality Assessment Based on Saliency Map Analysis, J. Imaging Sci. Technol. 54(3), (2010). [2] Menegaz, G., Zambon, R., Towards a Semantic-Driven Metric for Image Quality, Proc. IEEE ICIP, vol. 3, 1176 (2005). [3] Lang, L., Gu, W., Study of Face Detection Algorithm for Real-time Face Detection System, Proc. ISECS, (2009). [4] David, A., Panchanathan, S., Wavelet-histogram method for face recognition, Journal of electronic Imaging 9(2), (2000). [5] Marrszalec, E., Martinkauppi, B., Soriano, M., Pietikäinen, M., Physic-based face database for color research, Journal of Electronic Imaging 9(1), (2000). [6] Jin, E. W., Lin, S., Dharumalingam, D., Face detection assisted auto exposure: supporting evidence from a psychophysical study, Proc. SPIE 7537, [75370K] (2010). [7] Lajevardi, S. M., Hussain, Z. M., Contourlet Structural Similarity for Facial Expression Recognition, Proc. IEEE ICASSP, (2010). [8] Wang, Y.-K., Wang, C-F., Face Detection with Automatic White Balance for Digital Still Cameras, Proc. IEEE IIHMSP, (2008). [9] Rahman, M. T., Kehtanavaz, N., Real-Time Face Priority Auto Focus for Digital and Cell-Phone Cameras, IEEE Transaction on Consumer Electronics 54(4), (2008). [10] Delahunt, P. B., Zhang, X., Brainard, D. H., Perceptual image quality: Effects of tone characteristics, Journal of Electronic Ímaging 14(2), (2005). [11] Ciuc, M., Capata, A., Florea, C., Objective measures for quality assessment of automatic skin enhancement algorithms, Proc. SPIE 7529, [75290N] (2010).