Cares and Concerns of CIE TC8-08: Spatial Appearance Modeling and HDR Rendering

Transcription

1 Cares and Concerns of CIE TC8-08: Spatial Appearance Modeling and HDR Rendering Garrett M. Johnson 1 Munsell Color Science Laboratory, Rochester Institute of Technology, 54 Lomb Memorial Drive, Rochester, NY ABSTRACT CIE Division 8 concerns itself with suggesting methods for interpreting complex color stimuli, or more specifically images. To accurately model color appearance of images we can no longer consider pixels as simple patches viewed in isolation. TC8-08 is tasked with examining spatial color appearance models, with an emphasis on high-dynamic range images. High-dynamic range (HDR) images are typically images that contain a large range of luminance information, and are represented by more than 8-bits per channel. The luminance range of the real world is typically much greater than the luminance range of any color reproduction device. Imaging technology has advanced so as to capture, or synthetically generate, this large luminance range. The question of how to reproduce the images on a lower dynamic range device has not been solved. TC8-08 is examining the use of spatial appearance models to aid in this task. This paper discusses some of the many problems and pitfalls that TC8-08 are looking into with regards to testing spatial appearance models. Among these concerns are: algorithm choices and implementations, psychophysical experimental design, HDR image availability, preference verses accuracy, and the use of next-generation HDR displays as validation. Keywords: High Dynamic Range, Tone-Mapping, HDR, CIE, Spatial Appearance Models, TC INTRODUCTION The International Commission on Illumination (CIE) is dedicated to providing discussion, information, and guidance in the science and art of light and lighting. The terms of reference of Division 8 of the CIE is to study procedures and prepare guides and standards for the optical, visual and metrological aspects of the communication, processing, and reproduction of images, using all types of analogue and digital imaging devices, storage media and imaging media. Along those lines, Technical Committee (TC) 8-08 is tasked with developing guidelines and testing methods for using spatial or image appearance models, specifically for use with High Dynamic Range (HDR) images. The goal of TC8-08 is not to create a CIE recommended image appearance model, but rather to design and conduct experimental techniques for evaluating these models. 1.2 What is High Dynamic Range? High dynamic range has been somewhat of a buzzword in the imaging community in recent times. It is important to precisely define what is meant when discussing HDR, specifically for TC8-08. Dynamic range can be described as the ratio between the brightest and darkest objects in a scene, in which case it can be thought of as form of image contrast. In real-world scenarios the luminance range we encounter can span over 9 log units, from below candelas per meter squared (cd/m 2 ) for typical starlight illumination, to over 10,000 cd/m 2 for the sun itself. This range is shown with some typical viewing environments in Figure 1. The human visual system is capable of adapting to this wide range of illumination in a matter of minutes, and is capable of simultaneously viewing between 3 and 5 orders of luminance variation through means of local adaptation. This is fortuitous, as most everyday scenes do not typically have more than 10000:1 contrast ratio between the brightest object and the darkest object. Jones and Condit conducted a seminal study in the 1940s examining the contrast ratio of typical outdoor scenes. 1 They found that on average the contrast range of a typical outdoor scene was around 3 orders of magnitude, while scenes of both higher and lower contrast were less likely. This study did not suggest that higher contrast ranges do not occur in nature, but rather that they are less common. This might explain why there has been such 1 garrett@cis.rit.edu,

2 success over the years with photographic reproductions. A a good photographic reproduction can capture and reproduce up to 3 orders of luminance differences. As suggested from Figure 1, typical computer displays range from 50 cd/m 2 for CRTs and perhaps 150 cd/m 2 for high-quality LCD panels. These displays might have a contrast ratio of approximately 2 orders of magnitude (100:1). Figure 1: Example scenes and their approximate luminance levels The high dynamic range that TC8-08 is concerned with is this absolute luminance range, or contrast ratio between the brightest pixels in the image to the darkest. These images are often stored as linear data in floating point (32-bits) per channel. Often, the term high dynamic range is also used to describe images that are encoded with more than 8-bits per color channel. These images also must be rendered to a lower bit-depth before they can be displayed, this can be thought of as a form of color-quantization. Images that are encoded at high bit-depths, but that do not span a large absolute luminance range can be thought of as relative HDR images. While the color-quantization of relative HDR images is indeed a very interesting problem, it is not the primary concern of TC Generating High Dynamic Range Images As discussed above, TC8-08 is focusing primarily on the display and appearance of images that have a high absolute luminance difference between the brightest and darkest pixels. Until recently it has been difficult digitally capture these images. Physically based rendering packages, such as RADIANCE 2 have been capable of producing HDR scenes for many years. Out of necessity the computer graphics community has been at the forefront of research on ways for displaying these scenes. Additional summaries and details can be found in references 3-8. A technique for combining multiple photographic exposures into a single HDR scene was described by Debevec 9, and remains the source of many of the images available today. New imaging technologies continue to emerge which allow for direct capture of high dynamic range images, including commercial products from Fuji Film and Pixim. More details can be found in references The images need to be stored in a file format capable of handling the high contrast ratios as well as extended bit-depth. Possible encoding formats include Radiance RGBE, TIFF Log-luv, and OpenEXR. 13 Ward has written some excellent overviews regarding HDR image encoding schemes. 3, Rendering High Dynamic Range Images Scenes in the real-world can cover a large absolute luminance range between the highlights and shadows. Image devices are maturing to the point that they can now capture this large range. Display devices, including CRTs, LCDs, and printers are still limited to a much lower dynamic range. There have been advances in display technologies that increase the overall range of the devices, though at this point those are still specialized and rare. 15 The act of converting HDR images such that they can be displayed on lower dynamic range displays is often called HDR rendering, or tone-mapping. The task of TC8-08 is to formulate and implement guidelines for testing these HDR rendering algorithms. 2. HDR IMAGE SELECTION An ideal HDR rendering algorithm should be image independent, or capable of performing the desired tone-mapping task regardless of the input image content. As such, it is important to have a very wide variety of images available for testing algorithms. These images should include, but are not limited to, photographs of both indoor and outdoor scenes, mostly dark photographs with small bright illuminators, mostly bright illumination with small shadow regions, and computer generated renderings. Using a variety of images can help identify both strengths and weaknesses of individual

3 rendering algorithms. CIE TC8-08 is making available a large database of HDR images for researchers to download and utilize. Links to many of these can be found at Figure 2. Example HDR images, along with luminance histograms: (Clockwise from top left) Colorcube, Garage, Tahoe, Plant An example of four different types of HDR images, along with their approximate luminance histograms, is shown in Figure 2. The ColorCube image shows a relatively flat histogram throughout the image, while the garage image is almost entirely towards the dark shadows, with a single bright peak for the bright lights. The plant image shows an almost bi-modal histogram, with information in the shadows and highlights, but not much in the mid-tones. The Tahoe image is similar in shape to the garage image with most the information in the shadows, with a very large highlight peak corresponding to the broad light source. These are just a few examples of the variety of images that should be used to test HDR rendering algorithms Baseline Images In some situations it might be desirable to have a set of baseline images that all researchers can include in their algorithm testing and experimentation. TC8-08 is looking into a small subset of the available images for this purpose. The group of baseline images should include as much variation as possible, to test the widest range of possible problems. Variety in the overall luminance of the brightest and darkest parts of the images, and in the color characterization of the source camera, can help test the device independence of the rendering algorithms implementation. The selection of the baseline image set will be one of the first recommendations TC8-08 makes. 3. HDR ALGORITHM SELECTION

4 Perhaps the question that is asked most often is which tone-mapping operator is best? Although this is not the question that CIE TC8-08 is tasked with answering. The TC is more concerned with creating a set of experimental guidelines so that others can test their own algorithms, and compare their results with others. For the purpose of this document the concept of algorithm and operator are considered synonymous. Still, it is impossible to design a set of experiments or provide guidance if we do not actually test some existing HDR algorithms. There are many algorithms available, and it seems that new ones are published every year. As such, it would be impossible for the CIE to test all the existing algorithms. By providing tools and techniques, however, it will be possible to aid others in their own testing. Details about many existing algorithms, though by no means a complete survey, can be found in references 4-8 and Figure 3 shows the same image rendered by 5 different algorithms. These algorithms are, clockwise from the top left: Durand & Dorsey 6, Multiscale Retinex 22, Ward 4, Reinhard 5 and icam 16. Figure 3: Example renderings of same image by five different algorithms The purpose of Figure 3 is not to illustrate which of the algorithms is best, but rather to give an example of the different approaches many of these operators take for rendering the same input image. While all of the available algorithms possess similarities and differences, there has been some success in splitting the description of them into two distinct groups: global and local operators. The global operators tend to apply the same tone-mapping scheme to every single pixel in the image, while the local operators might change the tonemapping on a per-pixel basis. It is important to stress that for global algorithms it is not necessarily the same operator applied identically for every image, as the global operator can be a function of image content such as the histogram or absolute dynamic range. Likewise the local algorithms can take different approaches for determining the spatial extent of the operator, and can use low-pass filters, edge preserving low-pass filters, and multi-scale pyramids.

5 There are strengths and weaknesses to both the global and local tone-mapping approaches. The global operators tend to be computationally simpler and as a result can be easier to implement and faster to perform. The spatial processing of the local operators tends to be computationally more expensive, but can allow for a more dramatic reduction in overall dynamic range. The local algorithms can be prone to inducing artifacts in the process of reducing the dynamic range, with the most common artifact being ringing around bright light sources. Since the local algorithms are capable of larger dynamic range compression and also tend to mimic the local adaptation behavior of the human visual system, TC8-08 is very interested in testing these specifically. The global operators are intriguing for their simplicity, and should be considered as a type of baseline rendering algorithm. Figure 4: An example of algorithm hand tuning to minimize artifacts in a rendered image, from Reference Algorithm Coding & Tuning One of the biggest concerns when designing tests of many different algortithms is their actual implementation of each algorithm. Care must be taken to assure that each algorithm meets the original author s intent and is not used improperly. Obviously the easiest way to do this would be to have original source code provided by the authors themselves. This is not always possible, as the original source might not be available or might be in a form that does not allow for easy use by the community. In this case the experiment must rely on a re-implementation by a third-party, which may not be identical. One of the goals of CIE TC8-08 is to make available the source (or at least executable) of as many algorithms as possible. Another important consideration for algorithm testing is the manual tuning of free parameters. Many of the algorithms are quite complicated and have a large number of free parameters that can be set individually for specific applications or images. This example is shown in Figure It would be very easy for an experiment to be swayed by comparing a series of images from algorithms at their default settings to one that had been hand-tuned. A good compromise between the simplicity of a no-parameter algorithm with the complexity of a many-parameter algorithm

6 seems to be through the use of default settings or a computational approach for hands-free tuning. TC8-08 is most interested in the testing procedures themselves and not with specific algorithm tuning. 4. PSYCHOPHYSICAL EXPERIMENTATION The meat of TC8-08, so to speak, is in the psychophysics. As mentioned above, there have been scores of HDR algorithms written and published in the last decade. The majority, though not all, of these publications tend to describe the new algorithm, show some example images, and conclude that is superior based upon those few images. Traditionally little has been done to actually verify these conclusions using robust psychophysical techniques and actual human observers. Thankfully, this practice seems to be coming to an end, and the goal of CIE TC8-08 is to aid and assist with algorithm validation guidelines What are the goals of the algorithms themselves? Perhaps the biggest question facing any researcher attempting to scale HDR rendering algorithms is the purpose for applying the algorithm itself. While all of these operators attempt to scale an image with a high luminance range down for display on a lower dynamic range display, the actual goal of the algorithms vary considerably. It is important to consider the application and desired affect in order to adequately define testing procedures. For instance, is the goal of the algorithm to provide the maximum amount of detail information to the end-user, as might be desirable for a radiologist examining medical MR images? Other goals might include producing pleasing or preferred pictorial images, reproducing overall appearance between the original and display, maintaining contrast relationships between objects in the scene, maintaining the original photographers intent, and predicting visibility of specific objects in a scene. It is important to consider the desired purpose when determining which algorithm to chose, and it is entirely plausible that no single algorithm will be best suited for all tasks. There is interest in TC8-08 for all of the abovementioned tasks, though initially the focus is on preference and appearance, or accuracy mapping. 4.2 Experimental Techniques There are many valid psychophysical techniques that can be applied to testing HDR rendering algorithms. If a perceptual scale is desired, one can perform a pair-comparison or rank order experiment. Using Thurstonian analysis it is possible to produce interval scales along a psychological scale, such as image preference. More details on traditional pyschometric scaling can be found in Engledrum. 23 Kuang et al used a paired-comparison paradigm to scale the preference of 8 HDR rendering algorithms using 10 different image scenes. 17 Tone-mapped images were displayed sideby-side on an LCD monitor with a maximum luminance of 180 cd/m 2 and observers were asked to choose the image they thought was better. This experiment was conducted in a darkened room, so the LCD monitor was the only object setting the observers adaptation level. The observers were not presented with the original scene, and there were no right or wrong answers. The results of this experiment were interval scales of preference. It is tempting to use the results of this type of experiment to claim superiority of a given algorithm. This can be dangerous. Since very few of these types of experiments have been performed, it is unknown what the effect of viewing conditions are on overall performance. The interval scale of algorithm preference provides a good starting point for this type of research but should only be considered valid for the specific viewing conditions of the experiment The effect of viewing conditions on experiments One of the important goals of TC8-08 is to determine the effects of viewing conditions on the performance of HDR rendering algorithms, and to provide recommendations for future testing. We can borrow a page from the colorappearance world and start with a unified vocabulary, such as that used for color-appearance models. This is illustrated in Figure 5.

7 Figure 5: Viewing condition definitions for color appearance models. There is an immediate discrepancy between the color appearance stimulus, which is generally considered a simple color patch, and the stimulus used in HDR imaging. Should each pixel in an image be treated as an individual stimulus? What do we consider the stimulus when looking at a real world scene? This concept is illustrated in Figure 6. Figure 6: Ambiguity of traditional color appearance vocabulary for HDR testing It is important to consider the viewing conditions and the effect they might have on the perception of HDR scenes, as well as on the lower dynamic range rendering of the scene. Perhaps the greatest concern is the overall surround, and similarly the spatial extent of the reproduced scene. It is known that the visual systems is capable of viewing a large range of absolute luminance levels, as shown in Figure 1, through both local and global adaptations. The overall luminance of the visual field is strongly tied to the global state of adaptation. This suggests that the surround may play a crucial role in determining the overall state of adaptation of the visual system, and may have a strong influence in the perception of high dynamic range images. Similarly, the spatial extent of the stimulus, which we will assume for now to be the entire image as well as the background, may also play significant role.

8 Consider an experiment where an observer views an actual high dynamic range scene, such as an outdoor landscape, and then a reproduction of that scene displayed on a small display in a darkened room. Even if that display matches the exact luminance range of the original outdoor scene, there is a good chance that the appearance of the reproduction will be different. In the darkened room the visual system may be adapted to an overall lower light level then would be faced with a scene of a brighter extent. Ledda et al examined a similar situation using both a small extent and large extent HDR viewer. 24 In this particular experiment they were interested in visibility matches between the real world, and the two HDR viewers. They used contrast charts to determine that on average visibility was similar for all three viewing conditions, though with bright sources in the periphery the visibility can be affected. This study also illustrates the importance of defining the specific perceptions (e.g. contrast and visibility) that are being judged. 4.3 Accuracy scaling & HDR displays One of the primary concerns of TC8-08 is in scaling HDR rendering algorithms for accuracy in reproduction. Accuracy is probably a misleading term, as it is clearly not the goal of the algorithms to create a physical match between the original scene and the reproduction. A better term might be accuracy of appearance, such that the reproduction produces a close perceptual match to the original scene. Similar techniques as those used in the preference experiment described above can be used to scale HDR accuracy and have been used for many years in the field of cross-media color reproduction. Unlike the preference experiment, it is necessary to make comparisons against an original. When the original is a HDR scene this makes the experiment all the more difficult to perform in a controlled lab environment. One possibility is to build a small scene in a lab, and then photograph that scene using a high-resolution HDR camera or use the multiple exposure technique described by Debevec 9. The scene can then be tone mapped onto a lower dynamic range display and compared against the original. A paired comparison paradigm can still be used to generate scales of accuracy. There are several caveats with this technique. The captured images must be colorimetrically accurate to begin with, so that any affect the rendering algorithm has on color is captured. There is also concern with adaptation between viewing the HDR original image and the low-dynamic range display. It might be necessary to institute a delay after viewing the original scene and the reproductions. This would be done to change both the overall adaptation state as well as to avoid any after-images that may be caused from a high luminance scene in a small spatial extent. TC8-08 hopes to examine these issues and provide guidance for appropriate experimental methods. Rather than building an actual scene in a lab environment, it might be possible to use a HDR display as an original image for testing the accuracy of the tone-mapping operators. This approach has the benefit of simplicity in experimental design, while maintaining the high dynamic range of the original scene. The difference between the spatial extent of a real scene and a display will need to be addressed. 4.4 Output Medium The discussion until now regarding the output of the tone-mapping algorithms has assumed a traditional soft-copy CRT or LCD display with a contrast range of approximately 100:1. TC8-08 is also interested in examining the performance of HDR rendering for different output medium. As display technology continues to improve softcopy displays are becoming both physically brighter with higher dynamic range. Do algorithms need to take the output medium into account for optimum performance? For instance, do we need different outputs for preference, accuracy, or visibility when rendering to a 50, 100, 200, or 500 cd/m 2 LCD? Hardcopy reflection output has a physical limitation to its overall dynamic range, is it necessary to have specific algorithms for this type of output? As HDR displays become more prevalent will it still be necessary to perform a tone-mapping operation to accurately match appearances original scenes? These are all questions that will hopefully be addressed by TC CONCLUSIONS This paper is meant to be an overview of the cares and concerns of CIE TC8-08. We are tasked with the job of developing guidelines for testing spatial appearance models, with emphasis on high dynamic range tone mapping or rendering. Humans encounter a wide range of luminance levels in the real world, spanning more than 9 orders of magnitude between starlight and sunlight. Oftentimes, a single scene can contain over 4 or 5 orders of magnitude between the highlights and shadow regions. Image capture technology has advanced to a stage such that is now possible to capture this wide luminance range without difficulty. While display technologies continue to improve, it is still

9 necessary to reduce the dynamic range of these images for output. This document attempts to illustrate many of the difficulties and caveats we are anticipating in developing experimental guidelines. 6. REFERENCES 1. L.A Jones and H.R. Condit, The Brightness of Exterior Scenes and the Computation of Correct Photographic Exposure, Journal of the Optical Society of America A, (1941). 2. RADIANCE Rendering Package, 3. G.Ward, High Dynamic Range Imaging, Proc. IS&T/SID 9th Color Imaging Conference, 9-16 (2001). 4. G. W. Larson, H. Rushmeier, and C. Piatko, A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes, IEEE Transactions on Visualization and Computer Graphics, 3, , (1997). 5. E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda, Photographic Tone Reproduction for Digital Images, Proceedings of SIGGRAPH 02, (2002). 6. F. Durand and J. Dorsey, Fast Bilateral Filtering for the Display of High-Dynamic Range Images, Proceedings of SIGGRAPH 02, (2002). 7. J. M. Dicarlo and B.A. Wandell, Rendering High Dynamic Range Images, Proceedings of SPIE Image Sensors, (1999). 8. S.N. Pattanaik, J.A. Ferwerda, M.D. Fairchild, and D.P. Greenberg, A Multiscale Model of Adaptation and Spatial Vision for Image Display, Proceedings of SIGGRAPH 98, (1998). 9. P. E. Debevec and J. Malik, Recovering High Dynamic Range Radiance Maps from Photographs, Proc. SIGGRAPH'97, (1997). 10. D. Yang, B. Fowler, A. El Gamal, and H. Tian, A 640x512 CMOS Image Sensor with Ultrawide Dynamic Range Floating-Point Pixel-Level ADC, IEEE Journal of Solid State Circuits, 34, , (1999). 11. S. K. Nayar and T. Mitsunaga, High Dynamic Range Imaging: Spatially Varying Pixel Exposures, Proc. IEEE CVPR, (2000). 12. K. Takemura, Challenge for improving image quality of a digital still camera, Proc. SPIE Electronic Imaging Conf. Santa Clara, (2003). 13. OpenEXR, G.Ward, High Dynamic Range Image Encoding, SIGGRAPH04 Course 13, (2004). 15. H. Seetzen, W. Heidrich, W. Stuezlinger, G. Ward, L. Whitehead, M. Trentacoste, A. Ghosh, A. Vorozcovs, High Dynamic Range Display Systems, Proceedings of SIGGRAPH04, (2004). 16. G.M. Johnson and M.D. Fairchild, Rendering HDR Images, IS&T/SID 11th Color Imaging Conference, Scottsdale (2003) 17. J. Kuang, H. Yamaguchi, G.M. Johnson and M.D. Fairchild, Testing HDR Image Rendering Algorithms, IS&T/SID 12th Color Imaging Conference, Scottsdale (2004). 18. A. Rizzi, C. Gatto, B. Piacentini, M. Fierro, and D. Marini, Human visual system inspired tone mapping algorithms for HDR images, Proc. SPIE Electronic Imaging Conf., (2004). 19. L. Meylan, and S. Susstrunk, Bio-inspired color image enhancement, Proc. SPIE Electronic Imaging Conf., (2004). 20. J. Tumblin and G. Turk, Low Curviture Image Simplifiers (LCIS): A boundary hierarchy for detail-preserving contrast reduction, Proceedings of SIGGRAPH99, (1999). 21. F. Drago, W. Martens, K. Myszkowski, and H. Seidel, Perceptual evaluation of tone mapping operators with regard to similarity and preference, Research Report MPI-I , Max-Planck-Institut fur Informatik, (2002). 22. B. Funt, F. Ciurea, and J. McCann, Tuning Retinex parameters, Proceedings of the IS&T/SPIE Electronic Imaging Conference, (2002). 23. P. Engeldrum, Psychometric Scaling: a Toolkit for Imaging Systems Development, Imcotek Press, Winchester, (2000). 24. P. Ledda, A. Chalmers, H. Seetzen, HDR Displays: a validation against reality, International Conference on Systems, Man and Cybernetics, (2004).