Digital Photography with Flash and No-Flash Image Pairs

Transcription

1 Digital Photogaphy with an No- Image Pais Geog Petschnigg Richa Szeliski Maneesh Agawala Michael Cohen Micosoft Copoation Hugues Hoppe Kentao Toyama No- Oig. (top) Detail Tansfe (bottom) No- Detail Tansfe with Denoising Figue 1: This canlelit setting fom the wine cave of a castle is ifficult to photogaph ue to its low light natue. A flash image captues the high-fequency textue an etail, but changes the oveall scene appeaance to col an gay. The no-flash image captues the oveall appeaance of the wam canlelight, but is vey noisy. We use the etail infomation fom the flash image to both euce noise in the no-flash image an shapen its etail. Note the smooth appeaance of the bown leathe sofa an cisp etail of the bottles. Fo full-size images, please see the supplemental DVD o the poject website Abstact Digital photogaphy has mae it possible to quickly an easily take a pai of images of low-light envionments: one with flash to captue etail an one without flash to captue ambient illumination. We pesent a vaiety of applications that analyze an combine the stengths of such flash/no-flash image pais. Ou applications inclue enoising an etail tansfe (to mege the ambient qualities of the no-flash image with the high-fequency flash etail), white-balancing (to change the colo tone of the ambient image), continuous flash (to inteactively ajust flash intensity), an e-eye emoval (to epai atifacts in the flash image). We emonstate how these applications can synthesize new images that ae of highe quality than eithe of the oiginals. Keywos: Noise emoval, etail tansfe, shapening, image fusion, image pocessing, bilateal filteing, white balancing, eeye emoval, flash photogaphy. 1 Intouction An impotant goal of photogaphy is to captue an epouce the visual ichness of a eal envionment. Lighting is an integal aspect of this visual ichness an often sets the moo o atmosphee in the photogaph. The subtlest nuances ae often foun in low-light conitions. Fo example, the im, oange hue of a canlelit estauant can evoke an intimate moo, while the pale blue cast of moonlight can evoke a cool atmosphee of mystey. When captuing the natual ambient illumination in such low-light envionments, photogaphes face a ilemma. One option is to set a long exposue time so that the camea can collect enough light to pouce a visible image. Howeve, camea shake o scene motion uing such long exposues will esult in motion blu. Anothe option is to open the apetue to let in moe light. Howeve, this appoach euces epth of fiel an is limite by the size of the lens. The thi option is to incease the camea s gain, which is contolle by the ISO setting. Howeve, when exposue times ae shot, the camea cannot captue enough light to accuately estimate the colo at each pixel, an thus visible image noise inceases significantly. photogaphy was invente to cicumvent these poblems. By aing atificial light to neaby objects in the scene, cameas with flash can use shote exposue times, smalle apetues, an less senso gain an still captue enough light to pouce elatively shap, noise-fee images. Bighte images have a geate signal-to-noise atio an can theefoe esolve etail that woul be hien in the noise in an image acquie une ambient illumination. Moeove, the flash can enhance suface etail by illuminating sufaces with a cisp point light souce. Finally, if one esies a white-balance image, the known flash colo geatly simplifies this task. As photogaphes know, howeve, the use of flash can also have a negative impact on the lighting chaacteistics of the envionment. Objects nea the camea ae ispopotionately bightene, an the moo evoke by ambient illumination may be estoye. In aition, the flash may intouce unwante atifacts such as e eye, hash shaows, an speculaities, none of which ae pat of the natual scene. Despite these awbacks, many amateu photogaphes use flash in low-light envionments, an consequently, these snapshots aely epict the tue ambient illumination of such scenes. Toay, igital photogaphy makes it fast, easy, an economical to take a pai of images of low-light envionments: one with flash to captue etail an one without flash to captue ambient illumination. (We sometimes efe to the no-flash image as the ambient image.) In this pape, we pesent a vaiety of techniques that

2 analyze an combine featues fom the images in such a flash/noflash pai: Ambient image enoising: We use the elatively noise-fee flash image to euce noise in the no-flash image. By maintaining the natual lighting of the ambient image, ou appoach ceates an image that is close to the look of the eal scene. -to-ambient etail tansfe: We tansfe high-fequency etail fom the flash image to the enoise ambient image, since this etail may not exist in the oiginal ambient image. White balancing: The use may wish to simulate a white illuminant while peseving the feel of the ambient image. We exploit the known flash colo to white-balance the ambient image, athe than elying on taitional single-image heuistics. Continuous flash intensity ajustment: We povie continuous intepolation contol between the image pai so that the use can inteactively ajust the flash intensity. The use can even extapolate beyon the oiginal ambient an flash images. Re-eye coection: We pefom e-eye etection by consieing how the colo of the pupil changes between the ambient an flash images. While many of these poblems ae not new, the pimay contibution of ou wok is to show how to exploit infomation in the flash/no-flash pai to impove upon pevious techniques 1. One featue of ou appoach is that the manual acquisition of the flash/no-flash pai is elatively staightfowa with cuent consume igital cameas. We envision that the ability to captue such pais will eventually move into the camea fimwae, theeby making the acquisition pocess even easie an faste. One ecuing theme of ecent compute gaphics eseach is the iea of taking multiple photogaphs of a scene an combining them to synthesize a new image. Examples of this appoach inclue ceating high ynamic ange images by combining photogaphs taken at iffeent exposues [Debevec an Malik 1997; Kang et al. 2003], ceating mosaics an panoamas by combining photogaphs taken fom iffeent viewpoints [e.g. Szeliski an Shum 1997], an synthetically elighting images by combining images taken une iffeent illumination conitions [Haebeli 1992; Debevec et al. 2000; Masselus et al. 2002; Akes et al. 2003; Agawala et al. 2004]. Although ou techniques involve only two input images, they shae the simila goal of synthesizing a new image that is of bette quality than any of the input images. 2 Backgoun on Camea Noise The intuition behin seveal of ou algoithms is that while the illumination fom a flash may change the appeaance of the scene, it also inceases the signal-to-noise atio (SNR) in the flash image an povies a bette estimate of the high-fequency etail. As shown in Figue 2(a), a bighte image signal contains moe noise than a ake signal. Howeve, the slope of the cuve is less than one, which implies that the signal inceases faste than the noise an so the SNR of the bighte image is bette. While the flash oes not illuminate the scene unifomly, it oes significantly incease scene bightness (especially fo objects nea the camea) an theefoe the flash image exhibits a bette SNR than the ambient image. As illustate in Figue 2(b), the impovement in SNR in a flash image is especially ponounce at highe fequencies. Popely expose image pais have simila intensities afte passing though 1 In concuent wok, Eisemann an Duan [2004] have evelope techniques simila to ous fo tansfeing colo an etail between the flash/no-flash images. Figue 2: (a-left) Noise vs. signal fo a full-fame Koak CCD [2001]. Since the slope is less than one, SNR inceases at highe signal values. (bight) The igital senso pouces simila log powe specta fo the flash an ambient images. Howeve, the noise ominates the signal at a lowe fequency in the high-iso ambient image than in the low-iso flash image. the imaging system (which may inclue apetue, shutte/flash uation, an camea gain). Theefoe thei log powe specta ae oughly the same. Howeve, the noise in the high-iso ambient image is geate than in the low-iso flash image because the gain amplifies the noise. Since the powe spectum of most natual images falls off at high fequencies, wheeas that of the camea noise emains unifom (i.e. assuming white noise), noise ominates the signal at a much lowe fequency in the ambient image than in the flash image. 3 Acquisition Poceue. We have esigne ou algoithms to wok with images acquie using consume-gae igital cameas. The main goal of ou acquisition poceue is to ensue that the flash/noflash image pai captue exactly the same points in the scene. We fix the focal length an apetue between the two images so that the camea s focus an epth-of-fiel emain constant. Ou acquisition poceue is as follows: 1. Focus on the subject, then lock the focal length an apetue. 2. Set exposue time t an ISO fo a goo exposue. 3. Take the ambient image A. 4. Tun on the flash. 5. Ajust the exposue time t an ISO to the smallest settings that still expose the image well. 6. Take the flash image F. A ule of thumb fo hanhel camea opeation is that exposue times fo a single image shoul be une 1 30 s fo a 30mm lens to pevent motion blu. In pactice, we set the exposue times fo both images to 1 60 s o less so that une ieal cicumstances, both images coul be shot one afte anothe within the 1 30 s limit on hanhel camea opeation. Although apily switching between flash an non-flash moe is not cuently possible on consumegae cameas, we envision that this capability will eventually be inclue in camea fimwae. Most of the images in this pape wee taken with a Canon EOS Digital Rebel. We acquie all images in a RAW fomat an then convet them into 16-bit TIFF images. By efault, the Canon convesion softwae pefoms white balancing, gamma coection an othe nonlinea tone-mapping opeations to pouce peceptually pleasing images with goo oveall contast. We apply most of ou algoithms on these non-linea images in oe to peseve thei highquality tone-mapping chaacteistics in ou final images. Registation. Image egistation is not the focus of ou wok an we theefoe acquie most of ou pais using a tipo setup. Nevetheless we ecognize that egistation is impotant fo images taken with hanhel cameas since changing the camea settings (i.e. tuning on the flash, changing the ISO, etc.) often esults in camea motion. Fo the examples shown in Figue 11 we took the photogaphs without a tipo an then applie the egistation technique of Szeliski an Shum [1997] to align them.

3 Bilateal filte. The bilateal filte is esigne to aveage togethe pixels that ae spatially nea one anothe an have simila intensity values. It combines a classic low-pass filte with an egestopping function that attenuates the filte kenel weights when the intensity iffeence between pixels is lage. In the notation of Duan an Dosey [2002], the bilateal filte computes the value of pixel p fo ambient image A as: 1 Ap = g( p p) g( Ap Ap ) Ap, (2) k( p) p Ω whee k( p ) is a nomalization tem: Figue 3: Oveview of ou algoithms fo enoising, etail tansfe, an flash atifact etection. While we foun this technique woks well, we note that flash/noflash images o have significant iffeences ue to the change in illumination, an theefoe obust techniques fo egistation of such image pais eseve futhe stuy. eaization. Some of ou algoithms analyze the image iffeence F A to infe the contibution of the flash to the scene lighting. To make this computation meaningful, the images must be in the same linea space. Theefoe we sometimes set ou convesion softwae to geneate linea TIFF images fom the RAW ata. Also, we must compensate fo the exposue iffeences between the two images ue to ISO settings an exposue times t. If A an F ae the linea images output by the convete utility, we put them in the same space by computing: ISOF tf A = A ISOA t. (1) A Note that unless we inclue the supescipt, F an A efe to the non-linea vesions of the images. 4 Denoising an Detail Tansfe Ou enoising an etail tansfe algoithms ae esigne to enhance the ambient image using infomation fom the flash image. We pesent these two algoithms in Sections 4.1 an 4.2. Both algoithms assume that the flash image is a goo local estimato of the high fequency content in the ambient image. Howeve, this assumption oes not hol in shaow an specula egions cause by the flash, an can lea to atifacts. In Section 4.3, we escibe how to account fo these atifacts. The elationships between the thee algoithms ae epicte in Figue Denoising Reucing noise in photogaphic images has been a long-staning poblem in image pocessing an compute vision. One common solution is to apply an ege-peseving smoothing filte to the image such as anisotopic iffusion [Peona an Malik 1990] o bilateal filteing [Tomasi an Manuchi 1998]. The bilateal filte is a fast, non-iteative technique, an has been applie to a vaiety of poblems beyon image enoising, incluing tonemapping [Duan an Dosey 2002; Chouhuy an Tumblin 2003], sepaating illumination fom textue [Oh et al. 2001] an mesh smoothing [Fleishman et al. 2003; Jones et al. 2003]. Ou ambient image enoising technique also buils on the bilateal filte. We begin with a summay of Tomasi an Manuchi s basic bilateal filte an then show how to exten thei appoach to also consie a flash image when enoising an ambient image. k( p) = g( p p) g( Ap A p ). (3) p Ω The function g sets the weight in the spatial omain base on the istance between the pixels, while the ege-stopping function g sets the weight on the ange base on intensity iffeences. Typically, both functions ae Gaussians with withs contolle by the stana eviation paametes σ an σ espectively. We apply the bilateal filte to each RGB colo channel sepaately with the same stana eviation paametes fo all thee channels. The challenge is to set σ an σ so that the noise is aveage away but etail is peseve. In pactice, fo 6 megapixel images, we set σ to cove a pixel neighbohoo of between 24 an 48 pixels, an then expeimentally ajust σ so that it is just above the theshol necessay to smooth the noise. Fo images with pixel values nomalize to [0.0, 1.0] we usually set σ to lie between 0.05 an 0.1, o 5 to 10% of the total ange. Howeve, as shown in Figue 4(b), even afte caefully ajusting the paametes, the basic bilateal filte tens to eithe ove-blu (lose etail) o une-blu (fail to enoise) the image in some egions. Joint bilateal filte. We obseve in Section 2 that the flash image contains a much bette estimate of the tue high-fequency infomation than the ambient image. on this obsevation, we moify the basic bilateal filte to compute the ege-stopping function g using the flash image F instea of A. We call this technique the joint bilateal filte 2 : NR 1 Ap = g( p p) g( Fp Fp ) Ap, (4) k( p) p Ω NR whee k( p ) is moifie similaly. Hee A is the noise-euce vesion of A. We set σ just as we i fo the basic bilateal filte. Une the assumption that F has little noise, we can set σ to be vey small an still ensue that the ege-stopping function g( Fp F p ) will choose the pope weights fo neaby pixels an theefoe will not ove-blu o une-blu the ambient image. In pactice, we have foun that σ can be set to 0.1% of the total ange of colo values. Unlike basic bilateal filteing, we fix σ fo all images. The joint bilateal filte elies on the flash image as an estimato of the ambient image. Theefoe it can fail in flash shaows an speculaities because they only appea in the flash image. At the eges of such egions, the joint bilateal filte may une-blu the ambient image since it will own-weight pixels whee the filte stales these eges. Similaly, insie these egions, it may oveblu the ambient image. We solve this poblem by fist etecting flash shaows an specula egions as escibe in Section 4.3 an then falling back to basic bilateal filteing within these egions. Given the mask M 2 Eisemann an Duan [2004] call this the coss bilateal filte.

4 pouce by ou etection algoithm, ou impove enoising algoithm becomes: ( 1 ) NR NR A = M A + MA. (5) Results & Discussion. The esults of enoising with the joint bilateal filte ae shown in Figue 4(c). The iffeence image with the basic bilateal filte, Figue 4(), eveals that the joint bilateal filte is bette able to peseve etail while eucing noise. One limitation of both bilateal an joint bilateal filteing is that they ae nonlinea an theefoe a staightfowa implementation equies pefoming the convolution in the spatial omain. This can be vey slow fo lage σ. Recently Duan an Dosey [2002] use Fouie techniques to geatly acceleate the bilateal filte. We believe thei technique is also applicable to the joint bilateal filte an shoul significantly spee up ou enoising algoithm. (a) No- (c) Denoise via Joint Bilateal Filte (e) Detail Laye (f) (b) Denoise via Bilateal Filte Diffeence Detail Tansfe Figue 4: (a) A close-up of a flash/no-flash image pai of a Belgian tapesty. The no-flash image is especially noisy in the ake egions an oes not show the theas as well as the flash image. (b) Basic bilateal filteing peseves stong eges, but blus away most of the theas. (c) Joint bilateal filteing smoothes the noise while also etaining moe thea etail than the basic bilateal filte. () The iffeence image between the basic an joint bilateal filtee images. (e) We futhe enhance the ambient image by tansfeing etail fom the flash image. We fist compute a etail laye fom the flash image, an then (f) combine the etail laye with the image enoise via the joint bilateal filte to pouce the etail-tansfee image. () Filtee Signal Detail Laye Filtee/Signal Gaussian halo etail Bilateal Figue 5: (left) A Gaussian low-pass filte blus acoss all eges an will theefoe ceate stong peaks an valleys in the etail image that cause halos. (ight) The bilateal filte oes not smooth acoss stong eges an theeby euces halos, while still captuing etail To-Ambient Detail Tansfe While the joint bilateal filte can euce noise, it cannot a etail that may be pesent in the flash image. Yet, as escibe in Section 2, the highe SNR of the flash image allows it to etain nuances that ae ovewhelme by noise in the ambient image. Moeove, the flash typically povies stong iectional lighting that can eveal aitional suface etail that is not visible in moe unifom ambient lighting. The flash may also illuminate etail in egions that ae in shaow in the ambient image. To tansfe this etail we begin by computing a etail laye fom the flash image as the following atio: Detail F + ε F = F + ε, (6) whee F is compute using the basic bilateal filte on F. The atio is compute on each RGB channel sepaately an is inepenent of the signal magnitue an suface eflectance. The atio captues the local etail vaiation in F an is commonly calle a quotient image [Shashua an Riklin-Raviv 2001] o atio image [Liu et al. 2001] in compute vision. Figue 5 shows that the avantage of using the bilateal filte to compute F athe than a classic low-pass Gaussian filte is that we euce haloing. At low signal values, the flash image contains noise that can geneate spuious etail. We a ε to both the numeato an enominato of the atio to eject these low signal values an theeby euce such atifacts (an also avoi ivision by zeo). In pactice we use ε = 0.02 acoss all ou esults. To tansfe the NR etail, we simply multiply the noise-euce ambient image A Detail by the atio F. Figue 4(e-f) shows examples of a etail laye an etail tansfe. Just as in joint bilateal filteing, ou tansfe algoithm pouces a poo etail estimate in shaows an specula egions cause by the flash. Theefoe, we again ely on the etection algoithm escibe in Section 4.3 to estimate a mask M ientifying these egions an compute the final image as: Final NR Detail A = (1 M) A F + MA. (7) With this etail tansfe appoach, we can contol the amount of etail tansfee by choosing appopiate settings fo the bilateal filte paametes σ an σ use to ceate F. As we incease these filte withs, we geneate inceasingly smoothe vesions of Detail F an as a esult captue moe etail in F. Howeve, with excessive smoothing, the bilateal filte essentially euces to a Gaussian filte an leas to haloing atifacts in the final image. Results & Discussion. Figues 1, 4(f), an 6 8 show seveal examples of applying etail tansfe with enoising. Both the lamp (Figue 6) an pots (Figue 8) examples show how ou etail tansfe algoithm can a tue etail fom the flash image to the ambient image. The canlelit cave (Figue 1 an 7) is an exteme case fo ou algoithms because the ISO was oiginally set to 1600 an igitally booste up to 6400 in a post-pocessing step. In this

5 No- Oig. (top) Detail Tansfe (bottom) No- Detail Tansfe with Denoising Figue 6: An ol Euopean lamp mae of hay. The flash image captues etail, but is gay an flat. The no-flash image captues the wam illumination of the lamp, but is noisy an lacks the fine etail of the hay. With etail tansfe an enoising we maintain the wam appeaance, as well as the shap etail. In most cases, ou etail tansfe algoithm impoves the appeaance of the ambient image. Howeve, it is impotant to note that the flash image may contain etail that looks unnatual when tansfee to the ambient image. Fo example, if the light fom the flash stikes a sufaces at a shallow angle, the flash image may pick up suface textue (i.e. woo gain, stucco, etc.) as etail. If this textue is not visible in the oiginal ambient image, it may look o. Similaly if the flash image washes out etail, the ambient image may be ove-blue. Ou appoach allows the use to contol how much etail is tansfee ove the entie image. Automatically ajusting the amount of local etail tansfee is an aea fo futue wok. 4.3 No- Detail Tansfe with Denoising Long Exposue Refeence Figue 7: We captue a long exposue image of the wine cave scene (3.2 secons at ISO 100) fo compaison with ou etail tansfe with enoising esult. We also compute aveage mean-squae eo acoss the 16 bit R, G, B colo channels between the no-flash image an the efeence ( MSE) an between ou esult an the efeence ( MSE). Howeve, it is well known that MSE is not a goo measue of peceptual image iffeences. Visual compaison shows that although ou esult oes not achieve the fielity of the efeence image, it is substantially less noisy than the oiginal no-flash image. case, the exteme levels of noise foce us to use elatively wie Gaussians fo both the omain an ange kenels in the joint bilateal filte. Thus, when tansfeing back the tue etail fom the flash image, we also use elatively wie Gaussians in computing the etail laye. As a esult, it is possible to see small halos aoun the eges of the bottles. Nevetheless, ou appoach is able to smooth away the noise while peseving etail like the gentle winkles on the sofa an the glazing on the bottles. Figue 7 shows a compaison between a long exposue efeence image of the wine cave an ou etail tansfe with enoising esult. Detecting Shaows an Speculaities Light fom the flash can intouce shaows an speculaities into the flash image. Within flash shaows, the image may be as im as the ambient image an theefoe suffe fom noise. Similaly, within specula eflections, the flash image may be satuate an lose etail. Moeove, the bounaies of both these egions may fom high-fequency eges that o not exist in the ambient image. To avoi using infomation fom the flash image in these egions, we fist etect the flash shaows an speculaities. Shaows. Since a point in a flash shaow is not illuminate by the flash, it shoul appea exactly as it appeas in the ambient image. Ieally, we coul lineaize A an F as escibe in Section 3 an then etect pixels whee the luminance of the iffeence image F A is zeo. In pactice, this appoach is confoune by fou issues: 1) sufaces that o not eflect any light (i.e. with zeo albeo) ae etecte as shaows; 2) istant sufaces not eache by the flash ae etecte as shaows; 3) noise causes nonzeo values within shaows; an 4) inte-eflection of light fom the flash causes non-zeo values within the shaow. The fist two issues o not cause a poblem since the esults ae the same in both the ambient an flash images an thus whicheve image is chosen will give the same esult. To eal with noise an inte-eflection, we a a theshol when computing the shaow mask by looking fo pixels in which the iffeence between the lineaize flash an ambient images is small: 1 when F A τ Sha M Sha =. 0 else (8)

6 No- Oig. (top) Detail Tansfe (bottom) Detail Tansfe without Mask Shaow an Speculaity Mask Detail Tansfe using Mask No- Detail Tansfe with Denoising Figue 8: (top ow) The flash image oes not contain tue etail infomation in shaows an specula egions. When we naively apply ou enoising an etail tansfe algoithms, these egions geneate atifacts as inicate by the white aows. To pevent these atifacts, we evet to basic bilateal filteing within these egions. (bottom ow). The ak bown pot on the left is extemely noisy in the no-flash image. The geen pot on the ight is also noisy, but as shown in the flash image it exhibits tue textue etail. Ou etail tansfe technique smoothes the noise while maintaining the textue. Also note that the flash shaow/speculaity etection algoithm popely masks out the lage specula highlight on the bown pot an oes not tansfe that etail to the final image. We have evelope a pogam that lets uses inteactively ajust the theshol value τ Sha an visually veify that all the flash shaow egions ae popely captue. Noise can contaminate the shaow mask with small speckles, holes an agge eges. We clean up the shaow mask using image mophological opeations to eoe the speckles an fill the holes. To pouce a consevative estimate that fully coves the shaow egion, we then ilate the mask. Speculaities. We etect specula egions cause by the flash using a simple physically motivate heuistic. Specula egions shoul be bight in F an shoul theefoe satuate the image senso. Hence, we look fo luminance values in the flash image that ae geate than 95% of the ange of senso output values. We clean, fill holes, an ilate the specula mask just as we i fo the shaow mask. Final Mege. We fom ou final mask M by taking the union of the shaow an specula masks. We then blu the mask to feathe its eges an pevent visible seams when the mask is use to combine egions fom iffeent images. Results & Discussion. The esults in Figues 1 an 6 8 wee geneate using this flash atifact etection appoach. Figue 8 (top ow) illustates how the mask coects flash shaow atifacts in the etail tansfe algoithm. In Figue 1 we show a failue case of ou algoithm. It oes not captue the stipe specula highlight on the cente bottle an theefoe this highlight is tansfee as etail fom the flash image to ou final esult. Although both ou shaow an specula etection techniques ae base on simple heuistics, we have foun that they pouce goo masks fo a vaiety of examples. Moe sophisticate techniques evelope fo shaow an specula etection in single images o steeo pais [Lee an Bajcsy 1992; Funka-Lea an Bajcsy 1995; Swaminathan et al. 2002] may povie bette esults an coul be aapte fo the case of flash/no-flash pais. 5 White Balancing Although peseving the oiginal ambient illumination is often esiable, sometimes we may want to see how the scene woul appea une a moe white illuminant. This pocess is calle white-balancing, an has been the subject of much stuy [Aams et al. 1998]. When only a single ambient image is acquie, the ambient illumination must be estimate base on heuistics o use input. Digital cameas usually povie seveal white-balance moes fo iffeent envionments such as sunny outoos an fluoescent lighting. Most often, pictues ae taken with an auto moe, wheein the camea analyzes the image an computes an imagewie aveage to infe ambient colo. This is, of couse, only a heuistic, an some eseaches have consiee semantic analysis to etemine colo cast [Schoee an Mose 2001]. A flash/no-flash image pai enables a bette appoach to white balancing. Ou wok is heavily inspie by that of DiCalo et al. [2001], who wee the fist to consie using flash/no-flash pais fo illumination estimation. They infe ambient illumination by pefoming a iscete seach ove a set of 103 illuminants to fin the one that most closely matches the obseve image pai. We simplify this appoach by fomulating it as a continuous optimization poblem that is not limite by this iscete set of illuminants. Thus, ou appoach equies less setup than theis. We can think of the flash as aing a point light souce of known colo to the scene. By setting the camea white-balance moe to flash (an assuming a calibate camea), this flash colo shoul appea as efeence white in the acquie images. The iffeence image = F A coespons to the illumination ue to the flash only, which is popotional to the suface albeo at each pixel p. Note that the albeo estimate has unknown scale, because both the istance an oientation of the suface ae unknown. Hee we ae assuming eithe that the suface is iffuse o that its specula colo matches its iffuse colo. As a

7 Oiginal No- Estimate ambient illumination White-Balance Figue 9: (left) The ambient image (afte enoising an etail tansfe) has an oange cast to it. The insets show the estimate ambient illumination colos C an the estimate oveall scene ambience. (ight) Ou white-balancing algoithm shifts the colos an emoves the oange coloing (No-) () Figue 10: An example of continuous flash ajustment. We can extapolate beyon the oiginal flash/no-flash pai. counte-example, this is not tue of plastics. Similaly, semitanspaent sufaces woul give eoneous estimates of albeo. Since the suface at pixel p has colo Ap in the ambient image an the scale albeo p, we can estimate the ambient illumination at the suface with the atio: Cp = p Ap, (9) which is compute pe colo channel. Again, this estimate colo C p has an unknown scale, so we nomalize it at each pixel p (see inset Figue 9). Ou goal is to analyze C p at all image pixels to infe the ambient illumination colo c. To make this infeence moe obust, we isca pixels fo which the estimate has low confience. We can affo to o this since we only nee to eive a single colo fom millions of pixels. Specifically, we ignoe pixels fo which eithe Ap < τ 1 o the luminance of p < τ 2 in any channel, since these small values make the atio less eliable. We set both τ 1 an τ 2 to about 2% of the ange of colo values. Finally, we compute the ambient colo estimate c fo the scene as the mean of C p fo the non-iscae pixels. (An altenative is to select c as the pincipal component of C, obtaine as the eigenvecto of C T C with the lagest eigenvalue, an this gives a simila answe.) Having infee the scene ambient colo c, we white-balance the image by scaling the colo channels as: AWB p = 1 Ap. c White-balancing is a challenging poblem because the peception of white epens in pat on the aaptation state of the viewe. Moeove, it is unclea when white-balance is esiable. Howeve we believe that ou estimation appoach using the known infomation fom the flash can be moe accuate than techniques base on single-image heuistics. 6 Again, the computation is pefome pe colo channel. Results & Discussion. Figue 9 shows an example of white balancing an ambient image. The white balancing significantly changes the oveall hue of the image, setting the colo of the woo table to a yellowish gay, as it woul appea in white light. In infeing ambient colo c, one coul also pune outlies an look fo spatial elationships in the image C. In aition, the scene may have multiple egions with iffeent ambient colos, an these coul be segmente an pocesse inepenently. Continuous Ajustment When taking a flash image, the intensity of the flash can sometimes be too bight, satuating a neaby object, o it can be too im, leaving mi-istance objects une-expose. With a flash an non-flash image pai, we can let the use ajust the flash intensity afte the pictue has been taken. We have exploe seveal ways of intepolating the ambient an flash images. The most effective scheme is to convet the oiginal flash/no-flash pai into YCbC space an then linealy intepolate them using: F Ajuste = (1 α ) A + (α ) F. (11) To povie moe use contol, we allow extapolation by letting the paamete α go outsie the nomal [0,1] ange. Howeve, we only extapolate the Y channel, an estict the Cb an C channel intepolations to thei extema in the two oiginal images, to pevent excessive istotion of the hue. An example is shown in Figue (10) 1.5 Re-Eye Coection Re-eye is a common poblem in flash photogaphy an is ue to light eflecte by a well vasculaize etina. Fully automate eeye emoval techniques usually assume a single image as input an ely on a vaiety of heuistic an machine-leaning techniques to localize the e eyes [Gaubatz an Ulichney 2002; Patti et al. 1998]. Once the pupil mask has been etecte, these techniques aken the pixels within the mask to make the images appea moe natual. We have evelope a e-eye emoval algoithm that consies the change in pupil colo between the ambient image (whee it is usually vey ak) an the flash image (whee it may be e). We convet the image pai into YCbC space to ecoelate luminance

8 fom chominance an compute a elative eness measue as follows: R = F A. (12) C C We then initially segment the image into egions whee: R > τ Eye. (13) We typically set τ Eye to 0.05 so that the esulting segmentation efines egions whee the flash image is ee than the ambient image an theefoe may fom potential e eyes. The segmente egions also ten to inclue a few locations that ae highly satuate in the C channel of the flash image but ae elatively ak in the Y channel of the ambient image. Thus, if µ R an σ R enote the mean an stana eviation of the eness R, we look fo see pixels whee: No- Re-Eye Coecte R > max[0.6, µ + 3 σ ] an AY < τ Dak. (14) R R We usually set τ Dak = 0.6. If no such see pixels exist, we assume the image oes not contain e-eye. Othewise, we use these see pixels to look up the coesponing egions in the segmentation an then apply geometic constaints to ensue that the egions ae oughly the same size an elliptical. In paticula, we compute the aea of each egion an isca lage outlies. We then check that the eccenticity of the egion is geate than These egions then fom a e-eye pupil mask. Finally to coect these e-eye egions we use the technique of Patti et al.[1998]. We fist emove the highlights o glints in the pupil mask using ou flash speculaity etection algoithm. We then set the colo of each pixel in the mask to the gay value equivalent to 80% of its luminance value. This appoach popely akens the pupil while maintaining the specula highlight which is impotant fo maintaining ealism in the coecte output. Results & Discussion. Figue 11 illustates ou e-eye coection algoithm with two examples. The secon example shows that ou algoithm pefoms well even when the e-eye is subtle. In both examples ou algoithm is able to istinguish the pupils fom the eish skin. Moeove, the specula highlight is peseve an the eye shows no unnatual iscoloation. Both of these examples wee automatically aligne using the appoach of Szeliski an Shum [1997]. Since colo noise coul invaliate ou chomaticity compaison, we assume a elatively noise fee ambient image, like the ones geneate by ou enoising algoithm. 8 Futue Wok an Conclusions While we have evelope a vaiety of applications fo flash/noflash image pais, we believe thee emain many iections fo futue wok. In some cases, the look of the flash image may be pefeable to the ambient image. Howeve, the flash shaows an speculaities may be etacting. While we have evelope algoithms fo etecting these egions, we woul like to investigate techniques fo emoving them fom the flash image. shaows often appea at epth iscontinuities between sufaces in the scene. Using multiple flash photogaphs it may be possible to segment foegoun fom backgoun. Raska et al. [2003] have ecently exploe this type of appoach to geneate non-photoealistic eneings. Motion blu is a common poblem fo long-exposue images. It may be possible to exten ou etail tansfe technique to e-blu such images. Recent wok by Jia et al. [2004] is beginning to exploe this iea. No- Closeup with Faint Re-Eye Re-Eye Coecte Figue 11: Examples of e-eye coection using ou appoach. Although the e eye is subtle in the secon example, ou algoithm is still able to coect the poblem. We use a Nikon CoolPix 995 to acquie these images. While ou appoach is esigne fo consume-gae cameas, we have not yet consiee the joint optimization of ou algoithms an the camea hawae esign. Fo example, iffeent illuminants o illuminant locations may allow the photogaphe to gathe moe infomation about the scene. An exciting possibility is to use an infae flash. While infae illumination yiels incomplete colo infomation, it oes povie high-fequency etail, an oes so in a less intusive way than a visible flash. We have emonstate a set of applications that combine the stengths of flash an no-flash photogaphs to synthesize new images that ae of bette quality than eithe of the oiginals. The acquisition poceue is staightfowa. We theefoe believe that flash/no-flash image pais can contibute to the palette of image enhancement options available to igital photogaphes. We hope that these techniques will be even moe useful as cameas stat to captue multiple images evey time a photogaphe takes a pictue. Acknowlegements We wish to thank Mac Levoy fo mentoing the ealy stages of this wok an consistently setting an example fo scientific igo. Steve Maschne gave us many pointes on the fine etails of igital photogaphy. We thank Chis Begle an Stan Bichfiel fo thei encouagement an Rico Malva fo his avice. Mike Baun collaboate on an ealy vesion of these ieas. Geog woul like to thank his paents fo maintaining Castle Pymont, which povie a beautiful setting fo many of the images in the pape. Finally, we thank the anonymous eviewes fo thei exceptionally etaile feeback an valuable suggestions.

9 Refeences ADAMS, J., PARULSKI, K. AND SPAULDING, K., Colo pocessing in igital cameas. IEEE Mico, 18(6), pp AGARWALA, A., DONTCHEVA, M., AGRAWALA, M., DRUCKER, S., COLBURN, A., CURLESS, B., SALESIN, D. H., AND COHEN, M., Inteactive Digital Photomontage. ACM Tansaction on Gaphics, 23(3), in this volume. AKERS, D., LOSASSO, F., KLINGNER, J., AGRAWALA, M., RICK, J., AND HANRAHAN, P., Conveying shape an featues with image-base elighting. IEEE Visualization 2003, pp CHOUDHURY, P., AND TUMBLIN, J., The tilateal filte fo high contast images an meshes. In Euogaphics Reneing Symposium, pp DEBEVEC, P. E., AND MALIK, J., Recoveing high ynamic ange aiance maps fom photogaphs. ACM SIGGRAPH 97, pp DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H., SAROKIN, W. AND SAGAR, M., Acquiing the eflectance fiel of the human face. ACM SIGGRAPH 2000, pp DICARLO, J. M., XIAO, F., AND WANDELL, B. A., Illuminating illumination. Ninth Colo Imaging Confeence, pp DURAND, F., AND DORSEY, J., Fast bilateal filteing fo the isplay of high-ynamic-ange images. ACM Tansactions on Gaphics, 21(3), pp EISEMANN, E., AND DURAND, F., photogaphy enhancement via intinsic elighting. ACM Tansactions on Gaphics, 23(3), in this volume. FLEISHMAN, S., DRORI, I. AND COHEN-OR, D., Bilateal mesh enoising. ACM Tansaction on Gaphics, 22(3), pp FUNKA-LEA, G., AND BAJCSY, R., Active colo an geomety fo the active, visual ecognition of shaows. Intenational Confeence on Compute Vision, pp GAUBATZ, M., AND ULICHNEY, R., Automatic e-eye etection an coection. IEEE Intenational Confeence on Image Pocessing, pp HAEBERLI, P., Synthetic lighting fo photogaphy. Gafica Obscua, JIA, J., SUN, J., TANG, C.-K., AND SHUM, H., Bayesian coection of image intensity with spatial consieation. ECCV 2004, LNCS 3023, pp JONES, T.R., DURAND, F. AND DESBRUN, M., Non-iteative featue peseving mesh smoothing. ACM Tansactions on Gaphics, 22(3), pp KANG, S. B., UYTTENDAELE, M., WINDER, S., AND SZELISKI, R., High ynamic ange vieo. ACM Tansactions on Gaphics, 22(3), pp KODAK, CCD Image Senso Noise Souces. Application Note MPT/PS LEE, S. W., AND BAJCSY, R., Detection of speculaity using colo an multiple views. Euopean Confeence on Compute Vision, pp LIU, Z., SHAN., Y., AND ZHANG, Z., Expessive expession mapping with atio images. ACM SIGGRAPH 2001, pp MASSELUS, V., DUTRE, P., ANRYS, F., The fee-fom light stage. In Euogaphics Reneing Symposium, pp OH, B.M., CHEN, M., DORSEY, J. AND DURAND, F., Imagebase moeling an photo eiting. ACM SIGGRAPH 2001, pp PATTI, A., KONSTANTINIDES, K., TRETTER, D. AND LIN, Q., Automatic igital eeye euction. IEEE Intenational Confeence on Image Pocessing, pp PERONA, P., AND MALIK, J., Scale-space an ege etection using anisotopic iffusion. IEEE Tansactions on Patten Analysis an Machine Intelligence, 12(7), pp RASKAR, R., YU, J. AND ILIE, A., A non-photoealistic camea: Detecting silhouettes with multi-flash. ACM SIG- GRAPH 2003 Technical Sketch. SCHROEDER, M., AND MOSER, S., Automatic colo coection base on geneic content-base image analysis. Ninth Colo Imaging Confeence, pp SHASHUA, A., AND RIKLIN-RAVIV, T., The quotient image: class base e-eneing an ecognition with vaying illuminations. IEEE Tansactions on Patten Analysis an Machine Intelligence, 23(2), pp SWAMINATHAN, R., KANG, S. B., SZELISKI, R., CRIMINISI, A. AND NAYAR, S. K., On the motion an appeaance of speculaities in image sequences. Euopean Confeence on Compute Vision, pp. I: SZELISKI, R., AND SHUM, H., Ceating full view panoamic image mosaics an envionment maps. ACM SIGGRAPH 97, pp TOMASI, C., AND MANDUCHI, R., Bilateal filteing fo gay an colo images. IEEE Intenational Confeence on Compute Vision, pp