Exposing Photo Manipulation from Shading and Shadows

Transcription

1 Exposing Photo Mnipultion from Shing n Shows Eri Kee, Columi University Jmes F. O Brien, University of Cliforni, Berkeley n Hny Fri, Drtmouth College We esrie metho for eteting physil inonsistenies in lighting from the shing n shows in n imge. This metho imposes multitue of shing- n show-se onstrints on the projete lotion of istnt point light soure. The onsisteny of olletion of suh onstrints is pose s liner progrmming prolem. A fesile solution inites tht the omintion of shing n shows is physilly onsistent, while filure to fin solution provies eviene of photo tmpering. Ctegories n Sujet Desriptors: I.. [Artifiil Intelligene]: Vision n Sene Unerstning Sene Anlysis; I.. [Computer Grphis]: Methoology n Tehniques Imge Forensis; K..m [Computing Milieux]: Computers n Soiety Misellneous Aitionl Key Wors n Phrses: Imge forensis, photo mnipultion, imge mnipultion, forgery etetion, lighting, shows. INTRODUCTION Avnes in photo eiting softwre hve me it possile to rete visully ompelling photogrphi forgeries. The uiquity of these imges hs unermine our trust in photogrphy, n impts lw enforement, ntionl seurity, the mei, vertising, ommere, n more. The fiel of photo forensis hs emerge to help restore trust in photogrphy [Fri 9; Roh et l. ]. In the sene of n emee wtermrk or signture, forensi methos ssume tht imge mnipultion will isrupt sttistil, geometri, or physil properties in n imge. To the extent tht suh perturtions n e quntifie n etete, forgeries n e ojetively invlite. For exmple, formt-speifi nlyses This pper hs een epte to ACM TOG for pulition. This work ws supporte y NSF grnts IIS-9, IIS-, n CNS-9, Intel s Siene n Tehnology Center for Visul Computing, n y gifts from Aoe Systems, Mirosoft, NVIDIA, n Pixr. Authors emil resses: erikee@s.olumi.eu, jo@erkeley.eu, fri@s.rtmouth.eu Permission to mke igitl or hr opies of prt or ll of this work for personl or lssroom use is grnte without fee provie tht opies re not me or istriute for profit or ommeril vntge n tht opies show this notie on the first pge or initil sreen of isply long with the full ittion. Copyrights for omponents of this work owne y others thn ACM must e honore. Astrting with reit is permitte. To opy otherwise, to repulish, to post on servers, to reistriute to lists, or to use ny omponent of this work in other works requires prior speifi permission n/or fee. Permissions my e requeste from Pulitions Dept., ACM, In., Penn Plz, Suite, New York, NY - USA, fx + () 9-, or permissions@m.org. ACM -//-ARTXXX $. DOI./XXXXXXX.YYYYYYY Originl imge opyright Eri Shrmnn, Fig.. The projete lotion of the light soure is onstrine y shing long oluing ontours n y st shows. The intersetion of the shing onstrints (outline in lk) is inonsistent with the intersetion of the show onstrints (outline in white), reveling this imge to e fke (not surprisingly). ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

2 XXX : E. Kee, J. O Brien, H. Fri exploit rtifts tht re introue when JPEG imge is ompresse multiple times [Popesu n Fri ; Mhin n Si 9; Kee et l. ]. Pixel-se nlyses etet low-level orreltions tht re introue y loning [Fririh et l. ], re-sizing [Popesu n Fri ], or non-liner filtering [Lin et l. ]. Sensor-se nlyses n etet inonsistenies in hromti errtions [Johnson n Fri ], olor filter rry interpoltion [Popesu n Fri ; Kirhner ], or sensor noise [Chen et l. ; Fririh 9]. Aitionlly, physilly se nlysis n etet inonsistenies in refletions [O Brien n Fri ], lighting [Johnson n Fri ; Kee n Fri ; Johnson n Fri ], or shows [Kee et l. ]. Any one of these tehniques my e pplile to only suset of imges, n my even e vulnerle to ounter ttk. Working in onert, however, ll of these tehniques mke it inresingly iffiult n time onsuming ut never impossile for forger to rete ompelling fke. We propose new physilly se forensi metho tht simultneously nlyses shing n shows in photo. Suh methos re ttrtive euse shing n shows result from the -D intertion of lighting n geometry, n n therefore e iffiult to urtely moify using -D photo eiting softwre. In ition, these effets re often mesurle even in low resolution n low qulity imges. It hs een shown tht the humn visul system n e surprisingly t eteting even lrge physil inonsistenies in lighting n shows [Joson n Werner ; Ostrovsky et l. ; Fri n Brvo ]. As result, forger is likely to leve ehin inonsistent shing n shows, n visul inspetion y forensi nlyst is likely to e sujetive or unrelile. In ontrst our propose geometri nlysis n ojetively etet inonsistenies. It hs een well estlishe tht urtely estimting lighting from single imge is iffiult. This estimtion is me more iffiult when n imge is of low resolution n poor qulity, n me even more omplex when n imge is omposite of multiple photos tken uner vrying lighting. In forensi setting, however, our gol is not neessrily to estimte the lotion of the light, ut rther to etermine if the shing n shows in photo re onsistent with single light soure. To this en, we seek only to onstrin the projete lotion of the light soure in the imge plne. By omining multitue of potentilly wek onstrints, we n etermine if the shing n shows in photo re physilly onsistent. Shown in Fig. is n exmple of our shing n show nlysis pplie to flsifie photo. A olletion of shing n show onstrints restrit the projete lotion of the light soure. The shing onstrints re lele, n the show onstrints re lele. Mgnifie views of eh onstrint re shown in the ottom left. The intersetion of the four shing onstrints is outline in lk (ottom right pnel). The intersetion of the three show onstrints is outline in white (ottom right pnel). Beuse these regions o not interset, we n prove tht there oes not exist single istnt point light soure tht n explin the shing n shows in this imge. It n e ojetively rgue, therefore, tht the shing n shows in Fig. re physilly inonsistent, orretly reveling this photo to e fke.. RELATED WORK A lrge oy of work hs een estlishe for the estimtion of lighting from the shing on n ojet. Stnr tehniques often ssume, s we o, single istnt point light soure illuminting Lmertin surfe of onstnt refletne. Within the vision literture, methos inlue shpe-from-shing [Zheng n Chellpp 99; Hougen n Ahuj 99; Kim et l. 99], n inverse lighting in whih the -D geometry is ssume to e known [Mrshner n Greenerg 99; Zhng n Yng ; Hr et l. ; Zhou n Kmhmettu ]. Some methos tke vntge of oth shing n show ues, [Li et l. ; Wng n Smrs ], ut, unlike our pproh, require omplete -D geometri moels. Methos tht o not require known -D geometry hve use mnully-pproximte -D geometry [Krsh et l. ]. When -D moels re not ville or nnot e esily estimte, -D oluing ontours n e use to estimte -D suset of the -D light iretion [Nillius n Eklunh ]. In the forensis literture, shing n shows hve een iniviully exploite to etet photo tmpering [Johnson n Fri ; ; Zhng et l. 9; Kee n Fri ; Liu et l. ; Riess n Angelopoulou ]. The omintion of shing n show hs een onsiere for estimting -D light iretion [Johnson et l. ], n for estimting the lotion of light soures [Stork n Johnson ; ]. Their work, emonstrte in the nlysis of pintings, ssumes orthogrphi projetion whih n e n overly limiting ssumption in mny prtil situtions. Our pproh iffers in three funmentl wys from these previous methos. First, inste of expliitly estimting lighting, we seek only to onstrin the light soure. This llows us to nlyze wie vriety of imges, even those in whih there is insuffiient informtion to uniquely estimte lighting. Seon, we formulte the estimtion of lighting from shing uner moel of liner perspetive. In previous work [Johnson n Fri ; ; Johnson et l. ; Stork n Johnson ; ], orthogrphi projetion ws ssume to simplify estimtion of surfe normls long oluing ounries (y ssuming tht the z-omponent of the -D surfe norml is zero). This ssumption, however, les to errors in the estimte lighting when ojets re on opposing sies of n imge, n prtiulrly when the projete lotion of the light iffers signifintly etween the two moels. An thir, euse the nlysis of st shows [Kee et l. ] impliitly ssumes moel of perspetive projetion, our shing-se onstrints n e omine with show onstrints. In ontrst to [Kee et l. ], this omintion fores woul-e forger to onsier n potentilly orret oth the shing n shows, rther thn the shows lone. This mkes the retion of forgery more iffiult n time onsuming. Our forensi tehnique hs severl vntges in tht it mkes miniml ssumptions out the unerlying sene geometry, mkes reltively few ssumptions out the sene (single istnt point light soure illuminting Lmertin surfe of onstnt refletne), ssumes more plusile moel of perspetive projetion, omines oth shing n show ues within single unifie frmework, n, perhps most importntly, n e pplie to roer rnge of imges in whih it woul e iffiult or impossile to expliitly estimte lighting.. METHODS In this setion we formulte how the shing on n ojet onstrins the lotion of istnt point light soure in -D n in the projete -D imge. We show tht these -D shing onstrints n e omine with -D show onstrints within liner frmework to etermine if the shing n shows in n imge re physilly onsistent with single light soure. Throughout, the following nottionl onventions re use. Slr vlues re lowerse letters (), vetors re ol lowerse ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

3 Exposing Photo Mnipultion from Shing n Shows XXX : letters (v), n mtries re ol pitl letters (A). Itliize vetors enote quntities in the -D imge plne (v), while romn vetors enote quntities in the -D worl or higher imensions (v). Iniviul elements of vetors or olumns of mtries re referene y susripts: v k is the k th element of vetor v, n k is k th olumn of mtrix A. Vetors tht represent homogeneous quntities in the imge plne re enote s ṽ n their Crtesin equivlent is simply v.. Shing onstrints (-D) Uner n infinitely istnt point light soure, the rine r of n imge pth tht orrespons to surfe with onstnt Lmertin refletne n norml n = `n x n y n z T is given y r = n s +, () where s = `s x s y s z T speifies the iretion to the light soure, n the mgnitue of s is proportionl to the rightness of the light. The onstnt term is the so-lle mient term whih pproximtes iniret illumintion. Note tht this expression ssumes tht the ngle etween the surfe norml n light is less thn 9. The omponents of this lighting moel n e estimte from k pthes with known surfe normls. The equtions for eh surfe norml n orresponing rine re pke into the following liner system: n T n T «s C = r (). A n T k Ne = r, () where r is vetor of oserve rine for eh pth. The lighting prmeters e n e estimte using stnr lest squres: e = (N T N) N T r, () where e enotes the estimte prmeters, n its first three omponents speify the rightness-sle light iretion, s. The normlize light iretion, s / s, n e visulize s point on the surfe of the unit sphere, Fig. (). When the mtrix N is well onitione, this estimtion is strightforwr. In prtie, however, the mtrix N my e poorly onitione or rnk efiient ue to limite numer, or rnge, of known surfe normls. In suh situtions it is not possile to etermine n ext solution, e, of the lighting prmeters ut it is still possile to onstrin the solution to spe of possile solutions. Consier the singulr vlue eomposition (SVD) of the liner system in Eqution (): UΛV T e = r. () This eomposition projets the lighting prmeters e onto n orthonorml sis V: V = `v «w w v v v = w w () where w i re vetors n i re slr vlues. Note tht w i n i form sis for the rightness-sle light iretion s n mient omponent. After projetion onto vetors v i, e is sle y the igonl mtrix Λ of singulr vlues λ i, n then trnsforme into imge rine y the mtrix U. Using the onvention tht the singulr vlues, λ,...,λ, re sorte from lrgest to smllest, the well-onitione imensions of the solution spe re given y the olumns of Λ n V tht orrespon to singulr vlues λ i /λ. The lighting prmeters e n e estimte within the suspe spnne y the well-onitione vetors v i of the singulr vlue eomposition: e = N + r, () where N + enotes the pseuo-inverse. When one singulr vlue is egenerte, λ /λ, possile lighting solutions re given y e + αv, where e is the solution to the liner system in Eqution (), α is slr, n v is the fourth olumn of mtrix V. This onstrins the rightness-sle iretion of the light soure s to line in -D: s = s + αw, () where s is the first three omponents of e orresponing to the light iretion, n w is the first three omponents of singulr vetor v. This line psses through the point s n is prllel to w. The orresponing set of normlize light iretions, s/ s, forms semi-irulr r, whih is the projetion of the line onto the unit sphere, Fig. (). When two singulr vlues re egenerte, λ /λ n λ /λ, possile lighting solutions re given y e + αv + βv, where e is the solution to the liner system in Eqution (), n oth α n β re slrs. This onstrins the rightness-sle iretion of the light soure s to plne in -D: s = s + αw + βw, (9) whih psses through s n is prllel to oth w n w. The orresponing set of normlize light iretions forms hemispheril surfe, whih is the projetion of the plne onto the unit sphere, Fig. (). When three singulr vlues re egenerte, λ /λ, λ /λ, n λ /λ, the null spe of the liner system spns the full -D spe, thus proviing no onstrint on the light iretion s.. Shing onstrints (-D) In the previous setion we esrie how the shing on n ojet n onstrin the -D iretion of istnt point light. In this setion we evelop representtion of these onstrints tht n e omine with onstrints from shows [Kee et l. ] (etile in Setion.). Light iretions in -D n e trete s points t infinity n projete into the -D imge plne. Constrints on the -D lighting iretion n, therefore, e expresse s onstrints on the projete light lotion in the -D imge plne. For simpliity, ut without loss of generlity, let the surfe normls n n iretion of the light soure s e speifie in the mer oorinte system in whih the mer fes own the z xis. Uner n ielize perspetive mer with fol length f n imge enter, the light soure t infinity in iretion s projets to point in the imge plne: s f x f y A s () = Ks () where the imge point s is speifie in homogeneous oorintes, n K is the intrinsi mtrix of the mer. When the liner system, Eqution (), is fully onitione, the iretion of the light soure, s, n e unmiguously estimte, ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

4 XXX : E. Kee, J. O Brien, H. Fri () () () () (e) (f ) Fig.. Shing onstrints in -D (top) n -D (ottom): () when the liner system in Eqution () is fully onitione, the -D iretion of the light soure, s, n e uniquely etermine; (-) if the liner system is not fully onitione, then the -D iretion of the light soure is onstrine to semi-irulr r or hemisphere, epening on the rnk of the liner system (w n w re the vetors orresponing to the singulr vlues of the liner system). Shown in pnels ()-(f) is the projetion of these onstrints onto the imge plne, where the mer s enter of projetion is enote y. s = s, Fig. (). This onstrins the projete lotion, s, ofthe light soure s to single point in the imge: s = Ks. () Shown in Fig. () is the projetion, s,ofthe-d iretion, s, onto -D imge plne (the mer enter of projetion is enote y ). When one singulr vlue is egenerte, the rightness-sle light iretion is onstrine to line in -D, s = s + αw, whih orrespons to hlf-irle of iretions, Fig. (). These iretions projet to line in the imge plne: s = K(s + αw ). () Shown in Fig. (e) is the projetion of the -D onstrint ( hlfirle of iretions) onto -D imge plne. Note tht light iretions ehin the mer s enter of projetion (lue r) re projete into the lower prt of the line (lue line segment), while light iretions in front of the mer (re r) re projete into the upper prt of the line (re line segment). These two line segments meet t the imge of vetor w, Kw (enote y in the imge plne). This line n its segments n e esrie y position p n iretion in the imge: p = Kw () = sign(s z )(p s ), () where s is the projete lotion of the lest-squres solution of the light iretion, s, n s z is its z-omponent. Diretion ientifies the line segment onto whih light iretions in front of the mer projet (see Appenix A for erivtion). When two singulr vlues re egenerte, the rightnesssle light iretion is onstrine to plne in -D, s = s + αw + βw, whih orrespons to hlf-sphere of iretions, Fig. (). These iretions projet to plne in the imge: s = K(s + αw + βw ). () Although this plne enompsses the entire imge, it onsists of two hlf-plnes, Fig. (f): one for light iretions in front of the mer (re), n one for iretions ehin the mer (lue). These hlf-plnes re esrie y position p n iretion in the imge: p = Kw () = sign` h s ` h h T, () where h is the homogeneous line through the imge of w n w : h = Kw Kw. (9) Diretion ientifies the hlf-plne into whih light iretions in front of the mer projet (see Appenix A for erivtion). Finlly, s note previously, when three singulr vlues re egenerte, the light iretion is unonstrine in -D n thus provies no onstrint on the projete lotion of the light in the imge plne. ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

5 Exposing Photo Mnipultion from Shing n Shows XXX : () () () Fig.. Geometri onfigurtions tht le to point (), line (), n hlfplne () onstrints on the projete lotion of the light soure. Shown in eh pnel is the projetion of -D ojet (right) into the imge plne (the mer s enter of projetion is enote y ). A point onstrint n e speifie when full -D moel is ville. Surfe normls long n extene portion of n oluing ontour provie line onstrint, n normls long smll portion of the oluing ontour provie hlf-plne onstrint. (Note tht, euse the light is ehin the mer, its projete lotion (yellow ot) is inverte uner perspetive projetion.) Fig. illustrtes how ifferent onfigurtions of surfe normls n le to the point, line, n hlf-plne onstrints on the projete lotion of the light soure. Shown in pnel () is the se where full -D moel of the ojet is known, yieling full rnk mtrix N in Eqution () n point onstrint. Shown in pnel () is the se where only pproximtely oplnr normls long n extene portion of the ojet s oluing ounry re known. These oluing normls yiel one egenerte singulr vlue in N, prouing line onstrint. Lstly, shown in pnel () is se where only normls long smll portion of the ojet s oluing ounry re speifie, yieling two egenerte singulr vlues in N n hlf-plne onstrint.. Stisfiility of multiple onstrints In forensi setting, we woul like to etermine if the shing on ojets in photo re physilly onsistent with single light soure. We therefore omine the shing onstrints from multiple ojets in sene into single liner system to etermine if they re olletively stisfile. Consier first the se in whih the light soure is in front of the mer. When the light position n e onstrine to point in the imge, Eqution (), the following equlity onstrint (in homogeneous oorintes) is ple on the lotion of the light soure s: s = Ks. () When the light position n e onstrine to line, Eqution (), the following onstrint is ple on s (in rtesin oorintes): s = α + p () α, () where p n re given y Equtions ()-(). Lstly, when the light position is onstrine y hlf-plne, Eqution (), the following inequlity onstrint is ple on s: s p, () where p n re given y Equtions ()-(9). Eh of these onstrints re liner, n n therefore e omine into liner progrm to hek for mutul onsisteny. A vile solution to the liner progrm mens tht the shing in n imge is onsistent, while filure to fin vile solution mens tht one or more of the onstrints re inonsistent with single light soure. When the light soure is ehin the mer, its projetion into the imge is inverte y liner perspetive projetion. To onten with this possiility, seon liner progrm is onstrute in whih ll onstrint inequlities re reverse. The shing is etermine to e onsistent if either liner progrm is stisfile.. Show onstrints (-D) The previous setions hve esrie how the shing of n ojet n e use to onstrin the projete lotion of point light soure. In ition, st n tthe shows lso onstrin the projete lotion of point light soure [Kee et l. ]. These -D show onstrints tke the form of liner inequlities, n n therefore e omine with our shing onstrints to further onstrin the projete lotion of the light soure. We riefly summrize the onstrints provie y st n tthe shows. Cst shows. Consier ry tht onnets point in showe region to its orresponing point on the show-sting ojet, Fig. (soli line). In the -D sene, this ry intersets the light soure. In -D imge of the sene rete uner liner perspetive, the projetion of this ry remins stright line tht must onnet the imges of the show n ojet points, n must interset the projete lotion of the light soure. These onstrints hol regrless of the geometry of the ojet n the surfe onto whih the shows re st, n hol for either n infinitely istnt or lol light. Cst shows onstrin the projete lotion of the light soure s to line: s = α + p () α, () where p is point in show, n is vetor onneting p to the orresponing point on the ojet. Notie tht this onstrint hs ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

6 XXX : E. Kee, J. O Brien, H. Fri Fig.. A st n tthe show onstrint. The projete lotion of the light soure lies on line tht onnets point in the one s st show with its orresponing point on the one. An tthe show on the one onstrins the projete lotion of the light soure to e in hlf plne, speifie y the orienttion n polrity of the show termintor. the sme form s the line onstrint provie y shing, Equtions ()-(). As suh, these onstrints n e semlessly omine with the shing onstrints, Setion.. Atthe shows. Atthe shows our when ojets olue the light from themselves, leving portion of the ojet in show, Fig. (she line n rrow). Points re in show if the surfe norml mkes n ngle greter thn 9 egrees with the iretion towr the light. The surfe ontour tht ivies points in show from those reeiving light is known s the termintor. Surfe normls long this ontour form 9 egree ngle with the iretion to the light. Similr to st show, there is orresponene etween points in n out of show on either sie of the termintor. This orresponene, however, n only e speifie to within hlf-plne euse the light s elevtion is miguous to within egrees. Atthe shows therefore onstrin the projete lotion of the light soure s to hlf-plne: s p, () where p is point on termintor, n is vetor orthogonl to the termintor in the iretion of the illuminte sie of the tthe show. Notie tht this onstrint hs the sme form s the hlf-plne onstrint provie y shing, Eqution (). As suh, these onstrints n e semlessly omine with the shing onstrints, Setion.. As with the shing onstrints, when the light soure is ehin the mer, its projetion into the imge is inverte y liner perspetive projetion. To onten with this possiility, seon liner progrm is onstrute in whih ll onstrint inequlities re reverse (s with the shing onstrints). The shows re etermine to e onsistent if either liner progrm is stisfile [Kee et l. ].. Unertin shing onstrints To this point we hve ssume n iel system in whih the shing n show onstrints n e preisely n urtely speifie. However, errors will e introue into the estimte light iretion y luminne non-linerities, impreise normls (ue to limite pixel resolution), poor signl-to-noise rtio (ffete y surfe leo n exposure), n evitions from the ssumptions of Lmertin refletne n istnt point-soure. It is therefore importnt to inorporte moel of unertinty into the estimtion of lighting. In forensi setting, full -D moel of n ojet is not typilly ville, n we fous only on the more ommonly ourring sitution in whih the lighting n only e onstrine to line or hlf-plne. For simpliity, n euse it yiels liner onstrints, we ssume tht unertinty in the shing onstrints ffets only the onstrint iretion in Equtions () n () n tht p is not signifintly ffete. This orrespons to the ssumption tht the ominnt soure of unertinty is in the mesure imge rine, rther thn in the speifie surfe normls (this ssumption proves resonle for normls on oluing ontours). Shown in Fig. () n () re illustrtions in whih the estimte lighting onstrints (prmeterize y p n ) o not orretly onstrin the true projete lotion of the light soure, s. By introuing unertinty in the estimte iretion, these onstrints tke on wege-shpe s shown in Fig. () n (). In eh se, the wege-shpe region speifies rnge of plusile projete light positions, n ontens with unertinty in the initil estimte lighting onstrints. Convex weges, Fig. (), n e esrie y the intersetion of two hlf-plnes, eh of whih re liner. Conve weges, Fig. (), however, re not liner, n prelue liner solution to etermine the stisfiility of onstrints, Setion.. This prolem of onve onstrints n e overome y noting tht, lthough the line n hlf-plne onstrints re esrie in terms of egenery in the liner system, oth onstrints n e onstrute for prtiulr liner system. Tht is, inste of onstruting only one onstrint for eh ojet (either line or hlf-plne onstrint), we onstrut oth onstrints regrless of onitioning, n ngulr unertinty s itte y the orresponing singulr vlues of the liner system, Fig. (e). (More etils on this onstrution re given elow.) The onstrints re then intersete to proue omine onstrint, Fig. (f). If this omine onstrint speifies onvex wege, it n e inorporte into the liner progrm, Setion., otherwise, it n e omitte (thus proviing no onstrint). The unertinty of onstrints, hrterize s the with of the resulting wege-shpe region, speify onfiene intervl on the ngle etween the onstrint iretion n the vetor from p to the light soure (up to n inversion of ue to lights tht lie ehin the mer). In prt, this unertinty epens upon mesurle hrteristis of the liner system, whih inlues the singulr vlue ssoite with the onstrint, the rnge of surfe normls tht re ville, n the extent to whih the estimte lighting prmeters, e, n explin the vrition in the imge intensity. We therefore esrie the with of the wege-shpe region for line n hlf-plne onstrints s funtions Φ L ( ) n Φ H ( ) of three prmeters κ, θ, n ρ. The first prmeter, κ, is the onition of the line onstrint, κ L = λ /λ, or hlf-plne onstrint, κ H = λ /λ. The seon prmeter, θ, is the rnge of surfe normls, mesure to e inepenent of imge resolution y onstruting histogrm over the iretions of the normls n omputing the numer of egrees spnne y non-empty ins. Lstly, the extent to whih the lighting moel n explin the oserve imge intensity is mesure s the R-vlue, ρ, of the lest-squres solution, Eqution (). We use lrge-sle simultions to estimte the funtions Φ L ( ) n Φ H ( ), s etile in Appenix B. Rnom -D shpes re imge uner rnom point lights, n y mers with rnom fol lengths n imge enters. Common soures of unertinty re inlue: seonry point light is e to simulte inter-refletions n other effets, n the imges re sujete to noise, rnom gmm orretions, n quntiztion. Appenix C etils how the ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

7 Exposing Photo Mnipultion from Shing n Shows XXX : () () (e) () () (f ) Fig.. Shown in eh pnel is n oluing ontour long the top of n ojet. The projete lotion of the light soure is enote s s n the lest squres estimte of the light soure is enote s s. Shown re the following onstrints, prmeterize y position p n iretion : () line onstrint; () its orresponing onvex wege-shpe onstrint; () hlf-plne onstrint; () its orresponing onve wege-shpe onstrint; (e) the pir of wege-shpe onstrints superimpose; n (f) the intersetion of these two onstrints yieling single onvex onstrint. ojets re she. Rnom portions of the she ojets re selete to ompute the iretions of the line n hlf-plne onstrints, n their ngles φ to the light soure. This simultion is repete for mny suh smples φ, n eh is ssigne to its requisite quntize lotion in the omin of the funtions Φ L ( ) n Φ H ( ). Those funtions funtions speify the ngle elow whih mjority of the ngles φ t eh quntize lotion fll (e.g., 99.9%). To uil onstrint from n oluing ontour, the position, iretion, n with of the line n hlf-plne onstrints re estimte, n their intersetion ompute. This proues omine onstrint with hlf-with, Φ C, iretion, n position p = p. Convex weges (Φ C 9 ) n e esrie y pir of hlfplnes: s n C p C n C n s n C p C n C, () where s is the unknown lotion of the light soure. Vetors n i C re n C = H T R C n n C = HR T C, () where H is 9 rottion mtrix, n mtrix R speifies rottion of Φ C egrees. Lstly, onve weges (Φ C > 9 ) nnot e esrie y liner onstrints, n re omitte for simpliity.. Unertin show onstrints It my e iffiult to urtely speify st show onstrint ue to finite resolution, miguity in mthing st show to its orresponing ojet, or when the show is inistint. Unlike the shing onstrints, moeling unertinty in st show onstrints is more strightforwr. The line onstrint for st shows n e relxe to wegeshpe onstrint y llowing forensi nlyst to selet point on show n rnge of possile orresponing points on the ojet tht sfely enompss the orret ojet-show piring [Kee et l. ]. These relxe onstrints n e esrie y pir of hlfplnes: s n i p i n i n s n i p i n i, (9) where the norml vetors n i re efine suh tht points within the wege re on the positive sie of oth hlf-plne onstrints. These wege onstrints re liner, n the stisfiility of olletion of these, long with other liner shing-se onstrints, n e etermine y liner progrmming, Setion.. Note tht these onstrints re ientil to the wege-shpe shing onstrints, Eqution (). Lstly, n nlyst my sometimes prefer to inste selet point on n ojet n rnge of possile orresponing points in show (not vie vers). In this se the vetors n in Eqution (9) re simply reple y n. As note previously [Kee et l. ], we ssume tht n tthe show onstrint n e relily speifie ue to the reltive ese with whih the termintor on n tthe show n e speifie. This ssumption vois non-liner (onve) onstrints whih woul prelue liner solution, Setion... Forensis A photo my ontin vriety of she ojets long with st n tthe shows. Show onstrints re speifie mnully [Kee et l. ], while shing onstrints re speifie semiutomtilly. In prtie full -D geometri moels of sene ojets re rrely ville, n etermining surfe normls from single imge is generlly iffiult n error prone. We therefore restrit ourselves to using points long oluing ontours, for whih normls n e resonly estimte. The normls long oluing ontours will generlly not spn the full -D spe n therefore proue uner-onstrine systems orresponing to line n hlf-plne shing onstrints. ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

8 XXX : E. Kee, J. O Brien, H. Fri An nlyst speifies shing onstrint y first ientifying segment of n oluing ontour hving onstnt Lmertin refletne. The -D imge normls, n = `n x n y T, re ompute from this oluing ontour ( Bezier or other generl-purpose urve n e fit to this ontour, from whih the -D normls n e nlytilly estimte). Uner moel of liner perspetive projetion, the orresponing -D surfe norml is given y: n x n = n y, () n (x ) f where x is the -D imge lotion of eh orresponing -D norml, f is the mer s fol length, is the imge enter, n it is ssume tht n is susequently sle to unit length (see Appenix D for erivtion of this expression). These normls n the orresponing imge rine r long the oluing ontour re use to estimte the lighting prmeters e, Eqution (). This estimtion is strightforwr if the mer fol length, f, n imge enter,, re known, or n e estimte. As we will esrie elow, however, the mer prmeters nee not e known. The liner system use to estimte the lighting prmeters e, Eqution (), will typilly e egenerte ue to the limite rnge of normls provie y n oluing ontour. This system is therefore solve y omputing the pseuo-inverse, Eqution (). The SVD of the mtrix of surfe normls, N = UΛV T, is then ompute. Using the estimte lighting prmeters, e =(s ) T, n the mtrix V, the position n iretion of the line onstrint, p L n L, re given y Equtions ()-(), n the position n iretion of the hlf-plne onstrint, p H n H, re given y Equtions ()-(). Next, the line n hlf-plne onstrints of n oluing ontour re juste to onten with unertinty in the estimte lighting prmeters, Setion.. The line onstrint tkes on wege shpe with position p L, iretion L, n ngulr hlf-with Φ L (κ L,θ,ρ); the hlf-plne onstrint lso tkes on wege shpe with position p H, iretion H, n ngulr hlf-with Φ H (κ H,θ,ρ). These two onstrints re intersete to proue single wege-shpe region with position p C = p L = p H. Beuse oth onstituent weges my e onve, the iretion C, n ngulr hlf-with, Φ C,of the omine wege is ompute using simple se-se lgorithm. If the resulting wege is onve, n hene non-liner, no onstrint is speifie. Eh onvex onstrint (p,, Φ C ) is e to liner progrm y onstruting pir of hlf-plnes, Equtions ()-(). Similrly, eh show onstrint, speifie y the nlyst s either wege or hlf-plne onstrint is e to the liner progrm. A solution to the liner progrm mens tht the shing n shows in n imge re onsistent, while filure to fin vile solution mens tht one or more of the onstrints re inonsistent with single light soure. To onten with the sitution when the light soure is ehin the mer, eh onstrint is reverse to yiel seon liner progrm. Both of these systems re solve to etermine if vile light lotion exists tht stisfies ll onstrints. Lstly, rell tht the mer fol length f n imge enter must e known to estimte the -D surfe normls, Eqution (). Beuse it n e iffiult, or impossile, to estimte these prmeters from single imge, we forgo their estimtion, n inste serh for mer prmeters tht yiel vile solution to the speifie onstrints. Speifilly, we quntize the rnge of plusile fol lengths n imge enters into smll set of vlues. The funtions Φ L ( ) n Φ H ( ), whih ount for unertinty in the es- Fig.. A syntheti sene renere with the physilly se renerer Mitsu [Jko ]. This si sene ws use to test the reliility n speifiity of the shing onstrints. timte lighting prmeters, re juste to inlue the itionl unertinty tht is introue y this quntiztion. Eh quntize tuple of mer prmeters is use to onstrut the require -D surfe normls, n the stisfiility of the orresponing liner progrms is heke. This pproh is omputtionlly effiient ue to the simpliity of the liner formultion, n euse reltively orse quntiztion of the fol length n imge enter hs proven to e suffiient, Appenix B.. RESULTS We prtition our results into two min setions. In the first setion we report on set of lrge-sle simultions use to vlite the moel of unertinty employe y the shing-se onstrints, Setion.. In prtiulr, we report on the reliility n speifiity of the shing onstrints uner vriety of ssumptions tht re onsistent with, n tht violte, our imging ssumptions. In the seon setion we pply our forensi tehnique to uthenti n visully ompelling forgeries.. Simultions In the iel sitution, the wege-shpe shing onstrints will e just ro enough so s to lwys enompss the estimte light position, ut nrrow enough so s to relily etet ifferenes in lighting. We quntify these riteri in terms of the reliility n speifiity of the wege-shpe onstrints. The reliility is mesure s the proility tht onstrint orretly enompsses the true light soure. The speifiity is mesure s the proility tht rnom light position stisfies one or more onstrints... Reliility. Shown in Fig. is sene renere with the physilly se renerer Mitsu [Jko ]. This rok gren sene ws renere with either point light, n re light sutening, or physilly se ylight moel [Preethm et l. 999]. In eh se, one of these istnt light soures ws ple in one of 9 possile iretions, uniformly smple over hemisphere. The sene ws lso renere with the ojets hving ifferent mteril properties. Eh ojet s mteril ws of onstnt refletne n ws either Lmertin, rough iffuse mteril (moeling surfes suh s onrete n ly [Oren n Nyr 99]), rough plsti mteril (moele y the physilly-se Bek- ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

9 Exposing Photo Mnipultion from Shing n Shows XXX : 9 mnn mrofet istriution [Bekmnn 9]), or woven loth mteril [Irwn ]. The imge renering ws onfigure to yiel ojets with n verge resolution of (efult) or squre pixels, % (efult) or % itive noise, n gmm orretion in the rnge [.,.] (efult) or [.,.]. Shing onstrints were onstrute y iviing the illuminte portion of the oluing ontour on eh rok into segments, n seleting rnom suset of etween n segments. The orresponing -D oluing normls were estimte y seleting fol length n imge enter within the preision ssume y the unertinty funtions, Φ L n Φ H, Appenix B. A totl of, onstrints were speifie for eh of 9 types of senes ( lighting senrios, mterils, n renering onfigurtions). The onstrints uilt roun the line n hlf-plne were esigne to hve n verge reliility of 99.9%, Appenix B. These onstrints re intersete to proue omine onstrint whih therefore hs n verge reliility of.999 =99.%. We therefore expet tht this perentge of onstrints will enompss the projete lotion of the light soure. In the se of istnt point light soure, Lmertin refletne, squre pixel resolution, % itive noise, n gmm in the rnge [.,.], the projete lotion of the light soure ws enompsse y 99.% of the onstrints. An re light yiele the sme reliility, n ylight moel yiele higher reliility of 99.9%. Reliility improves uner the ylight moel euse it ontriutes stronger mient illumintion, whih ereses the signl-to-noise rtio. This erese proues wier onstrints, n improves reliility. In the se of non-lmertin surfes illuminte y istnt point light soure n the efult remining imging prmeters, rough iffuse ojets, rough plsti, n loth mteril yiele onstrints with reliility of 99.%, 9.%, n 99.% respetively. Rough plsti le to reution in reliility y severl perentge points euse the roughene speulr omponent ises the estimte light iretion, ut oes not yiel orresponing inrese in the onstrint with. Returning to istnt point light soure n Lmertin surfe, erese in resolution to squre pixels yiele reliility of 9.%. An inrese in the itive noise to % yiele reliility of 99.%. An inrese in gmm into the rnge [.,.] yiele reliility of 99.%. As expete, the reue resolution impts the reliility. The inrese gmm h little ffet on reliility, n interestingly n inrese level of noise improves reliility. This higher reliility is ue to reue signl-to-noise rtio, whih reues the R-vlue of the lest-squres solution n therefore yiels wier onstrint. To summrize, the overll reliility of the onstrints ws ompute y omining onstrints ross ll sene types (lights, refletne moels, n imge egrtions). This yiels men reliility of 99.%. This reliility oes not epen on onstrint with: the men reliility for onstrints of the sme with ws lso 99.% (with minimum if 9.%, n mximum of 99.%)... Speifiity. The with of eh onstrint etermines its speifiity, whih is efine s the likelihoo tht rnom light iretion is stisfie y one, or more, onstrints. We ompute the speifiity for vrile numer n rnge of oluing ontours from the senes esrie in the preeing setion, Fig.. proility (%) entire ontour / ontour / ontour mixe, rel imges numer of onstrints Fig.. The mein proility tht rnomly selete light iretion will stisfy etween one n eight shing onstrints. Eh urve orrespons to onstrints uilt from the entire illuminte portion of the oluing ontour (ottom), hlf of the ontours (mile), or qurter of the ontours (top). The error rs ientify the th n th quntile. Also shown (she gry) is this proility when using ontours of mixe sptil extent from the uthenti photos in Figs.,,, n (see Setion.). The speifiity of these rel-worl onstrints mthes losely with the renere imges. Rnom -D light iretions were rwn from hemisphere, exluing elevtions elow, n projete into the imge. When the entire illuminte portion of n oluing ontour ws use, the mein onstrint with ws, n ws stisfie y 9% of rnom light iretions. As onstrints were e, up to totl of, this proility erese to %, Fig.. When one hlf of the ontour ws selete, the mein onstrint with ws, n ws stisfie y % of rnom light iretions. This proility erese to % when ll ville onstrints were use, Fig.. When one qurter of the ontour ws selete, the mein onstrint with ws 9, n ws stisfie y % of rnom light iretions. This proility erese to % when ll onstrints were use. Overll, the speifiity is iretly proportionl to the numer n extent of onstrints. We lso mesure the erese in speifiity tht results when the mer intrinsi prmeters re unknown. Unknown mer prmeters le to only moest inrese in the with of the onstrints, n hene the speifiity: the perentge of rnom light iretions tht stisfie onstrints with unknown prmeters inrese y n verge of only.% over those with known prmeters. Lstly, rell tht we exlue onve onstrints s they nnot e inorporte into the liner progrm. In the ove simultions, n verge of only.% of the onstrints were onve... Shing vs. Shows. The ition of shing onstrints llows n nlyst to etermine the onsisteny of regions tht o not ontin shows, n lso to etet mutully inonsistent omintions of shing n shows. A forensi nlysis of n imge is therefore likely to onsist of omintion of shing This mesure of speifiity ws hosen to mth [Kee et l. ], to whih omprison is me in Setion... ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

10 XXX : E. Kee, J. O Brien, H. Fri proility (%) proility (%) () e shing onstrints () speifiity e show onstrints N N shows shows Fig.. The effiy of omining shing n show onstrints is shown in terms of speifiity. Pnel (): speifiity improves s shing onstrints re e to {,,,, } show onstrints. Pnel (): ing or fewer itionl show onstrints inste of shing onstrints yiels similr improvement in speifiity. Aing more thn show onstrints yiels greter speifiity. n show onstrints. To ssess the enefit ffore y this omintion, we next ompre n evlute the usefulness of shing n show onstrints in isoltion n in omintion. Shown in Fig. () is the proility tht rnom light soure will stisfy omintion of shing n show onstrints rwn from the istriution foun in the simulte senes of Fig., n [Kee et l. ]. Eh olore urve orrespons to fixe numer of show onstrints, n the horizontl xis orrespons to the numer of itionl shing onstrints. Shown in Fig. () is similr result in whih show onstrints re e, rther thn shing onstrints (horizontl xis). With smll numer of onstrints, these results show tht the shing onstrints re similr in speifiity to the show onstrints of [Kee et l. ]. For exmple, rnom light soure will stisfy shing onstrints with proility of %, n for show onstrints the proility is similr 9%. These results lso show, however, tht for lrger numer of onstrints (greter thn ), the show onstrints re more speifi thn the shing onstrints. In seon set of simultions, we mesure the proility tht inonsistent lighting will e etete s funtion of the ngulr istne etween two light soures, n s funtion of the numer of shing n show onstrints. Speifilly, we simulte the retion of mny forgeries, eh y rnomly seleting two light soures from mong 9 iretions, uniformly smple on hemisphere. A set of m onstrints ws rete to e onsistent with the first light soure, n seon set of n onstrints ws rete to e onsistent with the seon light soure, oring to the istriutions of onstrint withs foun in the simulte senes. These two sets of onstrints were then omine, n forgery ws etete when the omine onstrints were not stisfile. The proility tht these m + n onstrints were unstisfile ws mesure s funtion of the mein lighting ifferene, ompute s the mein ngle etween the projete iretions of the two light soures t eh onstrint. Shown in Fig. 9()-(f) re the results when using either m = or m =onstrints from the first light soure, n vrying numer of n onstrints from the seon light soure. In pnels ()-(), the m onstrints re from shing, n re omine with n shing onstrints; in pnels ()-(), the m onstrints re from shows, n re omine with n shing onstrints; n in pnels (e)-(f) the m onstrints re from shows, n re omine with n show onstrints. In generl, show onstrints re helpful for smll lighting errors, while the shing onstrints re slightly more effetive overll. For exmple, using m =show onstrints, pnel (), rther thn m =shing onstrints, pnel (), lighting errors elow egrees n e etete with slightly higher proility. In ontrst, using n =shing onstrints, pnels () n (), lighting inonsisteny of egrees n e etete with proility of %; using shows lone, pnel (e), this sme lighting inonsisteny is etetle with proility %. To summrize, the loliztion of light soure n the likelihoo of eteting lighting inonsisteny improves s more onstrints re e. Shing onstrints offer n vntge of equl or etter isrimintion ompre to show onstrints. In prt, this is euse the show onstrints onsist of oth st n tthe shows. While st shows n e highly seletive, tthe shows re less seletive. Nevertheless, it will generlly e vntgeous to s mny onstrints s possile to oth lolize the light soure n inrese the likelihoo of fining inonsistent shing or show. Comining onstrints from two istint types of imge informtion, shing n shows, lso offers the vntge tht forger hoping to voi etetion must now work to mke oth spets of the imge iniviully n mutully onsistent.. Rel worl Shown in the top left pnel of Fig. is n uthenti photo. This sene ws illuminte y single light soure lote ehin the mer n pproximtely m from the sene. Shown in the top right pnel re vriety of shing n show onstrints, n mgnifie views re shown eneth. Shing onstrints (lk ots) re ompute from segments long oluing ontours (outline in yellow). These inlue loth, ermi, n orgni mterils. Only the reverse onstrints re shown, orresponing to the se when the light is ehin the mer. These shing onstrints re physilly onsistent, s inite y their fesile region (outline in lk). Show onstrints re speifie y ientifying orresponenes etween shows n ojets (white squres). Note tht these onstrints my e speifie from show to ojet (lels,, e ), or from ojet to show (lels,, f ). As with the shing These 9 lights mth [Kee et l. ], ensuring tht the nlysis fithfully ompres the etetion proility to our prior show-se work. ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

11 Exposing Photo Mnipultion from Shing n Shows XXX : () shing + shing () show + shing (e) show + show proility (%) proility (%) () shing +N shing () shows +N shing (f) shows +N shows shing +N shing 9 lighting ifferene (egrees) shows +N shing 9 lighting ifferene (egrees) shows +N shows 9 lighting ifferene (egrees) Fig. 9. The effiy of omining shing n show onstrints is shown in terms of eteting forgery. Forgery is etete y omining m = {, } onstrints from first light soure, n n onstrints from seon light soure. Pnels ()-(), or shing onstrints re omine with n shing onstrints. Pnels ()-(), or show onstrints re omine with n shing onstrints. Pnels (e)-(f), or show onstrints re omine with n show onstrints. onstrints, only the reverse onstrints re shown. These show onstrints re physilly onsistent, s inite y their fesile region (outline in white). Beuse the iniviul fesile regions interset, the shing n show onstrints re mutully onsistent. This ommon intersetion of ll shing n show onstrints orretly suggests tht the lighting in this sene is onsistent with single light soure. Shown in the ottom left pnel of Fig. is omposite imge in whih three ojets (the sqush n the green n ornge shirts) hve een reple with versions photogrphe uner ifferent lighting. The sme set of shing n shows onstrints were use s in the originl version of this imge. As shown in the lower right pnel, however, the onstrints re no longer stisfile: the show onstrints lele n f hve ommon intersetion to the right, while the shing onstrints lele, 9, hve ommon intersetion to the left. For the uthenti photo in Fig., vile solution ws foun with fol length f =mm n imge enter =(, ). For the omposite photo, no omintion of fol length n imge enter yiels vile solution. Although the shing inonsistenies on the moifie ojets re iffiult to etet visully, our lgorithm proues rilly ifferent onstrints n esily ientifies the forgery. Shown in the top left pnel of Fig. is n uthenti photo of sene illuminte y the sun. Ojets tht proue istint, pointe, shows were ple in the sene to llow for preise loliztion of the light soure. Shown in the top right pnel re vriety of shing n show onstrints; only the reverse onstrints re shown. The ville shing onstrints re physilly onsistent, s inite y their fesile region outline in lk. The show onstrints re lso physilly onsistent s inite y their fesile region outline in white. Furthermore, the shing n show onstrints re mutully onsistent euse these two fesile regions interset. Shown in the ottom left pnel of Fig. is omposite imge in whih pink ll hs een e. The remining st shows were then moifie so s to e physilly onsistent with the inserte sphere. As seen in the lower right pnel, the shing onstrints re physilly onsistent. This is euse the pink ll ws photogrphe uner lighting tht ws similr to the originl sene. The show onstrints re lso physilly onsistent. Beuse these two fesile regions o not interset, however, the shing n show onstrints together re not mutully physilly onsistent. Shown in top left pnel of Fig. is n uthenti photo tht t first glne my pper unlikely. Segments of pproximtely on- Its uthentiity is orroorte y itionl photos on the soure wesite. ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month.

12 XXX : E. Kee, J. O Brien, H. Fri e f 9 e f 9 f 9 Imges opyright, Kee, OBrien, n Fri 9 Fig.. A omplete shing n show nlysis of n uthenti (top) n omposite (ottom) photo. Top: shing onstrints, enote y lk ots, re selete from vriety of oluing ontours eh outline in yellow. The outline lk region is the ommon intersetion of these onstrints, showing tht they re ll mutully onsistent. Show onstrints re enote y white squres. The outline white region is the ommon intersetion of these onstrints, showing tht the shows re mutully onsistent. Beuse the vile shing n show regions interset, the lighting in this photo is physilly onsistent. Bottom: in this omposite photo three ojets with ifferent lighting were inserte (the sqush n the green n ornge shirts). The five onstrints shown in the lower right re inonsistent, reveling this photo to e omposite. stnt refletne re selete long vriety of oluing ontours, lele. The outline lk region is the ommon intersetion of these onstrints. Show onstrints, lele h hve ommon intersetion outline in white. Beuse the vile ACM Trnstions on Grphis, Vol. VV, No. N, Artile XXX, Pulition te: Month. shing n show regions interset, the lighting in this photo is physilly onsistent. Shown in Fig. is nother photo tht my, t first glne, pper unlikely (there oes in ft exist putting green on the front lwn of