3D Computer Vision Photometric stereo Prof. Didier Stricker Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de 1
Physical parameters of image formation Optical Sensor s lens type focal length, field of view, aperture Geometric Type of projection Camera calibration Two/Multi-views geometry Photometric (radiometry) Type, direction, intensity of light reaching sensor Surfaces reflectance properties Inference from shading 2
Image formation What determines the brightness of an image pixel? Light source properties Sensor characteristics Surface shape and orientation Exposure Optics Surface reflectance properties Slide by L. Fei-Fei
Radiance and irradiance Radiance (L) energy exiting a source or surface Irradiance (E) incoming energy E L E L E L E L Which (E or L) does a camera sensor array directly measure? 4
Radiometry Radiometry is the part of image formation concerned with the relation among the amounts of light energy emitted from light sources, reflected from surfaces, and registered by sensors. Bahadir K. Gunturk 5
Foreshortening A big source, viewed at a glancing angle, must produce the same effect as a small source viewed frontally. This phenomenon is known as foreshortening. Bahadir K. Gunturk 6
Solid Angle Solid angle is defined by the projected area of a surface patch onto a unit sphere of a point. dω = da 0 = da cos θ r 2 * da projected onto surface of the sphere of radius r: da cos theta * Ratio of surface areas of spheres of radii 1 and r: 4 pi / (4 pi r^2) = 1/r^2 (Solid angle is subtended by a point and a surface patch.) Bahadir K. Gunturk 7
Solid Angle Arc length d rd r Bahadir K. Gunturk 8
Solid Angle Solid angle is defined by the projected area of a surface patch onto a unit sphere of a point. 2 da rd r sin d r sin d d da 2 TotalArea r sin d d 4 r 0 0 2 2 dw da sin d d 2 r Bahadir K. Gunturk 9
Solid Angle Similarly, solid angle due to a line segment is dl d r dφ = dl cos θ r Bahadir K. Gunturk 10
Radiance The distribution of light in space is a function of position and direction. The appropriate unit for measuring the distribution of light in space is radiance, which is defined as the power (the amount of energy per unit time) traveling at some point in a specified direction, per unit area perpendicular to the direction of travel, per unit solid angle. In short, radiance is the amount of light radiated from a point (into a unit solid angle, from a unit area). Radiance = Power / (solid angle x foreshortened area) W/sr/m2 W is Watt, sr is steradian, m2 is meter-squared Bahadir K. Gunturk 11
Radiance Radiance from ds to dr Radiance = Power / (solid angle x foreshortened area) Bahadir K. Gunturk 12
Radiance Example: Infinitesimal source and surface patches Radiance = Power / (solid angle x foreshortened area) Power or radiant flux emitted of the source Illuminated surface L x 1, x 1 x 2 = Radiance leaving x1 in the direction of x2 dψ dw 2 cos θ 1 da 1 = r 2 dψ da 2 cos θ 2 cos θ 1 da 1 Source dw 2 = da 2 cos θ 2 r 2 Bahadir K. Gunturk 13
Radiance Radiance = Power / (solid angle x foreshortened area) Power at x1 leaving to x2 Illuminated surface dψ = L x 1, x 1 x 2 dw 2 cos θ 1 da 1 = L x 1, x 1 x 2 da 2 cos θ 2 cos θ 1 da 1 r 2 Source dw 2 = da 2 cos θ 2 r 2 Bahadir K. Gunturk 14
Radiance Illuminated surface Let the radiance arriving at x2 from the direction of x1 be dψ L x 2, x 1 x 2 = dw 1 cos θ 2 da 2 r 2 dψ = da 1 cos θ 1 cos θ 2 da 2 Source dw 1 = da 1 cos θ 1 r 2 Bahadir K. Gunturk 15
Radiance The medium is vacuum, that is, it does not absorb energy. Therefore, the power reaching point x2 is equal to the power leaving for x2 from x1. Radiance is constant along a straight line. Illuminated surface L( x, x x ) L( x, x x ) 1 1 2 2 1 2 Source Bahadir K. Gunturk 16
Point Source Many light sources are physically small compared with the environment in which they stand. Such a light source is approximated as an extremely small sphere, in fact, a point. Such a light source is known as a point source. Bahadir K. Gunturk 18
Radiance Intensity If the source is a point source, we use radiance intensity. Radiance intensity = Power / (solid angle) Illuminated surface I d dw da 2 r d cos 2 2 dw da cos r 2 2 2 Source Bahadir K. Gunturk 19
Light at Surfaces When light strikes a surface, it may be absorbed, transmitted, or scattered; usually, combination of these effects occur. It is common to assume that all effects are local and can be explained with a local interaction model. In this model: The radiance leaving a point on a surface is due only to radiance arriving at this point. Surfaces do not generate light internally and treat sources separately. Light leaving a surface at a given wavelength is due to light arriving at that wavelength. Bahadir K. Gunturk 20
Light at Surfaces In the local interaction model, fluorescence, [absorb light at one wavelength and then radiate light at a different wavelength], and emission [e.g., warm surfaces emits light in the visible range] are neglected. Bahadir K. Gunturk 21
Irradiance (E) Irradiance is the total incident power per unit area. Irradiance = Power / Area Bahadir K. Gunturk 22
Irradiance What is the irradiance due to source from angle? L i (x, θ i, φ i ) Bahadir K. Gunturk 23
Irradiance What is the irradiance due to source from angle θ i, φ i? L i (x, θ i, φ i ) da d Li ( x, i, i ) dwcos ida Irradiance Li ( x, i, i ) dwcos i da da radiance foreshortening factor Solid angle Bahadir K. Gunturk 24
Irradiance What is the total irradiance? Integrate over the whole hemisphere. solid angle Since dω = sin θ dθdφ Bahadir K. Gunturk 25
Irradiance due to a Point Source For a point source, Radiance intensity = Power / (solid angle) I d dw 2 r d da cos i i dw da i cos i 2 r da i cos d I i d cos Irradiance I i 2 2 r da r i Bahadir K. Gunturk 27
The Relationship Between Image Intensity and Object Radiance Diameter of lens We assume that there is no power loss in the lens. The power emitted to the lens is d L da cos dw object 0 0 Radiance of object Bahadir K. Gunturk 28
The Relationship Between Image Intensity and Object Radiance Diameter of The solid angle for the entire lens is lens 2 d dw 0 2 The power emitted to the lens is d L da cos dw L object object da / 4 cos r 0 0 2 Area of the lens with diameter d d cos cos 4r 0 2 Bahadir K. Gunturk 29
The Relationship Between Image Intensity and Object Diameter Radiance of lens The solid angle at O can be written in two ways. da0 cos 2 r Note that da p cos OA' 2 OA' f / cos Therefore da0 cos da r cos p 2 2 f 3 Bahadir K. Gunturk 30
The Relationship Between Image Intensity and Object Diameter Radiance of Combine lens 3 da0 cos dap cos 2 2 r f 2 d cos d Lobject da0 cos 2 4r to get 2 d 4 d Lobject cos dap 4 f Bahadir K. Gunturk 31
The Relationship Between Image Intensity and Object Diameter Radiance of lens Therefore the irradiance on the image plane is Irradiance = dψ da p = π 4 L object d f 2 cos 4 θ The irradiance is converted to pixel intensities, which is directly proportional to the radiance of the object. Bahadir K. Gunturk 32
Image irradiance on the image plane The image irradiance (E) is proportional to the object radiance (L) Lens diameter Angle off optical axis E = π 4 d f 2 cos 4 θ L Focus distance What the image reports to us via pixel values What we really want to know 33
Fundamental radiometric relation E 2 d cos 4 4 f L S. B. Kang and R. Weiss, Can we calibrate a camera using an image of a flat, textureless Lambertian surface? ECCV 2000. 34
Surface Characteristics We want to describe the relationship between incoming light and reflected light. This is a function of both the direction in which light arrives at a surface and the direction in which it leaves. θ i, φ i n θ o, φ o Bahadir K. Gunturk 35
Bidirectional Reflectance Distribution Function (BRDF) BRDF is defined as the ratio of the radiance in the outgoing direction to the incident irradiance. Bahadir K. Gunturk 36
Bidirectional Reflectance Distribution Function (BRDF) The radiance leaving a surface due to irradiance in a particular direction is easily obtained from the definition of BRDF: Bahadir K. Gunturk 37
Bidirectional Reflectance Distribution Function (BRDF) The radiance leaving a surface due to irradiance in all incoming directions is where Omega is the incoming hemisphere. Bahadir K. Gunturk 38
BRDFs can be incredibly complicated 39
Lambertian Surface A Lambertian surface has constant BRDF. A Lambertian surface looks equally bright from any view direction. The image intensities of the surface only changes with the illumination directions. constant Bahadir K. Gunturk 40
Lambertian Surface For a Lambertian surface, the outgoing radiance is proportional to the incident radiance. constant If the light source is a point source, a pixel intensity will only be a function of Remember, for a point source Irradiance d da i I cos i 2 r BRDF is constant, we speak about Albedo Albedo: fraction of incident irradiance reflected by the surface Bahadir K. Gunturk 41
Specular Surface The glossy or mirror like surfaces are called specular surfaces. Radiation arriving along a particular direction can only leave along the specular direction, obtained from the surface normal. *The term Specular comes from the Latin word speculum, meaning mirror. Bahadir K. Gunturk 42
Specular Surface Few surfaces are ideally specular. Specular surfaces commonly reflect light into a lobe of directions around the specular direction. Bahadir K. Gunturk 43
Lambertian + Specular Model Relatively few surfaces are either ideal diffuse or perfectly specular. The BRDF of many surfaces can be approximated as a combination of a Lambertian component and a specular component. Bahadir K. Gunturk 44
Lambertian + Specular Model Lambertian Lambertian + Specular Bahadir K. Gunturk 45
Radiosity Radiosity, defined as the total power leaving a point. To obtain the radiosity of a surface at a point, we can sum the radiance leaving the surface at that point over the whole hemisphere. Bahadir K. Gunturk 46
Part II Shading
Point Source For a point source, Radiance intensity = Power / (solid angle) I d dw 2 r d da cos i i dw da i cos i 2 r da i cos d I i d cos Irradiance I i 2 2 r da r i Bahadir K. Gunturk 48
A Point Source at Infinity The radiosity due to a point source at infinity is S( x) N( x) x B: radiosity (total power leaving the surface per unit area) ρ: albedo (fraction of incident irradiance reflected by the surface) N: unit normal S: source vector (magnitude proportional to intensity of the source) Bahadir K. Gunturk 49
Local Shading Models for Point Sources The radiosity due to light generated by a set of point sources is Radiosity due to source s Bahadir K. Gunturk 50
Local Shading Models for Point Sources If all the sources are point sources at infinity, then Bahadir K. Gunturk 51
Ambient Illumination For some environments, the total irradiance a patch obtains from other patches is roughly constant and roughly uniformly distributed across the input hemisphere. In such an environment, it is possible to model the effect of other patches by adding an ambient illumination term to each patch s radiosity. + B0 Bahadir K. Gunturk 52
Photometric Stereo If we are given a set of images of the same scene taken under different given lighting sources, can we recover the 3D shape of the scene? Bahadir K. Gunturk 53
Photometric Stereo For a point source and a Lambertian surface, we can write the image intensity as Suppose we are given the intensities under three lighting conditions: Camera and object are fixed, so a particular pixel intensity is only a function of lighting direction si. Bahadir K. Gunturk 54
Photometric Stereo Stack the pixel intensities to get a vector The surface normal can be found as Since n is a unit vector As a result, we can find the surface normal of each point, hence the 3D shape 55
More than Three Light Sources Get better results by using more lights Least squares solution: I = Sn S T I = S T Sn n = S T S 1 S T I N 1 = N 3 3 1 Solve for ρ, n as before pseudo inverse 56
Photometric Stereo When the source directions are not given, they can be estimated from three known surface normals. Bahadir K. Gunturk 57
Photometric Stereo Bahadir K. Gunturk 58
Photometric Stereo Surface normals 3D shape Bahadir K. Gunturk 59
Photometric Stereo (by Xiaochun Cao) Bahadir K. Gunturk 60
Results 1. Estimate light source directions 2. Compute surface normals 3. Compute albedo values 4. Estimate depth from surface normals 5. Relight the object (with original texture and uniform albedo) 61
Computer vision application Finding the direction of light source P. Nillius and J.-O. Eklundh, Automatic estimation of the projected light source direction, CVPR 2001 62
Computer vision application Detecting composite photos: Fake photo Real photo M. K. Johnson and H. Farid, Exposing Digital Forgeries by Detecting Inconsistencies in Lighting, ACM Multimedia and Security Workshop, 2005. 63
Application: Detecting composite photos Fake photo Real photo 64
Thank you! 28.01.2015 65