3D Computer Vision. Photometric stereo. Prof. Didier Stricker

Transcription

1 3D Computer Vision Photometric stereo Prof. Didier Stricker Kaiserlautern University DFKI Deutsches Forschungszentrum für Künstliche Intelligenz 1

2 Physical parameters of image formation Optical Sensor s lens type focal length, field of view, aperture Geometric Type of projection Camera calibration Two/Multi-views geometry Photometric (radiometry) Type, direction, intensity of light reaching sensor Surfaces reflectance properties Inference from shading 2

3 Image formation What determines the brightness of an image pixel? Light source properties Sensor characteristics Surface shape and orientation Exposure Optics Surface reflectance properties Slide by L. Fei-Fei

4 Radiance and irradiance Radiance (L) energy exiting a source or surface Irradiance (E) incoming energy E L E L E L E L Which (E or L) does a camera sensor array directly measure? 4

5 Radiometry Radiometry is the part of image formation concerned with the relation among the amounts of light energy emitted from light sources, reflected from surfaces, and registered by sensors. Bahadir K. Gunturk 5

6 Foreshortening A big source, viewed at a glancing angle, must produce the same effect as a small source viewed frontally. This phenomenon is known as foreshortening. Bahadir K. Gunturk 6

7 Solid Angle Solid angle is defined by the projected area of a surface patch onto a unit sphere of a point. dω = da 0 = da cos θ r 2 * da projected onto surface of the sphere of radius r: da cos theta * Ratio of surface areas of spheres of radii 1 and r: 4 pi / (4 pi r^2) = 1/r^2 (Solid angle is subtended by a point and a surface patch.) Bahadir K. Gunturk 7

8 Solid Angle Arc length d rd r Bahadir K. Gunturk 8

9 Solid Angle Solid angle is defined by the projected area of a surface patch onto a unit sphere of a point. 2 da rd r sin d r sin d d da 2 TotalArea r sin d d 4 r dw da sin d d 2 r Bahadir K. Gunturk 9

10 Solid Angle Similarly, solid angle due to a line segment is dl d r dφ = dl cos θ r Bahadir K. Gunturk 10

11 Radiance The distribution of light in space is a function of position and direction. The appropriate unit for measuring the distribution of light in space is radiance, which is defined as the power (the amount of energy per unit time) traveling at some point in a specified direction, per unit area perpendicular to the direction of travel, per unit solid angle. In short, radiance is the amount of light radiated from a point (into a unit solid angle, from a unit area). Radiance = Power / (solid angle x foreshortened area) W/sr/m2 W is Watt, sr is steradian, m2 is meter-squared Bahadir K. Gunturk 11

12 Radiance Radiance from ds to dr Radiance = Power / (solid angle x foreshortened area) Bahadir K. Gunturk 12

13 Radiance Example: Infinitesimal source and surface patches Radiance = Power / (solid angle x foreshortened area) Power or radiant flux emitted of the source Illuminated surface L x 1, x 1 x 2 = Radiance leaving x1 in the direction of x2 dψ dw 2 cos θ 1 da 1 = r 2 dψ da 2 cos θ 2 cos θ 1 da 1 Source dw 2 = da 2 cos θ 2 r 2 Bahadir K. Gunturk 13

14 Radiance Radiance = Power / (solid angle x foreshortened area) Power at x1 leaving to x2 Illuminated surface dψ = L x 1, x 1 x 2 dw 2 cos θ 1 da 1 = L x 1, x 1 x 2 da 2 cos θ 2 cos θ 1 da 1 r 2 Source dw 2 = da 2 cos θ 2 r 2 Bahadir K. Gunturk 14

15 Radiance Illuminated surface Let the radiance arriving at x2 from the direction of x1 be dψ L x 2, x 1 x 2 = dw 1 cos θ 2 da 2 r 2 dψ = da 1 cos θ 1 cos θ 2 da 2 Source dw 1 = da 1 cos θ 1 r 2 Bahadir K. Gunturk 15

16 Radiance The medium is vacuum, that is, it does not absorb energy. Therefore, the power reaching point x2 is equal to the power leaving for x2 from x1. Radiance is constant along a straight line. Illuminated surface L( x, x x ) L( x, x x ) Source Bahadir K. Gunturk 16

17 Point Source Many light sources are physically small compared with the environment in which they stand. Such a light source is approximated as an extremely small sphere, in fact, a point. Such a light source is known as a point source. Bahadir K. Gunturk 18

18 Radiance Intensity If the source is a point source, we use radiance intensity. Radiance intensity = Power / (solid angle) Illuminated surface I d dw da 2 r d cos 2 2 dw da cos r Source Bahadir K. Gunturk 19

19 Light at Surfaces When light strikes a surface, it may be absorbed, transmitted, or scattered; usually, combination of these effects occur. It is common to assume that all effects are local and can be explained with a local interaction model. In this model: The radiance leaving a point on a surface is due only to radiance arriving at this point. Surfaces do not generate light internally and treat sources separately. Light leaving a surface at a given wavelength is due to light arriving at that wavelength. Bahadir K. Gunturk 20

20 Light at Surfaces In the local interaction model, fluorescence, [absorb light at one wavelength and then radiate light at a different wavelength], and emission [e.g., warm surfaces emits light in the visible range] are neglected. Bahadir K. Gunturk 21

21 Irradiance (E) Irradiance is the total incident power per unit area. Irradiance = Power / Area Bahadir K. Gunturk 22

22 Irradiance What is the irradiance due to source from angle? L i (x, θ i, φ i ) Bahadir K. Gunturk 23

23 Irradiance What is the irradiance due to source from angle θ i, φ i? L i (x, θ i, φ i ) da d Li ( x, i, i ) dwcos ida Irradiance Li ( x, i, i ) dwcos i da da radiance foreshortening factor Solid angle Bahadir K. Gunturk 24

24 Irradiance What is the total irradiance? Integrate over the whole hemisphere. solid angle Since dω = sin θ dθdφ Bahadir K. Gunturk 25

25 Irradiance due to a Point Source For a point source, Radiance intensity = Power / (solid angle) I d dw 2 r d da cos i i dw da i cos i 2 r da i cos d I i d cos Irradiance I i 2 2 r da r i Bahadir K. Gunturk 27

26 The Relationship Between Image Intensity and Object Radiance Diameter of lens We assume that there is no power loss in the lens. The power emitted to the lens is d L da cos dw object 0 0 Radiance of object Bahadir K. Gunturk 28

27 The Relationship Between Image Intensity and Object Radiance Diameter of The solid angle for the entire lens is lens 2 d dw 0 2 The power emitted to the lens is d L da cos dw L object object da / 4 cos r Area of the lens with diameter d d cos cos 4r 0 2 Bahadir K. Gunturk 29

28 The Relationship Between Image Intensity and Object Diameter Radiance of lens The solid angle at O can be written in two ways. da0 cos 2 r Note that da p cos OA' 2 OA' f / cos Therefore da0 cos da r cos p 2 2 f 3 Bahadir K. Gunturk 30

29 The Relationship Between Image Intensity and Object Diameter Radiance of Combine lens 3 da0 cos dap cos 2 2 r f 2 d cos d Lobject da0 cos 2 4r to get 2 d 4 d Lobject cos dap 4 f Bahadir K. Gunturk 31

30 The Relationship Between Image Intensity and Object Diameter Radiance of lens Therefore the irradiance on the image plane is Irradiance = dψ da p = π 4 L object d f 2 cos 4 θ The irradiance is converted to pixel intensities, which is directly proportional to the radiance of the object. Bahadir K. Gunturk 32

31 Image irradiance on the image plane The image irradiance (E) is proportional to the object radiance (L) Lens diameter Angle off optical axis E = π 4 d f 2 cos 4 θ L Focus distance What the image reports to us via pixel values What we really want to know 33

32 Fundamental radiometric relation E 2 d cos 4 4 f L S. B. Kang and R. Weiss, Can we calibrate a camera using an image of a flat, textureless Lambertian surface? ECCV

33 Surface Characteristics We want to describe the relationship between incoming light and reflected light. This is a function of both the direction in which light arrives at a surface and the direction in which it leaves. θ i, φ i n θ o, φ o Bahadir K. Gunturk 35

34 Bidirectional Reflectance Distribution Function (BRDF) BRDF is defined as the ratio of the radiance in the outgoing direction to the incident irradiance. Bahadir K. Gunturk 36

35 Bidirectional Reflectance Distribution Function (BRDF) The radiance leaving a surface due to irradiance in a particular direction is easily obtained from the definition of BRDF: Bahadir K. Gunturk 37

36 Bidirectional Reflectance Distribution Function (BRDF) The radiance leaving a surface due to irradiance in all incoming directions is where Omega is the incoming hemisphere. Bahadir K. Gunturk 38

37 BRDFs can be incredibly complicated 39

38 Lambertian Surface A Lambertian surface has constant BRDF. A Lambertian surface looks equally bright from any view direction. The image intensities of the surface only changes with the illumination directions. constant Bahadir K. Gunturk 40

39 Lambertian Surface For a Lambertian surface, the outgoing radiance is proportional to the incident radiance. constant If the light source is a point source, a pixel intensity will only be a function of Remember, for a point source Irradiance d da i I cos i 2 r BRDF is constant, we speak about Albedo Albedo: fraction of incident irradiance reflected by the surface Bahadir K. Gunturk 41

40 Specular Surface The glossy or mirror like surfaces are called specular surfaces. Radiation arriving along a particular direction can only leave along the specular direction, obtained from the surface normal. *The term Specular comes from the Latin word speculum, meaning mirror. Bahadir K. Gunturk 42

41 Specular Surface Few surfaces are ideally specular. Specular surfaces commonly reflect light into a lobe of directions around the specular direction. Bahadir K. Gunturk 43

42 Lambertian + Specular Model Relatively few surfaces are either ideal diffuse or perfectly specular. The BRDF of many surfaces can be approximated as a combination of a Lambertian component and a specular component. Bahadir K. Gunturk 44

43 Lambertian + Specular Model Lambertian Lambertian + Specular Bahadir K. Gunturk 45

44 Radiosity Radiosity, defined as the total power leaving a point. To obtain the radiosity of a surface at a point, we can sum the radiance leaving the surface at that point over the whole hemisphere. Bahadir K. Gunturk 46

45 Part II Shading

46 Point Source For a point source, Radiance intensity = Power / (solid angle) I d dw 2 r d da cos i i dw da i cos i 2 r da i cos d I i d cos Irradiance I i 2 2 r da r i Bahadir K. Gunturk 48

47 A Point Source at Infinity The radiosity due to a point source at infinity is S( x) N( x) x B: radiosity (total power leaving the surface per unit area) ρ: albedo (fraction of incident irradiance reflected by the surface) N: unit normal S: source vector (magnitude proportional to intensity of the source) Bahadir K. Gunturk 49

48 Local Shading Models for Point Sources The radiosity due to light generated by a set of point sources is Radiosity due to source s Bahadir K. Gunturk 50

49 Local Shading Models for Point Sources If all the sources are point sources at infinity, then Bahadir K. Gunturk 51

50 Ambient Illumination For some environments, the total irradiance a patch obtains from other patches is roughly constant and roughly uniformly distributed across the input hemisphere. In such an environment, it is possible to model the effect of other patches by adding an ambient illumination term to each patch s radiosity. + B0 Bahadir K. Gunturk 52

51 Photometric Stereo If we are given a set of images of the same scene taken under different given lighting sources, can we recover the 3D shape of the scene? Bahadir K. Gunturk 53

52 Photometric Stereo For a point source and a Lambertian surface, we can write the image intensity as Suppose we are given the intensities under three lighting conditions: Camera and object are fixed, so a particular pixel intensity is only a function of lighting direction si. Bahadir K. Gunturk 54

53 Photometric Stereo Stack the pixel intensities to get a vector The surface normal can be found as Since n is a unit vector As a result, we can find the surface normal of each point, hence the 3D shape 55

54 More than Three Light Sources Get better results by using more lights Least squares solution: I = Sn S T I = S T Sn n = S T S 1 S T I N 1 = N Solve for ρ, n as before pseudo inverse 56

55 Photometric Stereo When the source directions are not given, they can be estimated from three known surface normals. Bahadir K. Gunturk 57

56 Photometric Stereo Bahadir K. Gunturk 58

57 Photometric Stereo Surface normals 3D shape Bahadir K. Gunturk 59

58 Photometric Stereo (by Xiaochun Cao) Bahadir K. Gunturk 60

59 Results 1. Estimate light source directions 2. Compute surface normals 3. Compute albedo values 4. Estimate depth from surface normals 5. Relight the object (with original texture and uniform albedo) 61

60 Computer vision application Finding the direction of light source P. Nillius and J.-O. Eklundh, Automatic estimation of the projected light source direction, CVPR

61 Computer vision application Detecting composite photos: Fake photo Real photo M. K. Johnson and H. Farid, Exposing Digital Forgeries by Detecting Inconsistencies in Lighting, ACM Multimedia and Security Workshop,

62 Application: Detecting composite photos Fake photo Real photo 64

63 Thank you!