Interpolation of RGB components in Bayer CFA images Demosaicing of Bayer-sampled color images Problem: Most digital color cameras, capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Objective: Develop algorithms to interpolate each color plane (called demosaicing) with best quality reconstruction, and with minimal computational complexity. 1
Construction of color image from color planes + original 2
red original green original 3
blue original Formation of Color planes 4
red subsampled green subsampled 5
blue subsampled Bayer CFA image 6
Color plane interpolation Green channel: bilinear interpolation G A G L G R G I G B 1 G I = ( GL + GR + GB + G 4 A ) Color plane interpolation Red channel: bilinear interpolation R NW R NE R C R SW R SE R S 1 R C = ( RNW + RNE + RSW + R 4 SE ) S ( R R ) 1 R = + 2 SW SE 7
red interpolated green interpolated 8
blue interpolated Interpolated color image 9
original Can we do better? Color planes have severe aliasing. Better interpolation of the individual planes has little effect. 10
red interpolated with bilinear interpolator red interpolated with bicubic interpolator 11
Can we do better? Color planes have severe aliasing. Better interpolation of the individual planes has little effect. We could optically prefilter the image (blur it) so that aliasing is less severe. red interpolated with bilinear interpolator 12
prefiltered red interpolated with bilinear interpolator Interpolated color image 13
Prefiltered Interpolated color image original 14
Demosaicing Approaches Non-Adaptive Single-Channel Interpolation: Interpolate each color channel separately using a standard technique, such as nearest-neighbor interpolation, bilinear interpolation, etc. Edge-Directed Interpolation: Estimate potential edges, avoid interpolating across the edges. Edge-directed interpolation 3 1 x 2 4 1.Calculate horizontal gradient ΔH = G1 G2 2.Calculate vertical gradient ΔV = G3 G4 3.If ΔH > ΔV, Gx = (G3 + G4)/2 Else if ΔH < ΔV, Gx = (G1 + G2)/2 Else Gx = (G1 + G2 + G3 + G4)/4 Demosaicing Approaches Edge-Directed Interpolation: Based on the assumption that color channels have similar texture, various edge detectors can be used. 3 1 2 4 5 6 7 8 9 Edge-directed interpolation 1. Calculate horizontal gradient ΔH = (R3 + R7)/2 R5 2. Calculate vertical gradient ΔV = (R1 + R9)/2 R5 3. If ΔH > ΔV, G5 = (G2 + G8)/2 Else if ΔH < ΔV, G5 = (G4 + G6)/2 Else G5 = (G2 + G8 + G4 + G6)/4 15
Demosaicing Approaches Constant-Hue-Based Interpolation: Hue does not change abruptly within a small neighborhood. Interpolate green channel first. Interpolate hue (defined as either color differences or color ratios). Estimate the missing (red/blue) from the interpolated hue. Red Interpolate Interpolate d Red Green Interpolate Demosaicing Approaches Edge-Directed Interpolation of Hue: It is a combination of edge-directed interpolation and constant-hue-based interpolation. Hue is interpolated as in constant-hue-based interpolation approach, but this time, hue is interpolated based on the edge directions (as in the edge-directed interpolation algorithm). 16
Demosaicing Approaches Using Laplacian For Enhancement: Use the second-order gradients of red/blue channels to enhance green channel. 3 1 2 4 5 6 7 8 9 1. Calculate horizontal gradient ΔH = G4 G6 + R5 R3 + R5 R7 2. Calculate vertical gradient ΔV = G2 G8 + R5 R1 + R5 R9 3. If ΔH > ΔV, G5 = (G2 + G8)/2 + (R5 R1 + R5 R9)/4 Else if ΔH < ΔV, G5 = (G4 + G6)/2 + (R5 R3 + R5 R7)/4 Else G5 = (G2 + G8 + G4 + G6)/4 + (R5 R1 + R5 R9 + R5 R3 + R5 R7)/8 Demosaicing CFA s bilinear interpolation Interpolation of green pixels G8 = (G3+G7+G9+G13) / 4 Interpolation of red/blue pixels : * Interpolation of a red/blue pixel at a green position: B7 = (B6+B8) / 2 R7 = (R2+R12) / 2 * Interpolation of a red/blue pixel at a blue/red position: R8 = (R2+R4+R12+R14) / 4 B12 = (B6+B8+B16+B18) / 4 17
Demosaicing CFA s Constant hue-based interpolation (Cok) Hue: Interpolate G first Demosaicing CFA s Median-based interpolation (Freeman) 1. Linear interpolation 2. Median filter on color differences 18
Demosaicing CFA s Gradient-based interpolation (LaRoche-Prescott) 1. Interpolation on G Demosaicing CFA s Gradient-based interpolation (LaRoche-Prescott) 2. Interpolation of color differences 19
Demosaicing CFA s bilinear Cok Freeman LaRoche Demosaicking CFA s Generally, Freeman s is the best, especially for natural images. 20
A Valuable Resource: A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras http://scien.stanford.edu/class/psych22 1/projects/99/tingchen/ URL on Resources page This week in Lab Implement Bilinear Interpolation Implement one other Non-Adaptive Algorithm Implement one Adaptive Algorithm 21