Optimal design and critical analysis of a high-resolution video plenoptic demonstrator

Transcription

1 Optimal design and critical analysis of a high-resolution video plenoptic demonstrator Valter Drazic Jean-Jacques Sacré Arno Schubert Jérôme Bertrand Etienne Blondé

2 Journal of Electronic Imaging 2(), 007 (Jan Mar 202) Optimal design and critical analysis of a high-resolution video plenoptic demonstrator Valter Drazic Jean-Jacques Sacré Arno Schubert Jérôme Bertrand Etienne Blondé Technicolor Research & Innovation Video Processing and Perception Lab., av. de Belle-Fontaine, CS Cesson-Sévigné Cedex, France valter.drazic@technicolor.com Abstract. A plenoptic camera is a natural multiview acquisition device also capable of measuring distances by correlating a set of images acquired under different parallaxes. Its single lens and single sensor architecture have two downsides: limited resolution and limited depth sensitivity. As a first step and in order to circumvent those shortcomings, we investigated how the basic design parameters of a plenoptic camera optimize both the resolution of each view and its depth-measuring capability. In a second step, we built a prototype based on a very high resolution Red One movie camera with an external plenoptic adapter and a relay lens. The prototype delivered five video views of The main limitation in our prototype is view crosstalk due to optical aberrations that reduce the depth accuracy performance. We simulated some limiting optical aberrations and predicted their impact on the performance of the camera. In addition, we developed adjustment protocols based on a simple pattern and analysis of programs that investigated the view mapping and amount of parallax crosstalk on the sensor on a pixel basis. The results of these developments enabled us to adjust the lenslet array with a submicrometer precision and to mark the pixels of the sensor where the views do not register properly. 202 SPIE and IS&T. [DOI:.7/.JEI ] Introduction With the advent of 3D, there is an increasing need for creating natural content in the additional dimension, as well as for storing, transmitting, decoding, and displaying it. These objectives are achieved often by using two recorded images instead of a single one and at other times by the use of one recorded and one calculated image encoding the depth of each pixel of the first image. The most popular method for passive depth measurement, which is also the most popular means for creating 3D content, is binocular stereo, where two separate cameras are used. Common convergence and focus, calibration, and bulkiness make the system very demanding. Moreover, creating multiview content for autostereoscopic displays necessitates either view interpolation between the captured left and right images or the use of Paper 79SSP received Apr. 22, 20; revised manuscript received Aug. 3, 20; accepted for publication Aug. 26, 20; published online Feb. 24, /202/$ SPIE and IS&T even more cameras. There are single sensor and single lens solutions 2 7 that avoid complex convergence, focus, and calibrations. Among single lens solutions, the plenoptic camera 4 6 can natively capture multiview content and show it on autostereoscopic displays without any need for complex correspondence calculations and view interpolations. Figure is a schematic of the plenoptic camera principle with five views. The object space is shown on the left side with sampled point objects x i. There is a main lens whose main aperture is subdivided into five virtual apertures. Each of the virtual apertures tunnels the light rays emitted by objects x i onto a corresponding lens of a lenslet array. A sharp image of the objects x i forms on the lenslets at xi 0. Each of the lenslets separates rays of light based on their direction, creating a focused image of the aperture of the main lens on the array of pixels below the lens. The five views are then rendered by collecting the images xi 0 for the different perspectives θ i. As the distance of the objects x i to the main lens is much bigger than the total span, x n, x þn, the object space is seen from the main lens point of view under different nonoverlapping parallaxes θ i. Finally, the field lens ensures telecentricity, so that every lens of the array makes an image of the entrance pupil of the main lens on exactly the five pixels behind the lens. This is also achieved if the lenses of the array are aperture matched to the main lens. 6 Two drawbacks of all single lens solutions for recording 3D by either providing multiple views or one view and a depth map are the loss of resolution and the loss of sensitiveness of their depth-sectioning skills. The loss of resolution in the plenoptic case is obvious from Fig. : Each rendered view has a spatial resolution that equals the sensor s, divided by the total number of captured views. Although this is a drawback, the plenoptic camera remains a very interesting instrument for capturing content for an autostereoscopic display, since its resolution is also scaled down by the number of displayable views. To favor the best sensor s fill factor and be sure that none of its pixels are unused, it is supposed here that the system has perfect aperture matching between the lenses of the array and the main lens. This assumption means that the lenses of the array have a rectangular Journal of Electronic Imaging 007- Jan Mar 202/Vol. 2()

3 From the lens maker Eqs. () and (2) and similar triangles in Eq. (3), d 0 þ d ¼ f : () d 0 þ z þ d x ¼ f : (2) Δu d d x ¼ A d x ; (3) Fig. Five views of the plenoptic camera principle. shape and that the entrance pupil of the main lens matches it and is also rectangular. It is the main purpose of this paper to report on the experience of using a high-resolution video camera combined with very few pixels per lenslet to obtain relatively high resolution multiview streams. The second drawback of a supposedly bad depth resolution has been recently addressed. 8 Previously, there had been opinions that the quality of the depth maps calculated from the multi-images from a plenoptic camera would have a lack of precision due to the small baseline, which equals the size of the entrance pupil of the main lens for the extreme views, but we were not able to find quantifying figures. We will in a first step recall the main results of Drazic, 8 which will be used to design a high-resolution video camera with the best possible compromise between a good spatial and a good depth resolution. 2 Optimization of the Camera for Best Depth Discrimination In this section, we will calculate the plenoptic system s depth discrimination, which is the smallest distance between two on-axis points that would give rise to a disparity of one pixel between the two views recorded through the upper and lower virtual apertures of the main lens. For simplicity, let us assume at this stage that the baseline for the most angularly separated views equals the size A of the entrance pupil of the main lens. Let us further assume that we have a point object at the distance d 0 from the entrance pupil of the main lens. The lenslet array is at the distance d from the exit pupil of the main lens. The lens has a focal length f. Figure 2(a), the top drawing, depicts this. For this situation, if we render the views, we will get only the central pixel ON in each of them. Let us consider Fig. 2(b), the bottom drawing: In this one, the object has been moved by a distance z off its initial position, and now there is a fan of rays spanning over the lenslet array by Δu. Fig. 2 Ray binning at the lenslets as a function of the defocus. the unknown distance d 0 þ z is linked to the span Δu by d 0 þ z ¼ Δu d 0 A f : (4) d 0 As long as the span Δu is smaller than the size of one lens of the array, only the central pixel receives light in each rendered view. There is no disparity between any of the recorded views. It is only when the spot size Δu spills over to a second lenslet that a second pixel on the rendered views receives light. Moreover, that second pixel registers in the views by shifting inwardly or outwardly with respect to the central pixel depending on the sign and amplitude of z. Hence, the disparity equals a measurable pixel in the two views that are the most separated angularly (views θ and θ in Fig., for instance) if the spot size Δu gets just over the dimension of one lenslet. If its size gets just bigger than one lenslet, if n is the number of pixels per lenslet and p the pixel pitch, then the span is Δu ¼ np. The span Δu is a continuous quantity, but if the disparity is not calculated with subpixel accuracy, distance d 0 þ z is not evaluated continuously but in steps, and so is Δu. The distance Δz by which the point object shall be moved from its initial position d 0 for generating a disparity of pixel is the depth discrimination: d 0 þ Δz ¼ d 0 np A f d 0 : () Rearranging Eq. () in order to emphasize the relative depth discrimination leads to: Δz d 0 ¼ np A d 0 f f : (6) The quantity np is measured in microns, whereas A is measured in millimeters. Equation (6) can be Taylor developed, and by stopping at the linear term we get a very simple expression of the depth discrimination: Δz np d 0 A d0 f : (7) Although it can be approximated, the aperture size of the entrance pupil A is known only to the lens maker. A more straightforward parameter to use would be the lens f -number, k. Another important lens characteristic is the focal length f, but with varying sensor sizes, the field angle of view is a more practical measure than the focal length. We can Journal of Electronic Imaging Jan Mar 202/Vol. 2()

4 Fig. 3 Field angle relationship to sensor s size and focal length. eliminate A and f from Eq. (7) by introducing the field of view α, the f -number k, and the size of the sensor D. Those parameters as defined in Fig. 3 are related for small α by the expressions f D and A ¼ f α k D αk : (8) Our final relative depth discrimination formula is then Δz α 2d0 Lk ; (9) d 0 D where L ¼ np is the size of a lens from the array; k is the main lens f -number; α is the field of view, in radians; D is the size of the sensor, in millimeters; and d 0 is the distance of the object, also in millimeters. Good depth discriminations are achieved for small results of Eq. (9). As this equation is a parametric description of the depth-sectioning capability of a plenoptic camera, it is easily possible to investigate each parameter separately and optimize all the critical components of the system for the best possible depth discrimination. 2. Focusing Distance d 0 To get small values for Eq. (9), it is best to use the camera at small focusing distances d 0. Depth discrimination gets worse at bigger distances, which is the case for any depth-measuring camera system. 2.4 Lenslet Size L and elemental image resolution n The elemental image is the image of the lens entrance pupil made by a lenslet onto the sensor. There are as many elemental images on the sensor as the number of lenslets in the array. The resolution of the elemental image, which is also the number of views, is n, since the analysis is restricted to a D case. In Fig., the elemental image resolution is n ¼. In order to get better depth performances, L needs to be small, and this means that the elemental resolution or the number of pixels per lenslet needs to be the smallest possible. Realizations of plenoptic cameras have used widely varying numbers of pixels. In their original paper, Adelson and Wang 4 reported used pixels, and we have seen reports 6 of up to 4 4 pixels. The number of pixels per lenslet has been kept small in order to render views with sufficient resolution, as the number of views trades off with the resolution of each view. Equation (9) teaches that it is also in the interest of the camera s depth discrimination performances to keep that value small. We will see in the next subsection that a plenoptic camera with good depth discrimination requires a big sensor area. Usually, a big sensor area implies a camera with one sensor, as opposed to those with three sensors and a color splitter. Typically, big sensors meant for producing color images out of a single device are provided with a Bayer color filter array. The color filters are arranged in quad pixels, two of them being green, one blue, and one red. If the number of pixels n per lenslet is even, then by rendering the views, we would get some views with just red and green pixels; adjacent views would have just blue and green pixels, which is improper to render the missing colors. So, the number n must be odd. There are other considerations to take into account; one of them is depicted in Fig. 4, which shows how the lenslet projects images of the pixels that are below one lenslet onto the exit pupil of the main lens. In this case, we have illustrated an elemental image with 3 horizontal and vertical 2.2 f -Number of Main Lens k Equation (9) teaches also that depth discrimination is better for small values of k. Small values of k mean big lens apertures. This is physically intuitive, since opening up the lens aperture provides the camera with a larger baseline for a more precise triangulation. So, wherever possible, main lenses with k ¼ 2.0 or 2.8 are the preferred choice. By stopping the lens down to.6, the relative depth discrimination worsens by a factor of 2 compared with an f -number of 2.8. Our analysis is based on first-order geometrical optics, but we will report in Sec. 4 how higher-order aberrations of the optics will impact the depth resolution. It is known that high-aperture lenses have also more aberrations, hence k ¼ 2.8 is a good compromise that maximizes the relative depth resolution and is still a reasonable system design value for lenses with limited aberrations. 2.3 Pixel Pitch p In Eq. (9), L ¼ np, where p is the pixel pitch. To get small values for Eq. (9), L should be a small value itself and hence the depth discrimination is better for sensors with a small pixel pitch. Fig. 4 Sampling of main lens aperture. Journal of Electronic Imaging Jan Mar 202/Vol. 2()

5 Table Various big area sensor characteristics. lenslet s size will suit the purpose of achieving the best possible depth resolution. Width (mm) Height (mm) Pixel pitch (μm) Diagonal (mm) Phantom Arri D Red One Scarlet FF Epic pixels. In order for the lens baseline A to be optimally used, the images of the pixels projected onto the main lens need to fill the lens aperture to get the biggest possible value of A. A value of pixels seems to be the best compromise for depth discrimination. However, autostereoscopic displays that are on the market today do not need vertical views, and a value of 3 pixels suits at the same time the need to fill the lens aperture and the current technology need for just horizontal views. In practice, the pixels at the edge of an elemental image are unusable due to light spillage from adjacent elemental images. Therefore, a value of 7 pixels seems to be the best compromise. This would lead to 3 ¼ views. This value is optimal regardless of future evolutions of sensor technology, as the parameter L is an independent quantity in Eq. (9). As pixel pitch get smaller, so also does L, and our recommended 7 pixel Pixel size of the sensor in µm Pixel size of the sensor in µm 0 Field angle 4.00, d0=3.0m, k=2.0 Arri D-2 20 Epic64 ScarletFF3 Red One Size of the sensor in mm 20 3 Phantom 6 Epic64 ScarletFF3 Red One Size of the sensor in mm 2 Field angle 22.0, d0=3.0m, k=2.0 Arri D-2 Phantom 6 Pixel size of the sensor in µm Pixel size of the sensor in µm 2. The Field Angle α and sensor size D These two parameters are the most important because they influence depth discrimination in a quadratic manner. If the field angle is divided by a factor of 2 by replacing, for instance, a regular 0-mm lens on a 3-mm camera with a 0-mm lens depth discrimination gets better by a factor of 4. Similarly, if sensor dimension D is doubled, depth discrimination is four times better. For an in-depth analysis, we set some of the parameters of Eq. (9) according to the previous subsection s discussions. The main lens f -number k is 2.0, and the number of pixels per lenslet is chosen as 7, which leads to 3 usable views. We will calculate the depth discrimination for α ¼ 4 deg and 22. deg field of view for d 0 ¼ 3 and m, where α ¼ 4 deg is a standard configuration, as it is the field of view of a full-field single lens reflex camera with a 0-mm lens. We will consider sensors of various sizes and pixel pitch, as listed in Table. The value that is used for D in Eq. (9) is the diagonal of the sensor. Figure shows the relative depth discriminations Δz d 0, whose values are shown as contours. The first graph shows the performances of a system for α ¼ 4 deg at d 0 ¼ 3m. As an example, the Scarlet Full Frame (FF) 3, which is a 3-mm sensor, would distinguish two objects separated by % of the distance, hence its depth resolution is 30 cm at 3 m focusing distance. The second graph is calculated for the same parameters but where d 0 ¼ m. In the following two graphs, the field angle has been reduced to half its value, and the Scarlet FF3 sensor has a depth Field angle 4.00, d0=.0m, k=2.0 0 Arri D-2 Epic64 ScarletFF3 Red One Size of the sensor in mm Field angle 22.0, d0=.0m, k=2.0 Arri D-2 Phantom 6 Epic64 ScarletFF3 Red One Size of the sensor in mm Phantom Fig. Relative depth discriminations, in % d 0. Journal of Electronic Imaging Jan Mar 202/Vol. 2()

6 Fig. 6 Schematic of the optical setup of a plenoptic add-on to a movie camera. resolution of 2.% at d 0 ¼ 3m, which is four times better than at α ¼ 4 deg.atd 0 ¼ m, it still has a depth resolution of 4%, which means 20 cm. A camera designed with this sensor, a main lens with an f -number of 2.0, and a lenslet array where each lenslet has the size of 7 pixels can distinguish two persons at m separated in depth by the thickness of one body. 2.6 Conclusion on the Depth Resolution of a Plenoptic Camera At this stage in this analysis, optical aberrations have been ignored; Eq. (9) is an upper performance criterion for the plenoptic camera to do depth measurement. It quantifies the best precision that can be achieved by this system. It also gives a direction for the design of the whole system, including lenslet size, sensor type, and main lens type. We can also infer that when equipped with a big area sensor, like the Epic 64, plenoptic cameras are effective because it is possible to discriminate two depth levels separated by less than 2 cm at a m distance and a field angle of 4 deg, which is quite precise. The lens aperture is hidden in Eq. (9) because it is embedded in the field angle and the sensor s size, but we can conclude that small apertures can be balanced by big sensor sizes and high pixel resolution for accurate depth mapping. 3 Design and Setup of a High-Definition Plenoptic Movie Camera Our purpose is to build a high-definition plenoptic movie camera with good depth resolution, so we will use the previous analyses to dimension the system. From Sec. 2., the sensor shall be the biggest possible with a small pixel pitch. The Red One with an Advanced Photo System type C- sized sensor and a 4k spatial resolution is a good candidate. In order to optimize depth sensitivity, we used a commercial main lens with an aperture of 2.8 and a field angle of 22. deg horizontal. To further have the best compromise between spatial and depth resolutions, the lenslet width has the dimension of five pixels. This means that this system is able to stream five views with different parallaxes, each view having a resolution of red/green/blue pixels. We used lenticular lenslets because vertical views are not needed in current autostereoscopic displays. Further, we wanted the system to be camera agnostic and built like an add-on to be put in front of any appropriate video camera. Of course lenslets have to be changed, but at least the camera sensor did not need to be customized by adding a lenslet array on top of it, which would require a redesign of the sensor and customized fabrication. 3. Plenoptic Add-on to a Movie Camera Figure 6 shows the general schematic of an add-on that can be used on any digital camera to transform it into a plenoptic system. This type of camera-agnostic system was used in the original work of Adelson and Wang. 4 The add-on consists of a main lens, which is in front of a telecentric lens, and a lenslet array. The plenoptic image is a real image formed just at the focal plane of the lenslets. The video camera is equipped with a high-quality macro relay lens, which makes a image of the plenoptic aerial image onto the sensor. As shown in Fig. 6, each individual lenslet spreads the aperture of the main lens onto five virtual pixels, forming a real plenoptic image in the focal plane of the lenslets. This image is then relayed onto the sensor. The add-on mechanics that have been developed especially for this demonstrator has the submicron precision adjustment required by the very small pixel pitch of the sensor. We have also developed specific adjustment procedures that will be explained in the following sections in order to achieve the required accuracy. Figure 7 shows a photograph of the realization with the different elements of the plenoptic add-on. The main lens has an f -number of 2.8, the captured field angle is 22. deg. The lenslet is a lenticular array, each lenticule having a pitch of 27 μm, the width of five pixels of the sensor. The most difficult part was to find a macro relay lens with an aperture of 2.8 because at that magnification, every commercial photo lens has an effective aperture that has an f -number of 4 or Fig. 7 Photograph of the demonstrator with different added parts. Journal of Electronic Imaging 007- Jan Mar 202/Vol. 2()

7 .6, and this is not in line with the best possible depth resolution. We decided to use two 0-mm.8 optics mounted head to tail and focused at infinity. This also has the advantage of controlling the magnification very precisely by slightly defocusing one of the lenses. 3.2 Raw Frame Extraction The only way to get access to individual pixel values was to record the shots onto a flash card. The video container stores JPEG (Joint Photographic Experts Group) 2000 compressed frames with a compression ratio of 9 (Redcode 36 format) or 2 (Redcode 28 format). The video stream could be played back in VirtualDub thanks to a plug-in provided by Red. Each individual pixel value could then be accessed by setting the lowpass 6 6 filter kernel coefficients of the plug-in to zero and assuming that the provided plug-in has interpolated missing red/green/blue values by a nearestneighbor demosaicing algorithm. A JPEG 2000 compression ratio at 9 or 2 does not disturb the reconstruction of each view after a compression and decompression of a plenoptic frame. This was tested on synthetically generated plenoptic pictures, where we interleaved every fifth column of five still-picture shots taken at five different parallaxes. 3.3 Geometrical Adjustment by the Power-per-View Split Procedure As the lenslets are cylindrical, Fig. 8 depicts the fact that when they are centered with respect to the pupil of the main lens, the central view number 3 receives more light power than view 2 or 4, which receives an equal amount of light power and in turn receives more power than views and. The best, and a very simple, pattern that we can use for adjusting the whole system is a white object. A white object has no texture, hence it does not matter at what distance it is placed from the main lens. Figure 9 is the picture that is then recorded on the sensor. We take only green pixels of the color filter array into account because each column Fig. 9 Plenoptic image of a perfectly aligned plenoptic system. and each line of the sensor contains green pixels. As the plenoptic image is then separated into five views, each view has a power balance that should be as depicted in Fig.. When each of the five views is uniformly illuminated and the power balance is symmetrical with the indicated values, the system is perfectly aligned. Figure shows an example of the lenslets being tilted with respect to the columns formed by the sensor s pixels. Each view is not uniformly illuminated, but the power balance is equal in each view. 3.4 Magnification Adjustment A magnification is achieved when all lenslets are imaged on exactly five columns of pixels on the sensor. They do not need to be aligned with the columns. Once the magnification factor has been set, they could need a horizontal shift in order to have the optical center of each lenslet facing exactly one column of pixels. We used the columns of pixels of the sensor as a sampling grid for analyzing the D spatial frequency of the image of the lenslet array. When a white object is used as an input pattern for the camera, the lenslet array produces on the input light bundle a periodic pattern. This pattern is imaged on the sensor by the relay lens. The relay lens focusing is adjusted until a magnification of is achieved. A magnification of means that the periodicity of the pattern Fig. Power balance per view in a perfectly aligned system. Fig. 8 Correlation of the cylindrical lenslets with the aperture of the main lens. Fig. Effect of a slight lenslet tilt for each view and for the power split. Journal of Electronic Imaging Jan Mar 202/Vol. 2()

8 drifting across the sensor area; but its influence on the views is not significant. This value of pixel shift is calculated globally by the FFT. Locally, the shift could be bigger at some fields of the sensor, because of magnification distortion, lens aberration, lateral chroma, or other influences, so that local miss-registrations have to be analyzed more deeply, as in the next section. Fig. 2 View mapping on the focal plane of the lenslet array. within the light bundle modulated by the lenslet array is exactly five pixels. The analysis has to be conducted on only the green pixels, because the red and blue Bayer color patterns miss every second column. The analysis tool used a fast Fourier transform (FFT) of the sum of two consecutive rows in order to get a green sample at each pixel location. A very high peak appeared at the normalized frequency of the spatial period of the lenslet array relative to the spatial period of the pixels of the sensor. Hence, a peak at the normalized frequency of px ðpixelsþ ¼0.2 is adjusted for. The magnification adjustment is possible by focusing one of the head-to-tail mounted lenses in the relay. At the best possible adjustment, the lenslet array imaged onto the sensor had a frequency of pixels, which means a magnification factor of G ¼ The system cannot be adjusted better than that, because of the total number of pixels per line, which is 992 in this adjustment, padded to 2048 for the FFT and a precision of pixel for the determination of the normalized peak frequency. A 2kwide captured image implies 2048 ¼ 4 lenslets, and hence the achieved absolute global pixel misalignment is pixel over the whole 2k range, which is a small value, in particular if the registration shift is slowly 4 Local View Crosstalk The add-on is composed of three main parts, as shown in Fig. 2. The main lens images the scene on a microlens array. Then, a relay lens is used to conjugate the detector (here a charge-coupled device with a Bayer pattern filter) with the microlens focal plane. The field lens makes the object space telecentric. As a consequence, the main lens pupil is divided into five slices, each slice creating a new point of view of the scene. On the microlens focal plane, we get five interleaved images corresponding to the five views of the scene. Those three elements are the core of the plenoptic effect. All of them are linked by several conditions. The main lens must have its image plane on the microlenses while at the same time having its pupil on the focal plane of the field lens, so that the object space of the microlenses becomes telecentric. The last condition links the microlens focal length with the main lens aperture: The main lens pupil diameter must be calculated considering that its image on the microlens focal plane should size five pixels of the detector. But the key fact is that if we get five interleaved images of five views onto the microlens focal plane, those views shall be conjugated onto the sensor exactly as they are. If this is not the case, view crosstalk will result. 4. Problem of View Overlap The relay lens is one of the most important parts of the system. We have seen that the five different views generated by the microlenses, on their focal plane, are interleaved like this: :::. Theoretically, the relay lens should be perfect, to avoid superposition of the views. In particular its point spread function (PSF) should be smaller than a pixel. If it is not, adjacent views will overlap. The effective views acquired by the sensor will be a superposition of successive views. The extreme views and have very different parallaxes, and a superposition here will blend together two images with noncontinuous disparity variations. Figure 3 shows an example where view number gets mixed with Fig. 3 (Left) Schematic of the view overlaps. (Right) Real results on a shot with a superposition of elements of view onto the image of view. Journal of Electronic Imaging Jan Mar 202/Vol. 2()

9 Fig. 4 Model used to simulate the PSF and vignetting. elements of view number. The red circles show the artifacts produced by mixed views. Extreme views are in general not usable, so that our prototype designed for five views is in practice a three-view system. Here we should have gone for a system with seven views, of which five would have been usable. 4.2 PSF and Vignetting Tradeoff for the Relay Lens The relay lens consists of two head-to-tail mounted lenses focused at infinity. We could both model and measure the PSF of the system. Both are in accordance. The lens is modeled as a variation of a double-gauss lens. It can be seen from the raytracing in Fig. 4 that the three top fields are vignetted because of the telecentric nature of the setup. The conclusion is that the lenses shall be used at their maximum apertures of.8 where vignetting has been measured to be flat across the field. On the other hand, at full aperture, the PSF is three pixels on-axis but up to eight pixels off-axis. The consequence is that in the center of the field, one view will be a mix of three adjacent views; and at the edge of the field, since we cannot shut the apertures because of the vignetting and because of the eight pixels, broad PSF disparity will be lost between views. 4.3 Accuracy of the Magnifying Power In this section, we will calculate the accuracy needed to achieve a magnification all over the field and compare it with the measured one from Sec Considering the total number of pixels on the sensor, we will calculate the accuracy of the magnifying power required so that there is less than pixel drift from the plenoptic to the relayed image. The original dimensions of the sensor are pixels. When adding pixel to the width, the new dimensions are pixels, which corresponds to a magnifying power equal to When removing pixel from the width, the new dimensions are pixels, which corresponds to a magnifying power of In conclusion, the magnifying power has to be From Sec. 3.4 we have measured the accuracy to be G ¼ The accuracy is six times greater than the needed one. The practical issues will be shown later on. 4.4 Coma, Spherical Aberration, Astigmatism All those aberrations have to be minimized, to avoid the problem of mixed views. Their only consequence is to spread the image spot, depending on the position in the field. For example, coma increases with the field. As a consequence, the different views are separable on-axis, while they are all blended off-axis. The result of inter-view disparity calculations is a disparity map that has values at its center and zero values at its borders, whatever the difference of parallax. Other aberrations are not field dependent, and they only blend some views together. The maximum acceptable PSF is two pixels in diameter. In that case, the first and last views are unusable. In our practical case, we have an on-axis PSF of three pixels. As a consequence, views and are not usable, and views 2 through 4 show depth-related disparities that can be shown on 3D displays but cannot be used for depth mapping. 4. Distortion Distortion leads to wrong results. To understand the problem, we have to consider one vertical stripe of rays that corresponds to one view (Fig. ). Normally it should reach one column of pixels. But if the relay lens has some distortion, it will reach various columns, depending on the position of each ray. The result of that aberration will be a distortion of each view, leading to wrong view assignment. One particular effect of the distortion will be strong color ringing of flat white images, which will be very visible on each view. To evaluate the maximum distortion acceptable, we have to calculate the distortion corresponding to one pixel. The distortion is maximal at corners, which is where we will study that aberration. Actually the maximal allowable distortion corresponds to one pixel (.4 μm). In our case the sensor is 24.4 mm 3.7 mm. Applying the well-known radial dependency distortion formulas to those sizes, we can conclude that the maximal distortion acceptable is 0.022% for a 28-mm field. For the sake of comparison with real-world numbers, the distortion of one 0-mm.8 lens used in the relay has been measured to be 0.2%. Fig. Effect of distortion bad view registration and color ringing. Journal of Electronic Imaging Jan Mar 202/Vol. 2()

10 4.6 Chromatic Aberrations Lateral color can be considered a variation of the magnifying power with the wavelength. So it leads us back to Sec. 4.3 concerning the accuracy of the magnifying power. 4.7 Field Curvature This is again an aberration that has to be eliminated, more particularly when the distance between the microlenses and their focal plane is about 0 μm. If the field curvature of the relay lens is about that dimension, some problems could appear, like focusing on the microlenses at the center of the sensor and on their focal plane at borders. The result will be an image that has areas that are plenoptic and areas that are not. Experimentally, the double 0-mm relay lens seems to have some field curvature aberration, because it is quite hard to focus the whole field simultaneously. This may be due to the fact that field curvature of each lens is additive. In practice, this will also result in views that have disparities at the center of the field and that will have a loss of any at the borders. Thus, 3D will flatten to 2D with the field. 4.8 Influence of the Main Lens The aberrations of the main lens can produce errors of depth registration. But problems here are different from the ones due to the relay lens. The relay lens blurs the image on the charge-coupled device, so that the calculated disparity cannot be related to actual depth, while the main lens can make some points look closer or farther away than they really are. The main lens aberrations disturb the depth/ disparity relation when they spread a point over one entire microlens, it means that the PSF should be at most 2 μm in diameter (this is the diameter of a microlens). This would otherwise lead to bad depth registration. The aberration constraints of the main lens are less critical for the global working of the plenoptic camera. 4.9 Possible Problems Induced by the Field Lens The field lens has been designed for the green wavelength. This leads again to different view registration for blue and red and color ringing on the views. The field lens should be made achromatic. Plenopticity Check In this section, we will report about a procedure to check on a recorded picture, the plenoptic quality of the camera. Quantifying numbers will indicate how big the area on the sensor will be that receives the view it is supposed to register. This can be taken as a plenopticity test for the camera, where each pixel will be marked as good or bad. Good means it will work in conformity with the plenoptic principle, and bad means it will not, i.e., it will not record the view it is supposed to record. This check will not quantify the amount of superposition of views of the kind described in the previous subsections of Sec. 4 but will show areas where, in spite of the crosstalk, the main contributor is still the view that is supposed to be registered there. Once the plenoptic system is well adjusted, we can repeat the procedure from subsection 3.3 and record a plenoptic image of a white field like the one shown in Fig. 7. In this image, we consider only the green pixels and fuse two successive rows in order to get continuous green pixels over one row. We can now search for local maxima over each row, and as soon as a local maximum is found, we mark that pixel from the plenoptic image by, for instance, setting its level to 2 and setting the levels of nonmaxima pixels to 0. Once this is done for each row, we can de-multiplex the plenoptic image into the five views and we get what can be seen in Fig. 4. According to Fig. 8, when the system is perfectly adjusted, there is only one view, the central view number 3, which should receive the maximum amount of light. In an ideal plenoptic system, all the maxima shall map into that view. This would mean that the system is perfectly adjusted; the lenslets have each exactly the size of five pixels and there is no miss-registration due to distortion or field curvature. In our prototype and in this example, where a 2k shot has been recorded, 9.% of the field maps correctly. We can see also on which pixels the mapping is correct, these being all white pixels from the central view in Fig. 6. Further, 4.4% of the field registers in a bad location shifted by one pixel in view number 2, and.8% misses also by one pixel in view number 4. The rest misses by two pixels. As a consequence, only the white marked pixels of view 3 can be considered for a disparity calculation. If we do the same for a 4k shot, the situation changes dramatically. We can see the result in Fig. 7. Only 2.7% of the pixels in view number 3 receive effectively the light passing through the central part of the main lens. The mapping of the views of a 4k image is so bad that displaying two of them on a stereoscopic display shows very shallow 3D effects limited to the central zone of the image and vanishing outwardly. This check works only for evaluating how good the relay registers for the central view because it is based on the difference of energy in each channel. Fig. 6 Plenopticity check on a 2k image. Fig. 7 Plenopticity check on a 4k image. Journal of Electronic Imaging Jan Mar 202/Vol. 2()

11 The disparity map is calculated by a hierarchical block matching and is postprocessed by a recursive cross-lateral filter; it is shown at the bottom of Fig. 8, where bright values represent positive disparity and dark ones negative disparities. Plenoptic technology has a high potential for video plus depth recording in real time to feed multiview displays. We showed that the realization of an external plenoptic adapter requires very high optical and mechanical precision to record good-quality multiple views. Putting the microlenses directly on the camera sensor avoids some of the constraints, but local view crosstalk remains a challenge if disparity has to be correlated with depth in high-resolution captures. Fig. 8 (Top) Views 2 and 4 of a 2k real scene shot. (Bottom) A calculated disparity map. 6 Prototype Evaluation and Conclusion Due to all previous analyses, we can say that we fell into some of the difficulties of plenoptic design. The most important one was the will to design a camera-agnostic add-on. We have used the geometrical optics rules from Sec. 2 to build up the best possible camera in terms of image resolution and 3D effect. When we introduced the wave theory to analyze the system, it proved to put an extremely high constraint on the relay lens, which shall be telecentric and have an aperture of.8 to get the telecentric rays issued from the main lens field lens lenslet array triplet. Its distortion should be limited almost to zero, its field curvature flat, its PSF smaller than one pixel, and the magnification ratio equal to It is questionable whether the add-on is easier to implement than customizing directly the sensor by replacing one of its windows by a lenslet array. The conclusion is that it is very difficult, to say the least, to have the sensor and the lenslet array separated by a piece of optics. The 4k video images played back on a 3D-capable display by just feeding two views to the system proved to be nonusable because only one-fourth of the area of each view got proper view registration. For 2k-shot video, there is a nice 3D effect on the screen because 60% of the pixel received good view registration. The 3D depth, on the other hand, was not big because of all the crosstalk (explained in Sec. 4) responsible for mixing one or two views on top of a third one, diminishing the disparity to values that are not correlated to a depth within the scene but to an optical aberration of the relay lens. Figure 8 shows two views from one frame of a real scene capture. The shooting distance was m and the lens aperture 2.8, and we limited the recording to a 2k plenoptic image, so that each video frame had a resolution of pixels. Figure 8 shows views 2 and 4. Focusing is set in the middle of the scene so that we have a positive disparity in the foreground and a negative one in the background. From the analysis in Sec. 2, we should have a total disparity of 30 pixels, and due to crosstalk between the views, there is a total estimated disparity of 2 pixels that cannot be related to a depth using Eq. (9), since crosstalk has not been taken into account. Acknowledgments This work has received research funding from the European Union 6th Framework Program under contract number IST OSIRIS ( References. P. Fua, A parallel stereo algorithm that produces dense depth maps and preserves image features, Mach. Vision Appl. 6(), 3 49 (993). 2. H. P. Moreton and B. E. Loucks, Optical system for single camera stereo video, U. S. Patent No.,83,33 ( Nov 998). 3. Z. Perisic, Apparatus for three dimensional photography, U. S. Patent No. 6,72,00 B2 (3 April 2004). 4. E. H. Adelson and J. Y. Wang, Single lens stereo with a plenoptic camera, IEEE Trans. Pattern Anal. Mach. Intell. 4(2), 99 6 (992).. E. H. Adelson, Optical ranging apparatus, U. S. Patent No.,076,687 (3 Dec 99). 6. R. Ng et al., Light field photography with a hand-held plenoptic camera, Stanford Tech. Rep. CTSR (200). 7. A. Subramanian et al., Segmentation and range sensing using a movingaperture lens, IEEE International Conference on Computer Vision 2, 00 (200). 8. V. Drazic, Optimal depth resolution in plenoptic imaging, 20 IEEE International Conference on Multimedia and Expo, pp (9 23 July 20), Singapore, ICME 20. Valter Drazic is a senior scientist at Technicolor Research & Innovation France. He received an engineering degree in optics from Ecole Nationale Supérieure de Physique de Strasbourg (ENSPS) in 988 and a Ph.D from the University of Karlsruhe, Germany, in 993. He joined Thomson/Technicolor in 993 as an R&D engineer to work in the development of projection TV. In 200, he joined Thomson Indianapolis to participate in the development of the first commercialized LCOS rear projection TV. From 2007, he was in charge of the 3D Natural Content Acquisition System within the European OSIRIS funded project and developed a multiview single lens camera. His main research area covers projection technologies, nonimaging optical systems, optical design, flat displays, HDR, color management, 3D, and disparity map calculation. Arno Schubert started his career as a professional photographer before graduating as an optical engineer at the Cologne University of Applied Science in 990. He joined Thomson/Technicolor in 99 and was involved in different programs as research engineer and project leader covering projection technologies, flat displays, HDR, color management, and 3D, including the European funded research project OSIRIS with the study on single lens multiview acquisition. Today he is based in Rennes, France, leading a research group on 3D scene production tools. Journal of Electronic Imaging 007- Jan Mar 202/Vol. 2()