Image formation analysis and high resolution image reconstruction for plenoptic imaging systems

Image formation analysis and high resolution image reconstruction for plenoptic imaging systems Sapna A. Shroff* and Kathrin Berkner Computational Optics Sensing and Visual Processing Research, Ricoh Innovations Inc., 88 Sand Hill Road, Suite 115, Menlo Park, California 9405, USA *Corresponding author sapna@rii.ricoh.com Received 14 November 01; accepted 18 January 013; posted 1 February 013 (Doc. ID 179899); published 0 March 013 Plenoptic imaging systems are often used for applications like refocusing, multimodal imaging, and multiview imaging. However, their resolution is limited to the number of lenslets. In this paper we investigate paraxial, incoherent, plenoptic image formation, and develop a method to recover some of the resolution for the case of a two-dimensional (D) in-focus object. This enables the recovery of a conventional-resolution, D image from the data captured in a plenoptic system. We show simulation results for a plenoptic system with a known response and Gaussian sensor noise. 013 Optical Society of America OCIS codes 110.1758, 100.3010, 100.6640, 070.0070, 100.1830, 100.0100. 1. Introduction Plenoptic imaging systems are being explored in many computational imaging applications, such as [1 3] computational axial refocusing, imaging different perspective angles, imaging with varying depths of field [4], and high dynamic range imaging. Plenoptic systems also find application in multimodal imaging [5] using a multimodal filter array in the plane of the pupil aperture. Each filter is imaged at the sensor, effectively producing a multiplexed image of the object for each imaging modality at the filter plane. The filter modalities may be spectral, polarization, neutral density, etc. A known drawback of plenoptic imaging systems is their loss of spatial resolution. A lenslet array at the conventional image plane spreads light on to the sensor. Every pixel behind a lenslet corresponds to a different angle or a different wavelength (in the case of multispectral imaging). Therefore, each angle image, or each wavelength image, is restricted to as many pixels as lenslets. When the multiangle images are further processed to obtain a refocused image, the result is also low resolution. A solution to this issue 1559-18X/13/100D-10$15.00/0 013 Optical Society of America has been discussed from the perspective of refocusing in prior work [6], using the repetition of an object point in multiple subimages, assuming priors on Lambertianity and texture statistics at the object surface, and using digital superresolution techniques. But the algorithm requires some amount of object defocus and does not perform when the object is in focus at the lenslet array. In the multimodal or multispectral imaging application, different pixels behind a lenslet give images with different modalities, such as different wavelength images or different dynamic range images. They are not processed further for refocusing. The object plane in this case is focused at the lenslet array, and the resultant low-resolution multimodal images do not give much perspective benefit. We did not find any literature supporting a higher resolution solution for the case of such an in-focus object. Plenoptic systems are flexible in allowing a user to obtain refocused images, multiview images, or multimodal images with minimal modifications of the same hardware. If it were possible to use the same hardware to obtain a two-dimensional (D) image with similar resolution as a conventional imaging system, that would further increase the flexibility and usability of these systems. Keeping in mind the in-focus object necessary for the multimodal D APPLIED OPTICS / Vol. 5, No. 10 / 1 April 013

application, and the insufficient prior work on recovering high resolution for the in-focus object, in this paper we propose a method to extract an image with resolution approaching that obtained from a conventional imaging system. The solution we propose works for defocused objects, as well. In this paper we do not discuss the multimodal, refocusing, or light-field aspects of plenoptic systems and concentrate only on in-focus resolution recovery. In order to investigate the possibility of recovering spatial resolution in this system, it would be useful to understand image formation in a plenoptic system. Reference [7] is a related paper connecting Wigner distributions to light fields. Reference [8] also discusses light fields using generalized radiance and a radiance coding perspective. However, they do not discuss wavefront-propagation-based image formation. Reference [6] discusses an approximation to an image formation model, but their response matrix contains some geometric-optics-based approximations that do not support the in-focus case that interests us. Since we did not find any literature discussing detailed field propagation in a plenoptic system, in this paper we first analyze wavefront propagation for the case of paraxial, incoherent systems, such as fluorescence microscopes, use it to develop a forward image formation model, and propose a method to obtain high-resolution images. We provide simulation results for a D object such as may be used in multispectral plenoptic systems.. Incoherent Image Formation in a Paraxial Plenoptic System We consider the layout of a general plenoptic system. Figure 1 is a schematic diagram of such a system, not drawn to scale. The object, denoted by the coordinates ξ; η, is imaged by a main lens, which may be any general paraxial imaging system, considered, for the sake of simplicity, as a single thin lens. The spatial coordinates at the plane of the pupil aperture of the main lens are denoted by x; y. This lens produces a primary image of the object at the plane of the lenslet array, which is denoted by the coordinates u; v. The main lens is considered to be a distance z 1 from the object, and the lenslet array is placed at a distance z from the plane of the main lens. The sensor is placed a distance z 3 from the lenslet array. We denote the sensor plane coordinates by t; w. The focal lengths of Fig. 1. Schematic layout of a plenoptic imaging system, with the first subsystem containing the main lens (solid gray), and the second subsystem containing the microlens array (dotted gray) [10,11,1]. the main lens and the lenslet array are given by f 1 and f and their diameters are D 1 and D, respectively. In this manuscript we define the term image as a replica of a target. The target in this paper may be the object, the pupil aperture, or filters in the pupil. We define each separately as we introduce them. The main lens produces an image of the object, henceforth called the primary image. This is considered the first imaging subsystem, marked in solid gray in Figure 1. The lenslet array is placed approximately at the plane of this primary image. Each lenslet then images the pupil of the main lens at the sensor plane. This is the second imaging subsystem, also marked in Figure 1 in dotted gray, which partially overlaps with the first subsystem with distance z in common. We refer to the data collected at the sensor behind a single lenslet as the pupil image and the data collected behind the entire lenslet array the plenoptic image. In order to analyze this optical layout and its impact on the data collected at the sensor we perform a wave analysis of this system. A. Wave Propagation analysis Consider the first imaging subsystem, where the main lens has a generalized pupil function denoted as P 1, and its corresponding impulse response is given as [9] h 1 u; v; ξ; η ez 1 e z λ exp u v z 1 z z ZZ exp ξ η dxdyp z 1 x; y 1 1 exp 1 1 x y z 1 z f 1 exp u Mξx v Mηy ; z (1) where λ is the imaging wavelength, k π λ, and the magnification from object to the primary image plane is M z z 1. Substituting x 0 x λz and y 0 y λz, the above equation becomes h 1 u; v; ξ; ηe z 1 e z exp u v z exp ξ η z 1 ZZ M dx 0 dy 0 P 1 x 0 λz ;y 0 λz 1 exp 1 1 z 1 z f 1 x 0 λz y 0 λz expf jπu Mξx 0 v Mηy 0 g () 1 April 013 / Vol. 5, No. 10 / APPLIED OPTICS D3

Defining the Fourier transform of P 1 xλz ;yλz exp f 1 z 1 1 z 1 f 1 xλz yλz g as h 0 1 u;v makes Eq. () h 1 u; v; ξ; ηme z 1 e z exp u v z exp ξ η h 0 1u Mξ;v Mη z 1 Substituting ξ 0 Mξ and η 0 Mη in Eq. (3), and imaging an object field, U o, we obtain the field of the primary image formed by the main lens (3) U 0 i u; vu iu; v X m exp f X P u md ;v nd n u md v nd Assuming the lenslet imaging system is also paraxial, propagating this field to the sensor located at a distance z 3 from the lenslet array and using Eq. (4) gives the following field at the sensor (5) U f t; w ez 1 e z e z 3 X jλz 3 M exp X t w exp md z 3 f nd m n ZZ 1 dudvp u md ;v nd exp 1 1 u v z z 3 f t exp u md w v nd z 3 f z 3 f ZZ ξ dξ 0 dη 0 0 U o M ; η0 ξ 0 η 0 exp h 0 1 M z 1 M M u ξ0 ;v η 0 (6) U i u; v ez 1 e z M exp u v z ZZ ξ dξ 0 dη 0 0 U o M ; η0 M ξ 0 η 0 exp z 1 M M h 0 1 u ξ0 ;v η 0 ez 1 e z M exp u v z u U o M ; v u exp M z 1 M v h 01 M u; v ; (4) where the symbol denotes a convolution. In a conventional system the sensor would be placed at this plane and ju i j would give the intensity of the conventional image. In a plenoptic system [], however, a lenslet array is placed at this primary image plane. We assume each lenslet has a diameter D, focal length f, a pupil function given by P, and there are M N such lenslets in the array. The field after traversing the lenslet array is described by The analysis up to this part was briefly discussed in our prior work [10]. Now we introduce incoherence into the system, such as from an object with fluorescent emission hu o ξ 0 ; η 0 U o~ξ 0 ; ~η 0 i I o ξ 0 ; η 0 δξ 0 ~ξ 0 ; η 0 ~η 0 (7) Using Eqs. (6) and(7), the intensity at the sensor plane is given by ZZ ξ I f t; w dξ 0 dη 0 0 I o M ; η0 e z 1 e z e z 3 M jλz 3 M exp t w z 3 X X exp md f nd m n ZZ dudvp u md ;v nd 1 exp 1 1 u v z z 3 f t exp u md w v nd z 3 f z 3 f h 0 1 u ξ0 ;v η 0 (8) D4 APPLIED OPTICS / Vol. 5, No. 10 / 1 April 013

We introduced this analysis briefly in [11,1]. In the following section we explore this image formation model in greater detail with analyses and simulations. B. System Response In this paper we refer to the field of the conventional response for a point source imaged through the main lens, at the conventional or primary image plane, as the impulse response (IPR), and the square of its magnitude as the point spread function (PSF). This is typically an Airy disk for a conventional diffractionlimited imaging system with a circular pupil, and is spatially invariant within an isoplanatic region. The response of the plenoptic system must include the impact of the lenslet array and propagation to the sensor, in addition to the system response up to the plane of the lenslet array. Thus it should include the effect of the traditional PSF and contain additional information. As we show in the following work, this response does not look like an Airy disk, even when the system has no aberrations. We also show that the response for the plenoptic system is not as spatially invariant as the conventional PSF. For the sake of clarity and the convenience of being able to distinguish clearly between the conventional PSF, which characterizes only the first subsystem, versus the system response for the complete plenoptic system, we call them by different names. Since the response of the complete plenoptic system resembles an image of the pupil aperture of the main lens, in this paper we refer to it as the pupil image function (PIF) [10,11,1]. The term I o ξ 0 M; η 0 M in Eq. (8) refers to the intensity at a single point in the object plane. All the terms within the squared magnitude are related to parameters of the optical system, such as pupils, impulse responses for the main lens and lenslet, system distances, and can be included in the overall system response, PIF, written as PIF t;w ξ 0 ;η PIFt; w; 0 ξ0 ; η 0 e z 1 e z e z 3 jλz 3 M exp t w z 3 X X exp md f nd m n ZZ dudvp u md ;v nd 1 exp 1 1 u v z z 3 f t exp u md w v nd z 3 f z 3 f h 0 1 u ξ0 ;v η 0 (9) The PIF is a four-dimensional (4D) function of the sensor plane coordinates t; w and the scaled object coordinates ξ 0 ; η 0. Each point ξ 0 ; η 0 in the object plane has a corresponding response PIF t;w ξ 0 ;η0 that falls across all the sensor pixels. We simulated the system response using these equations. We assumed an imaging wavelength 450 nm, a thin aberration-free main lens with focal length 40 mm and diameter 4 mm. In most of the paper we assumed a single thin lenslet with focal length 4 mm and diameter 0.16 mm. Toward the end of the paper we discuss an 11 11 lenslet array with the same individual specifications for focal length and diameter of each lenslet. We assumed z 1 and z of 65 and 104 mm, respectively. This created a system where a flat object was imaged by the main lens and got perfectly focused at the primary image plane, which was also the plane of the lenslet array. The sensor was placed at a distance 4. mm from the lenslet plane. We assumed a sensor with pixel size of 1.3 μm, and a dynamic range of 0 55. We simulated the field of the conventional impulse response, IPR, using Eq. (4) with an impulse as the object. For pictorial succinctness we display its magnitude square, PSF, in Figure (a). This field is displayed, cropped by the extent of a single circular on-axis lenslet. In Figs. (c) and (e) we do the same for point sources that were off-axis by 50% and 85% of the on-axis lenslet radius. Figs. (b), (d), and (f) show the PIF responses of the entire plenoptic system for the same point sources, simulated using Eq. (9). Figures (a) and (b) show the case of an on-axis point source imaged through a single on-axis lenslet in a plenoptic system. The field associated with the conventional Airy disk reached the lenslet, got cropped by its extent and further propagated to the sensor plane. Figures (c) and (d) show the case of a slightly off-axis point source. Here, some of the rings of the Airy disk were cropped by the extent of the lenslet. Figures (e) and (f) show, as the object went more off-axis, the effect of the cropping was more severe. This caused diffraction effects and spatial variation in the effective PIF response for each object point, as seen in Figures (b), (d), and (f). The plenoptic system response, PIF, in Figure showed differences in spatial extent and diffraction patterns when compared to the PSF. Also, the PIF response includes the effect of the main lens (and therefore the PSF), and the rest of the system, while the converse is not true. Therefore, we consider the PIF response when dealing with the plenoptic system. The same equations can be used to obtain PIF responses for a defocused D object. C. Image Simulation When the integral in Eq. (8) is performed over all the ξ 0 ; η 0 coordinates, we effectively sum over all the points in an object and the result is a function of only sensor coordinates t; w. This implies that information from a single object point is spread out over multiple sensor pixels, and, conversely, a single sensor pixel contains information from multiple object points. We simulated this linear system by arranging 1 April 013 / Vol. 5, No. 10 / APPLIED OPTICS D5

the object and sensor data in column vectors Iv o and Iv f, and the PIF t;w ξ 0 ;η for a given object point 0 ξ0 ; η 0 in a column vector, and concatenating the column vector responses for every point in the object plane in a matrix PIFv as follows 6 4 I 1;1 f I 1; f I ;1 f I ; f I T;W f Iv f PIFvIv o ; (10) 3 3 PIF 1;1 1;1 PIF 1;1 1; PIF 1;1 M;N PIF 1; 1;1 PIF 1; 1; PIF 1; M;N PIF 1;W 1;1 PIF 1;W 1; PIF 1;W M;N PIF ;1 1;1 PIF ;1 1; PIF ;W M;N 7 6 76 5 4 54 PIF T;W 1;1 PIF T;W 1; PIF T;W M;N I o;1;1 I o;1; I o;;1 I o;;1 I o;m;n 3 7 5 (11) Note that PIFv is just a matrix-style rearrangement of the 4D quantity PIF t;w ξ 0 ;η0. Thus, the set of data captured by the sensor in a plenoptic system is a product of the response matrix, PIFv, and the object intensity vector, Iv o. This formulation was used to simulate image formation in a plenoptic imaging system with a single lenslet. We used a D object containing a sharp slant edge, dark towards the left and bright toward the right, shown in Fig. 3(a), as the object in this simulation. Figure 3(b) shows the data at the sensor when the object was perfectly focused at the lenslet array plane. The sensor data appeared disk-like, similar to an image of the circular main lens aperture. It contained no direct appearance of the edge in the object. When the object was moved closer to the main lens, as shown in the schematic layout in Fig. 4 (not drawn to scale), the system formed a roughly defocused, upside-down image of the object. Our simulation showed this effect for different amounts of defocus in Figs. 3(c) and 3(d) with the formation of a subimage at the sensor plane, where the dark portion Fig.. (a), (c), and (e) show the PSF responses on a stretched scale for impulses that are laterally shifted 0%, 50%, and 85% of the lenslet radius away from the optical axis, respectively, cropped by the extent of the on-axis lenslet. (b), (d), and (f) show the overall PIF responses of the system for the same impulses. Fig. 3. Simulation of incoherent plenoptic image for a single lenslet. (a) shows the pristine object, (b) is the sensor data when the object plane is focused at the plane of the lenslet array, and (c) (f) are the sensor data when the object is defocused by, 5,, and 5 mm, respectively. D6 APPLIED OPTICS / Vol. 5, No. 10 / 1 April 013

Fig. 4. Schematic layout of a plenoptic imaging system when the object is moved closer to the main lens. is now to the right and the bright portion is to the left. Figure 3(c) still shows energy from the pupil imaging subsystem and the circular shape of the pupil. As the amount of defocus increased in Fig. 3(d), the energy from the pupil imaging subsystem was lower than from the combined object imaging subsystem. Also, the object shown in Fig. 3(a) was surrounded by zeros. Therefore, Fig. 3(d) shows a roughly defocused version of this object and low energy around it. Figure 5 shows a similar illustration of the system when the object was moved away from the main lens. Here the primary image was formed before the lenslet array and the lenslet array relayed an upright defocused image of the object at the sensor. Figures 3(e) and 3(f) show this was indeed the case; the dark portion of the object is seen to the left in the data and the bright part is to the right. Here again, the amount of defocus in Fig. 3(e), being less than that in Fig. 3(f), shows more effect of pupil imaging with the object imparting only a shading of the data. Figure 3(f) shows more energy from the defocused image of the object. This appearance of flipped subimages of a defocused object was experimentally demonstrated by [1], corroborating the theory of image formation we have proposed. 3. Recovery of High Resolution Image The image formation theory and simulations developed in the previous section indicate the possibility of estimating the object intensity via solving a linear inverse problem, namely, calculating an estimate Îv o of the object by solving the following optimization problem Îv oargmin Ivo PIFvIv o Iv f (1) subject to possible constraints on Iv o. Solutions to this problem may be implemented in closed-form or via iterative algorithms, depending on the choice of the norm (or combination of norms) and the constraints on Iv o. The system matrix, PIFv, does not convolve with the object intensity if the object is focused at the plane of the lenslet array. Therefore, direct deconvolution techniques, such as Wiener filtering, are not useful for our problem. The PIFv matrix may be obtained by good system calibration, or by simulating or estimating an approximation to this response. In this paper we show results for a wellknown error-free PIFv matrix. More work could be done to incorporate approximations or errors in the knowledge of the PIFv matrix. A. Noise-Free Case For the results shown in this section we assumed a noise-free PIFv matrix, noise-free sensor data, and used pseudoinverse PIFv to compute a closed-form solution to Eq. (1). Figure 3(a) shows the object used in this simulation, a sharp slant edge, dark to the left and bright to the right of the edge. Figures 6(a) and 6(b) show a conventional image of the edge taken with just the main lens and the same after deconvolution using a Wiener filter in MATLAB [13] with a noise-to-signal power ratio of 0.00001. The sharpness of the edge was enhanced after deconvolution, but to a finite extent due to the low-pass nature of an imaging system. There was some ringing owing to oversharpening. The plenoptic system sensor data for the in-focus case in Fig. 3(b) showed the least similarity to object spatial content and portended to be the most difficult case to reconstruct. Comparatively, the sensor data in the defocused cases contained an increasing amount of direct object spatial content, like the shading and object imaging seen in Figs. 3(c) to 3(f). Therefore, here we mainly discuss the reconstruction using the more difficult, in-focus plenoptic data, shown in Fig. 3(b). To the best of our knowledge, this case has not been discussed in prior work. Figure 6(c) shows one such reconstruction, where we used a pseudoinverse to invert the matrix PIFv. Since PIFv is assumed to be known and there is no noise in this simulation, the pseudoinverse worked reasonably well, giving us a clearly visible edge. But it suffered from a dominant twin-image artifact. In a plenoptic system with radially symmetric lenses, a given focused object point has a diametrically opposite object point, which gives a flipped version of almost the same optical response. Therefore, a column in the resultant PIFv matrix, which is the vectorized response for one object point, is an upside-down version of another column in the matrix, Fig. 5. Schematic layout of a plenoptic imaging system when the object is moved farther from the main lens. 1 April 013 / Vol. 5, No. 10 / APPLIED OPTICS D7

corresponding to an object point that is located diametrically opposite in the optical system at the plane of the lenslet. The similarity of plenoptic systems to holography [14] and the resemblance of the twinimage problem to a similar issue in holography and phase retrieval [15] suggested that it could be handled by using a noncentrosymmetric response constraint for object points across a given lenslet. Since our main lens here was assumed to be aberration free, we incorporated local noncentrosymmetry in the shape, amplitude, or phase of each lenslet in our lenslet array. This imparted enough variation across the columns of matrix PIFv to avoid the twin-image reconstruction error. This reconstruction, shown in Fig. 6(e), was obtained using a lenslet with a notch at one side, shown in Fig. 6(f), making the response different for object points across the lenslet compared to the traditional circular lenslet pupil shown in Fig. 6(d). Other forms of lenslet shape, amplitude, and phase variations, shown in Fig. 7, were also able to reduce the solution space and eliminate the twinimage artifact. One could also leave the lenslets unaltered and place a filter or mask having amplitude or phase variations in a conjugate object plane to obtain such variation in the response. When the response across different parts of the object was different, the reconstruction got easier. Therefore, altering the main lens response could also help the reconstruction. The lenslet array is already in a plane conjugate to the object; therefore, adding such variations to the lenslet array was convenient for us and reduced image processing complexity. One could also avoid using such noncentrosymmetric constraints and instead use advanced image processing algorithms and prior assumptions, such as object statistics, to help with the inversion. However, in this paper, we did not make any such assumptions about the object, and used algorithms available in MATLAB libraries [13]. The reconstruction obtained in Fig. 6(e) shows a clearly defined edge, similar to that in the original object. The sharpness of the reconstructed edge is no more than that seen in the deconvolved conventional image in Fig. 6(b). There are some ringing artifacts present in both cases. Issues like ringing are well known and easy to manage in a Wiener filter deconvolution, but the investigation and management of these issues in a plenoptic reconstruction provides interesting opportunities for future work. Thus, the proposed image reconstruction for the case of an in-focus D object, imaged through a plenoptic system with a single lenslet, a known system response, and a noise-free setting provided us a reconstruction with similar spatial resolution information about the object as that obtained in a deconvolved image from a conventional system. B. Noise at the Sensor There are various kinds of noise in this system that should be considered for a thorough examination of system inversion, such as sensor noise, or errors, approximations and uncertainties in the PIFv response matrix, and the three-dimensional (3D) nature of the object. All these sources of noise are worthy of extensive exploration and may involve further algorithmic modifications. In this paper, we discuss only the effect of sensor noise modeled as white Gaussian noise. Assuming N is a vector of independently distributed Gaussian random variables of zero mean and standard deviation, σ, the linear forward model for image formation can be stated as Iv f PIFvIv o N (13) In this section, we show simulation results for a plenoptic system with sensor noise of SNR 40 db. We simulated an array of 11 11 lenslets. We showed results with an on-axis lenslet in the previous section. Here we show results for reconstruction using Fig. 6. Simulation for a single lenslet. (a) Conventional image. (b) Deconvolved conventional image. (c) Reconstruction with a twinimage error, obtained using a circle shaped lenslet shown in (d), and (e) is a reconstruction without the twin-image error, obtained using a lenslet with a noncentrosymmetric shape, shown in (f). D8 APPLIED OPTICS / Vol. 5, No. 10 / 1 April 013

data from a single 3; 3 off-axis lenslet. We show the complete array reconstruction thereafter. Figure 8(a) is the portion of the object containing blurred, alternate bright and dark, elongated patches of different thicknesses and slants. Figures 8(b) and 8(c) show a conventional, noise-free image of this object simulated with just the main lens PSF, and, after deconvolution with a Wiener filter in MATLAB [13], with a noise-to-signal power ratio of 0.00001. We obtained a reconstruction using the plenoptic system sensor data for a single lenslet, with the object plane focused at the lenslet. Figure 8(d) shows the reconstruction for the noise-free case, obtained using a pseudoinverse approach. In Subsection 3.A we used a lenslet with a noncentrosymmetric shape to eliminate a possible twin-image artifact. In the results in this subsection, we used a lenslet with an amplitude gradient across its face, as shown in Fig. 7(b), to incorporate the noncentrosymmetric constraint in the system. The reconstruction in the noise-free case was similar in content to the deconvolved conventional image in Fig. 8(c). When Gaussian noise was added to the sensor data, the pseudoinverse based solution did not work. Figure 8(e) shows a reconstruction obtained by solving the linear inverse problem in Eq. (13) with a constrained linear least squares approach, using the MATLAB library function [13] for the same, imposing upper and lower bounds on the dynamic range during optimization. Noise clearly affected the reconstruction, but the result showed recovery of some object spatial content. The quality of the reconstruction dropped beyond the extent of the lenslet as seen by the shadow of the lenslet, which appears in the corners of the reconstruction. The amplitude gradient also affected the quality of the reconstruction. We also applied an iterative nonlinear least squares algorithm that allowed us to incorporate smoothing, and lower and upper bounds on the dynamic range during optimization, using the MATLAB library function [13] for the same. The result in Fig. 8(f) showed less ringing and better recovery of the object spatial content than the linear least squares approach in Fig. 8(e). The amount of spatial content recovered was not equal to that in the noisefree case, but was more than the traditional plenoptic in-focus image, where each lenslet contributes only one pixel to the image (computed as an average of all the sensor data behind a lenslet). Figure 9(a) shows the complete pristine planar object used in our simulations, which would cover the entire field of view of an 11 11 lenslet array. The white circles mark the portion of the object and Fig. 7. Noncentrosymmetry in shape as well as amplitude improved image reconstruction. (a) and (b) show lenslets with noncentrosymetric shape and amplitude, respectively. Alternatively, phase aberrations in lenslets were also useful. (c) and (d) illustrate the real and imaginary parts of noncentrosymmetric phase in a lenslet. Fig. 8. Simulation for a single lenslet. (a) Pristine object used in simulation. (b) Conventional image, no noise. (c) Deconvolved noiseless conventional image. (d) Reconstruction with pseudoinverse for noiseless data. (e) and (f) are reconstructions using linear least squares and nonlinear iterative algorithms for noisy data, SNR 40 db. 1 April 013 / Vol. 5, No. 10 / APPLIED OPTICS D9

corresponding reconstructions shown in the simulation results in Fig. 8. The object was simulated to be perfectly focused at the lenslet array. This gave us the plenoptic sensor data shown in Fig. 9(b). In this paper we assume negligible lenslet crosstalk for the in-focus object. Figure 9(c) shows the traditional plenoptic image that would be obtained for this object, Fig. 9. Simulation for an 11 11 array of lenslets. (a) Pristine object used in simulation. (b) Plenoptic sensor data. (c) Traditional single plenoptic image obtained by binning light behind each lenslet into 1 pixel. (d) Reconstruction with pseudoinverse for noiseless data. (e) and (f) are reconstructions using linear least squares and nonlinear iterative algorithms for noisy data, SNR 40 db for the case where the object is in focus at the lenslet array. (g) and (h) are the same when the object is defocused by mm. by binning all the sensor pixels behind one lenslet into one effective image pixel. The reconstruction obtained when there is no sensor noise, using the pseudoinverse approach and an amplitude gradient across the lenslet array, is shown in Fig. 9(d). This result contained 440 440 pixels, which is equivalent sampling to a conventional image with a sampling factor Q 3. The reconstruction showed visible recovery of spatial content. Figures 9(e) and 9(f) show the reconstructions obtained for the case of sensor noise with SNR 40 db using the linear least squares and nonlinear iterative algorithms discussed before. As observed, there remain distinct artifacts in the reconstruction, but more of the object s spatial content was visible in this result when compared to the traditional plenoptic image in Fig. 9(c). Our simulation results suggest potential for future research on improved reconstruction algorithms, especially in the noisy case. We also did some simulations when the object was defocused by mm, with sensor noise of SNR 40 db. Figures 9(g) and 9(h) show the reconstructions obtained for this defocused case. They were obtained using the same two algorithms used in the in-focus case in Figs. 9(e) and 9(f). Similar results were obtained on the other side of focus. Our results showed an improvement over the in-focus reconstructions and supported the findings in [6] that it is possible to recover spatial resolution for the defocused case. The aforementioned assumption of a known or calibrated response matrix, PIFv, in the algorithm implemented in this paper also implies prior knowledge of defocus. We note that the defocused case simulations here are preliminary because we did not incorporate magnification changes associated with defocus and the subsequent interlenslet overflow. The magnification change affects the field of view for each lenslet. Overflow of the same object point into two or more lenslets allows reimaging of the same object point under multiple lenslets. This is best handled by taking this redundancy into account, and performing the response inversion at a global level, rather than using the lenslet-by-lenslet processing we have implemented in this paper. This aspect of overflow has been discussed to some extent in [6], where the object spatial content is recovered using multiframe digital superresolution techniques. While their approach is different from ours, the outcome indicates that the defocused object case does lend itself to object reconstruction. We will discuss a global level implementation using our approach and additional image processing in a future publication. 4. Conclusion In this paper we have introduced a detailed analysis of paraxial image formation for incoherent plenoptic imaging systems using a D object. We have shown simulation results for image formation that are corroborated by experimental images observed in [1]. We used our forward model, simulated images for a sample optical system with a known system D30 APPLIED OPTICS / Vol. 5, No. 10 / 1 April 013

response, and proposed a reconstruction method that allowed the recovery of some of the object information beyond the traditional resolution limited by the number of lenslets []. We have shown simulation results for the case where there is no noise at the sensor, and when there is 40 db Gaussian noise. The solution to the linear inverse problem for the plenoptic system is challenging and could result in a twin-image problem similar to holography. In order to overcome this problem, we used a noncentrosymmetric response constraint, incorporated in the design of lenslets having noncentrosymmetric shape, amplitude, or phase. The noiseless case allowed the use of a simple pseudoinverse to obtain a reconstruction having quality comparable with a deconvolved conventional image taken with just the main lens (no lenslet array), where the deconvolution was based on the main lens blur. The reconstruction for the noisy case needed iterative optimization methods with constraints on smoothness and dynamic range of the reconstruction. The resultant reconstruction showed better resolution than the traditional plenoptic system resolution, although not comparable to a deconvolved conventional image. We are optimistic about obtaining further improvement with better image processing for the noisy case and will present work on this in the future. In this paper we do not explicitly work on multimodal imaging. So we do not show results with a partitioned pupil as would be necessary for multimodal imaging, and which would accordingly reduce the effective resolution. However, we refer to some of our prior work [16] dealing with system responses for a partitioned pupil and propose to show results on that specific topic in future. More work could also be done in the areas of response estimation, calibration, and uncertainty, and incorporation of a 3D object model or generalization of the image formation theory for other wide-angle systems, such as cameras and other imagers. Experimental verification of the theoretical framework proposed in this paper is on-going and will be discussed in future. We have proposed a deconvolution-like method to recover spatial resolution content in plentoptic imaging systems even when the object is focused on the lenslet array, which could allow a paraxial, incoherent plenoptic system to also be used as a conventional imaging system. References 1. T. Adelson and J. Wang, Single lens stereo with a plenoptic camera, IEEE Trans. Pattern Anal. Mach. Intell. 14, 99 106 (199).. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, Light field photography with a hand-held plenoptic camera, Tech. Rep. (Stanford University, 005). 3. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, Light field microscopy, ACM Trans. Graph. 5, 94 934 (006). 4. C. Perwass and L. Wietzke, Single-lens 3D camera with extended depth-of-field, Proc. SPIE 891, 89108 (01). 5. R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, Flexible multimodal camera using a light field architecture, in Proceedings of the IEEE International Conference on Computational Photography (IEEE, 009). 6. T. E. Bishop, S. Zanetti, and P. Favaro, Light field superresolution, Proceedings of the IEEE International Conference on Computational Photography (IEEE, 009). 7. Z. Zhang and M. Levoy, Wigner distributions and how they relate to the light field, IEEE International Conference on Computational Photography (IEEE, 009). 8. D. J. Brady and D. L. Marks, Coding for compressive focal tomography, Appl. Opt. 50, 4436 4449 (011). 9. J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1986). 10. S. A. Shroff and K. Berkner, Defocus analysis for a coherent plenoptic system, in Frontiers in Optics, OSA Technical Digest (Optical Society of America, 011), paper FThR6. 11. S. A. Shroff and K. Berkner, High Resolution image reconstruction for plenoptic imaging systems using system response, in Computational Optical Sensing and Imaging, OSA Technical Digest (Optical Society of America, 01), paper CMB.. 1. S. A. Shroff and K. Berkner, Wave analysis of a plenoptic system and its applications, Proc. SPIE 8667, 86671L (013). 13. The Mathworks, Natick, MA, http//www.mathworks.com. 14. J. Goodman, Assessing a new imaging modality, in Optical Sensors, OSA Technical Digest (Optical Society of America, 01), paper JM1A.1. 15. J. R. Fienup and C. C. Wackerman, Phase-retrieval stagnation problems and solutions, J. Opt. Soc. Am. A 3, 1897 1907 (1986). 16. K. Berkner and S. A. Shroff, Optimization of spectrally coded mask for multi-modal plenoptic camera, in Computational Optical Sensing and Imaging/Information Photonics (Optical Society of America, 011), paper CMD4. 1 April 013 / Vol. 5, No. 10 / APPLIED OPTICS D31