Lensless Compressive Sensing Imaging

Transcription

1 ubtted January, 203 Lensless opressve ensng agng Gang Huang, Hong Jang, K Matthews and Paul Wlord Abstract n ths paper, we propose a lensless copressve sensng agng archtecture. The archtecture conssts o two coponents, an aperture assebly and a sensor. No lens s used. The aperture assebly conssts o a two densonal array o aperture eleents. The transttance o each aperture eleent s ndependently controllable. The sensor s a sngle detecton eleent, such as a sngle photo-conductve cell. ach aperture eleent together wth the sensor denes a cone o a bundle o rays, and the cones o the aperture assebly dene the pxels o an age. ach pxel value o an age s the ntegraton o the bundle o rays n a cone. The sensor s used or takng copressve easureents. ach easureent s the ntegraton o rays n the cones odulated by the transttance o the aperture eleents. A copressve sensng atrx s pleented by adustng the transttance o the ndvdual aperture eleents accordng to the values o the sensng atrx. The proposed archtecture s sple and relable because no lens s used. Furtherore, the sharpness o an age ro our devce s only lted by the resoluton o the aperture assebly, but not aected by blurrng due to deocus. The archtecture can be used or capturng ages o vsble lghts, and other spectra such as nrared, or lleter waves. uch devces ay be used n survellance applcatons or detectng anoales or extractng eatures such as speed o ovng obects. Multple sensors ay be used wth a sngle aperture assebly to capture ult-vew ages sultaneously. A prototype was bult by usng a L panel and a photoelectrc sensor or capturng ages o vsble spectru. ndex Ters opressve sensng, agng, lensless, sensor. NTOUTON OMPV sensng [][2] s an eergng technque to acqure and process dgtal data such as ages and vdeos [3][4][5][6]. opressve sensng s ost eectve when t s used n data acquston: to capture the data n the or o copressve easureents [7]. Wth copressve easureents, ages ay be reconstructed wth ar ewer easureents than the nuber o pxels n the orgnal ages. Thereore, by usng copressve sensng n acquston, ages are copressed whle they are captured, avodng hgh speed processng, or transsson, o a large nuber o pxels. The rst devce that drectly captures copressve easureents o an age s the sngle pxel caera o [8][9]. t s a caera archtecture that eploys a dgtal crorror array to peror optcal calculatons o lnear proectons o an age onto pseudorando bnary patterns. t has the ablty to The authors are wth Bell Labs, Alcatel-Lucent, 700 Mountan Ave, Murray Hll, NJ als: rstnae.lastnae@alcatel-lucent.co obtan an age wth a sngle detecton eleent whle saplng the age ewer tes than the nuber o pxels. The sae caera archtecture s also used or Terahertz agng [0][], and lleter wave agng [2]. These caeras all ake use o a lens to or an age n a plane beore the age s proected onto a pseudorando bnary pattern. Lenses, however, severely constran the geoetrc and radoetrc appng ro the scene to the age [3]. Furtherore, lenses add sze, cost and coplexty to a caera. n ths paper, we propose archtecture or copressve sensng agng wthout a lens. The proposed archtecture conssts o two coponents, an aperture assebly and a sngle sensor. No lens s used. The aperture assebly conssts o a two densonal array o aperture eleents. The transttance o each aperture eleent s ndependently controllable. The sensor s a sngle detecton eleent, such as a sngle photoconductve cell. ach aperture eleent together wth the sensor denes a cone o a bundle o rays, and the cones o the aperture assebly dene the pxels o an age. The sensor s used or takng copressve easureents. ach easureent s the ntegraton o rays n the cones odulated by the transttance o the aperture eleents. The proposed archtecture s derent ro the caeras o [8] and [3]. The undaental derence s how the age s ored. n both [8] and [3], an age o the scene s ored on a plane, by soe physcal echans such a lens or a pnhole, beore t s dgtally captured (by copressve easureents n [8], and by pxels n [3]). n the proposed archtecture o ths work, no age s physcally ored beore the age s captured. The detaled dscusson on the derence wll be gven n ecton. The proposed archtecture s dstnctve wth the ollowng eatures. No lenses are used. An agng devce usng the proposed archtecture can be bult wth reduced sze, weght, cost and coplexty. n act, our archtecture does not rely on any physcal echans to or an age beore t s dgtally captured. No scene s out o ocus. The sharpness and resoluton o ages ro the proposed archtecture are only lted by the resoluton o the aperture assebly (nuber o aperture eleents), there s no blurrng ntroduced by lens or scenes that are out o ocus. Mult-vew ages can be captured sultaneously by a devce usng ultple sensors wth one aperture assebly. The sae archtecture can be used or agng o vsble spectru, and other spectra such as nrared and lleter waves.

2 ubtted January, evces based on ths archtecture ay be used n survellance applcatons [6] or detectng anoales or extractng eatures such as speed o ovng obects. We bult a prototype devce or capturng ages o vsble spectru. t conssts o an L panel, and a sensor ade o a three-color photo-electrc detector. The organzaton o ths paper s as ollows. n the next secton, the archtecture o our work s descrbed. The related work s dscussed n ecton. The atheatcal orulaton or ages o the proposed archtecture s gven n ecton V, ollowed by a dscusson, n ecton V, o ultvew agng by usng ultple sensors wth one aperture assebly. n ecton V, ssues arsng ro practcal pleentatons o the archtecture are addressed. The prototype syste s descrbed n ecton V.. PTON OF AHTTU The proposed archtecture s shown n Fgure. t conssts o two coponents: an aperture assebly and a sensor. The aperture assebly s ade up o a two densonal array o aperture eleents. The transttance o each aperture eleent, T, can be ndvdually controlled. The sensor s a sngle detecton eleent, whch s deally o an nntesal sze. cene assebly ensor Fgure. The proposed archtecture. t conssts o two coponents: an aperture assebly and an nntesal sensor o a sngle detecton eleent. ach eleent n the aperture assebly together wth the sensor ors a cone o a bundle o rays, and the cones or the pxels o an age ach eleent o the aperture assebly, together wth the sensor, denes a cone o a bundle o rays, see Fgure, and the cones ro all aperture eleents are dened as pxels o an age. The ntegraton o the rays wthn a cone s dened as a pxel value o the age. Thereore, n the proposed archtecture, an age s dened by the pxels whch correspond to the array o aperture eleents n the aperture assebly. An age can be captured by usng the sensor to take as any easureents as the nuber o pxels. For exaple, each easureent can be ade ro readng o the sensor when one o the aperture eleents s copletely open and all others are copletely closed, whch corresponds to the bnary transttance T = (open), or 0 (closed). The easureents are the pxel values o the age when the eleents o the aperture assebly are opened one by one n certan scan order. Ths way o akng easureents corresponds to the tradtonal representaton o a dgtal age pxel by pxel. n the ollowng, we descrbe how copressve easureents can be ade n the proposed archtecture. A. opressve easureents Wth copressve sensng, t s possble to represent an age by usng ewer easureents than the nuber o pxels [3][4][5][6]. The archtecture o Fgure akes t sple to take copressve easureents. To ake copressve easureents, a sensng atrx s rst dened. ach row o the sensng atrx denes a pattern or the eleents o the aperture assebly, and the nuber o coluns n a sensng atrx s equal to the nuber o total eleents n the aperture assebly. n the context o copressve sensng, the two densonal array o aperture eleents n the aperture assebly s conceptually rearranged nto a one densonal array, whch can be done, or exaple, by orderng the eleents o the aperture assebly one by one n certan scan order. ach value n a row o the sensng atrx s used to dene the transttance o an eleent o the aperture assebly. A row o the sensng atrx thereore copletely denes a pattern or the aperture assebly, and t allows the sensor to ake one easureent or the gven pattern o the aperture assebly. The nuber o rows o the sensng atrx s the nuber o easureents, whch s usually uch saller than the nuber o aperture eleents n the aperture assebly (the nuber o pxels). Let the sensng atrx be a rando atrx whose entres are rando nubers between 0 and. To ake a easureent, the transttance, T, o each aperture eleent s controlled to equal the value o the correspondng entry n a row o the sensng atrx. The sensor ntegrates all rays transtted through the aperture assebly. The ntensty o the rays s odulated by the transttances beore they are ntegrated. Thereore, each easureent ro the sensor s the ntegraton o the ntensty o rays through the aperture assebly ultpled by the transttance o respectve aperture eleent. A easureent ro the sensor s hence a proecton o the age onto the row o the sensng atrx. Ths s llustrated n Fgure 2. By changng the pattern o the transttance o the aperture assebly, t s possble to ake copressve easureents correspondng to a gven sensng atrx whose entres have real values between 0 and. assebly cene ensor Fgure 2. Prograed aperture assebly or copressve easureents. The transttances o aperture eleents are controlled to atch the values o a row o the sensng atrx. A easureent s the ntegraton o all rays through the aperture assebly odulated by the transttance values.

3 ubtted January, LAT WOK The proposed archtecture s related to the sngle pxel caera o [8], whch captures copressve easureents but has lenses, and the lensless caera o [3], whch has no lenses but captures age pxels. At the rst glance, our proposed archtecture s sply a hybrd o the two; ndeed, as ar as the coponents and unctonalty are concerned, our archtecture sees as takng the lenses out o the caera o [8], or addng the proectng unctonalty nto the caera o [3]. However, there s a undaental derence between the archtecture o ths paper and the caeras o [8] and [3], whch s how the ages are ored. n both [8] and [3], a physcal echans s used to or an age o the scene on a plane, and then the age on the plane s pxelzed. n [8], a lens s eployed to or an age o the scene on the crorror array. The crorror array then perors the unctons o both pxelzaton and proecton. n [3], attenuatng aperture layers are used to create a pnhole whch ors an age o the scene on the sensor array. The sensor array then pxelzes the pnhole age. Thereore, both caeras o [8] and [3] create an analog age o the scene on a plane. n the caeras o [8] and [3], there are two processes that ay aect the qualty, sharpness and resoluton, o an age. The rst s the oraton o the analog age on the plane o pxelzaton, and the second s the pxelzaton o the analog age. The orer depends on the echans or orng the age. For exaple, n caera o [8], the sharpness ay depend on the ocal pont o the scene, so that an obect ay appear blurred because t s out o ocus. Furtherore, the artact o blurrng can occur even wth theoretcally perect lens, crorrors and sensor. n the archtecture o ths work, no planar age s explctly ored. One could argue that each easureent ro the sensor s a proecton o an age on the aperture assebly. However, ths vrtual age s not ored by any physcal echans, and thereore, t s an deal age that s ree o any artact such as blurrng due to deocus. Thereore, the qualty o age ro the archtecture o ths work s only aected by the resoluton o pxelzaton (the nuber o the aperture eleents n the aperture assebly) the aperture assebly and the sensor s theoretcally perect. V. MATHMATAL FOMULATON n ths secton, we orally dene what an age s n the proposed archtecture and how t s related to the easureents ro the sensor. n partcular, we wll descrbe how a pxelzed age can be reconstructed ro the easureents taken ro the sensor. A. Vrtual age Let the aperture assebly be a rectangular regon on a plane wth ( xy, ) coordnate syste. For each pont, ( xy, ), on the aperture assebly, there s a ray startng ro a pont on the scene, passng through the pont ( xy, ), and endng at the sensor, as shown n Fgure 3. Thereore, there s a unque ray assocated wth each pont ( xy, ) on the aperture assebly, and ts ntensty arrvng at the aperture assebly at te t s denoted by rxyt (, ; ). Then an age ( x, y ) o the scene s dened as the ntegraton o the ray n a te nterval Δ t : Δt ( x, y) = r( x, y; t) dt. 0 Note that although the denton o an age n s dened on the regon o the aperture assebly, there s not an actual age physcally ored n the archtecture o ths work. For ths reason, the age o s called a vrtual age. A vrtual age ( x, y ) can be consdered as an analog age because t s contnuously dened n the regon o the aperture assebly. Let the transttance o the aperture assebly be dene as T( x, y ). A easureent ade by the sensor s the ntegraton o the rays through the aperture assebly odulated by the transttance, and t s gven by zt = T( x, y) ( x, y) dxdy. ( xy, ) ensor Fgure 3. A ray s dened or each pont on the regon o aperture assebly. Although the vrtual age dscussed above s dened on the plane o the aperture assebly, t s not necessary to do so. The vrtual age ay be dened on any plane that s placed n between the sensor and the aperture assebly and parallel to the aperture assebly. B. Pxelzed age The vrtual age dened by can be pxelzed by the aperture assebly. Let the regon dened by one aperture eleent be denoted by as shown n Fgure 3. Then the pxel value o the age at the pxel (, ) s the ntegraton o the rays passng through the aperture eleent and t s gven by (, ) ( x, y) dxdy, (3) ( xyxydxdy, ) (, ). n above, the uncton s the characterstc uncton o the aperture eleent. The characterstc uncton o a regon s dened as =

4 ubtted January, 203 4, ( xy, ) ( xy, ) =. 0 ( xy, ) (4) Note that we use (, ) to denote a pxelzed age o a vrtual age ( x, y ) whch s analog. quaton (3) denes the pxelzed age (, ). n copressve sensng, t s oten atheatcally convenent to reorder a pxelzed age whch s a two densonal array nto a one densonal vector. Let q be a appng ro a 2 array to a vector dened by q:(, ) n, so that n = (, ). (5) Then the pxelzed age (, ) can be represented as a vector whose coponents are n. We wll sply use to denote the pxelzed age, ether as a two densonal array, or a one densonal vector, nterchangeably.. opressve easureents and reconstructon When the aperture assebly s prograed to pleent a copressve sensng atrx, the transttance T o each aperture eleent s controlled to equal the value o the correspondng entry n the sensng atrx. For the th easureent, the entres n row o the sensng atrx are used to progra the transttance o the aperture eleents. peccally, let the sensng atrx A be a rando atrx whose entres, a n, are rando nubers between 0 and. Let T ( x, y ) be the transttance o aperture eleent or the th easureent. Then, or the th easureent, the transttance o the aperture assebly s gven by T ( x, y) = T ( x, y),where, (6) T ( x, y) = a ( x, y)., q(, ) Thereore, accordng to, the easureents are gven by z = T ( x, y) ( x, y) dxdy, = = =,,, a a T q, (, ) q, (, ) ( x, y) ( x, y) dxdy, (, ). ( x, y) ( x, y) dxdy, quaton (7) s the alar or o copressve easureents the pxelzed age (, ) s reordered nto a vector by the appng q. ndeed, n the vector or, (7) s tantaount to z = a, q(, ) (, ) = ann, or, n (8) z = A. n above, z s the easureent vector, A s the sensng (7) atrx and s the vector representaton o the pxelzed age (, ). t s well known [3] that the pxelzed age can be reconstructed ro the easureents z by, or exaple, solvng the ollowng nzaton proble: n W, subect to A = z, (9) where W s soe sparsyng operator such as total varaton or raelets [4][5][6].. uary To suarze, the archtecture o ths work can be used to ake copressve easureents o the pxelzed age. For a gven sensng atrx A, the entres n each row o A are used to progra the transttance o the eleents o the aperture assebly. Wth each prograed pattern or the transttance, the sensor akes a easureent. The easureents ro all rows o A or a easureent vector z whch s gven by (8). Then the easureent vector z can be used to reconstruct the pxelzed age ro the nzaton proble (9). opressve sensng theory dctates that a good approxaton o the age can be coputed wth ar ewer easureents than the total nuber o aperture eleents (the nuber o pxels o ). Furtherore, the ore easureents are used n reconstructon, the better qualty o the reconstructed age s [3]. V. MULT-VW MAGNG Multple sensors ay be used n conuncton wth one aperture assebly as shown n Fgure 4. A vrtual age can be dened or each sensor, say, ( k ) ( x, y ) s the vrtual age assocated wth sensor, where the superscrpt k s used or ndexng the ultple sensors. These ages are ult-vew ages o a sae scene. P ( x+δ x, y+δy) ( xy, ) Fgure 4. Multple sensors are used wth one aperture assebly to ake ult-vew ages For a gven settng o transttance T( x, y ), each sensor takes a easureent, and thereore, or a gven sensng atrx, the sensors produce a set o easureent vectors, z, sultaneously. ach easureent vector z can be used to reconstruct a pxelzed age by solvng proble (9) ndependently wthout takng nto consderaton o other easureent vectors. However, although the ages

5 ubtted January, are derent, there s a hgh correlaton between the, especally when the sensors are close to one another and when the scene s ar away. The correlaton between the ages can be exploted to enhance the qualty o the reconstructed ages. Multple sensors wth one aperture assebly ay be used n the ollowng three ways: n a general settng, the easureent vectors ro ultple sensors represent ages o derent vews o a scene, creatng ult-vew ages. Thus, the archtecture allows a sple devce to capture ult-vew ages sultaneously. When the scene s planar, or sucently ar away, the easureent vectors ro the sensors ay be consdered to be ndependent easureents o a sae age (except or sall derence at the borders) and they ay be concatenated as a larger set o easureents to be used to reconstruct the age. Ths ncreases nuber o easureents that can be taken ro the sae age n a gven duraton o te. When the scene s planar, or sucently ar away, and when the sensors are properly postoned, the easureent vectors ro the sensors ay be consdered to be the easureents ade ro a hgher resoluton pxelzed age, and they ay be used to reconstruct an age o the hgher resoluton than the nuber o aperture eleents. The detaled dscussons wll be gven n the rest o ths secton. A. age decoposton For splcty, we consder two sensors, and, that are placed n a sae plane parallel to the plane o aperture assebly, as shown n Fgure 5. The sensors dene two vrtual ages ( x, y ) and ( x, y ). We want to explore coon coponent between the. cene assebly F cene assebly y+δy y (A) (B) Fgure 5. Varous dentons or two sensors on a plane parallel to the plane o aperture assebly. The llustraton s ade on a plane perpendcular to the plane o aperture assebly so that the aperture assebly s llustrated as a vertcal lne. The area o the aperture assebly can be dvded nto two dsont regons, and, accordng to. n the splest ter, conssts o the obects that can be also seen by ; that s, the obects appearng n are d coon n both ages, ( x, y ) and ( x, y ). conssts o the obects that can be only seen by ; that s, the obects appearng n can only be ound n ( x, y ). The denton o the two regons can be ade ore precse by usng the rays ro the two sensors. As shown n Fgure 4, any pont ( xy, ) denes a ray that starts ro the sensor and passes through ( xy, ). The ray ust ends at a pont P n the scene. Now a ray etted ro pont P can reach the sensor through the aperture assebly wthout obstructon by other obects o the scene (wth all aperture eleents open), then ( xy, ). Otherwse, no rays ro P can reach the sensor (wth all aperture eleents open), then ( xy, ). and can be slarly dened as above by reversng the role o and. and are llustrated n Fgure 5(A) n one densonal vew. ncdentally, the denton o and also denes a one-to-one appng between the. The ponts where the rays P and P ntersects the aperture assebly are apped nto each other. The appng s dened as 2 U :( x, y) ( x+δ x, y+δy), (0) 2 U :( x+δ x, y+δy) ( x, y) where the relatonshp between ( xy, ) and ( x+δ x, y+δ y) s shown n Fgure 4. ( k Now the vrtual ages ) ( x, y ) can be decoposed by usng the characterstc unctons o and as ollows ( k) ( k) ( k) ( x, y) = ( x, y) + ( x, y) ( k) ( k) ( x, y) = ( x, y) ( x, y), k =,2. () ( k) ( k) ( x, y) = ( x, y) ( x, y) Furtherore, ( x, y ) and ( x, y ) are related through the ollowng equatons: 2 ( x, y) = ( U ( x, y)), (2) 2 ( x, y) = ( U ( x, y)). ( k The decoposton, ) ( ( x, y ) and k ) ( x, y ), k =, 2, s llustrated n Fgure 6 gven below.

6 ubtted January, Fgure 6. ecoposton o the ages ro two sensors when the sensor dstance s an nteger ultple o the sze ( k) ( k) ( k o the aperture eleents. ) = +, k =,2. and are the coon age,, under a transor. The sgncance o the decoposton () s that the two vrtual ages are decoposed nto three coponents: one coponent s coon to both ages, and the other two coponents are unque to each ndvdual age. More speccally, we dene the coon coponent as ( x, y) = ( x, y), (3) then we have ( x, y) = ( x, y) + ( x, y), (4) 2 ( x, y) = ( U ( x, y)) + ( x, y). nce ( x, y ) s coon n both ages, ts reconstructon ay ake use o the easureents ro both sensors, and thereore, ts qualty ay be enhanced as copared to only one sensor s used. B. Jont reconstructon The coponents o the vrtual ages, ( x, y ), ( x, y ) and ( x, y ), can be pxelzed to get three vector coponents, and. eerrng to Fgure 6, the decoposton s slar to (4) and gven by = +, (5) = U +. n above, U s a atrx that perors sht and nterpolatng unctons to approxate the operaton o appng U 2 dened n (0). n other words, U s a vector that 2 approxates the pxelzed ( (, )) U x y, as gven by U ( q(, )) ( x, y) ( U ( x, y)) dxdy. (6) 2 The vector coponents, and ay be ontly reconstructed ro the two easureent vectors, z and z, ade ro the two sensors. Let A be the sensng atrx wth whch the easureents z and z are ade. Then the optzaton proble to solve s σ n W +, subect to 2 W 2 k =,. A + A = z AU + A = z (7) n (7), σ > 0 s a noralzaton constant to account or the areas o the our regons k and k, k =, 2. The value o the ont reconstructon (7) les n the act that there are only three unknown coponents n (7) wth two constrants (gven by z and z ), as copared to our unknown coponents wth two constrants the ages are reconstructed ndependently ro (9). Typcally, has uch ore nonzero entres than that o and, hence the nuber o unknowns s reduced by alost a hal. n general, proble (7) s qute dcult to solve because the regons k and k, k =, 2 are not known a pror, and they should be part o the soluton. However, the scene s planar and ts dstance s known, then t s possble to copute k and, k =, 2 beore (7) s solved. Thereore, n such cases when k and k are known, proble (7) ay be solved by well known establshed optzaton process such as those n [4][5][6].. Planar scene When the scene s on a plane parallel to and wth a known dstance ro the plane o aperture assebly, t s possble to 2 2 work out explct orulas or the appngs U and U o (0). As shown n Fgure 5(B), let us dene the dstance between two sensors to be d, the dstance between the plane o the sensors and the plane o aperture assebly to be and the dstance between the scene plane and the aperture 2 assebly to be F. Then the appng U s gven by 2 U ( x, y) = ( x+δ x, y+δy), 2 2 F Δ x +Δ y = d, (8) + F ( Δx, Δy). The last lne n (8) eans that the two vectors have the sae angle, or orentaton, n ther respectve planes. n general, when the scene s non-planar, equaton (8) stll holds, but F s no long a constant. t s rather a uncton o poston,.e., F = F( x, y), and t s also scene dependent. However, or the scene that s sucently ar away, F s F large copared to so that + F, and thereore, equaton (8) becoes

7 ubtted January, U x y = x+δ x y+δy (, ) (, ), 2 2 Δ x +Δy d, ( Δx, Δy). (9) Accordng to (9), when the scene s sucently ar away, the vrtual ages ro the two sensors are approxately the sae, except or a sht o dstance d. Thereore, the coon regon k covers the entre aperture assebly except or a border o wdth d. onsequently, copared to the coon age, the ages and have sall energy. Ths ples that proble (7) s anly a proble or the sngle age, whle usng two easureent vectors z and z, twce as any easureents as when each o the ages, and, s reconstructed ndependently as n (9). For ths reason, ultple sensors ay be consdered as takng ndependent easureents or a sae age the scene s sucently ar away. Ths can be used as a echans to ncrease the nuber o easureents taken durng a gven te duraton. the dstance between two sensors, d, s equal to an nteger ultple o the sze o the aperture eleents, as llustrated n Fgure 6, then atrx U n (7) s sply a sht atrx. n other words, the entres o U are zero except or the entres on an o-dagonal, whch are equal to.. Hgh resoluton For sucently ar away scenes, ultple sensors ay also be used as a echans to prove the resoluton o the coon age. the dstance d between two sensors s a non-nteger ultple o the sze o the aperture eleents, then and can be consdered as two down-sapled ages o a hgher resoluton age, see Fgure 7. The ont reconstructon can thereore be used to create a hgher resoluton age. peccally, equaton (4) can be rewrtten as ( x, y) = ( x, y) + ( x, y), (20) ( x, y) = ( x Δx, y Δ y) + ( x, y). the dstance d between two sensors s a non-nteger ultple o the sze o the aperture eleents, then there s no overlappng o grd ponts ( x Δx, y Δ y) wth the grd ponts ( xy, ). Thereore, equaton (20) shows that ages and coprse derent saplng o the sae age,.e., saples at ponts ( xy, ), whle saples at ponts ( x Δ x, y Δ y ). onsequently, the easureent vectors z and z can be used to reconstruct the age at both grd ponts ( xy, ) and ( x Δx, y Δ y). Ths results n an age that has a hgher resoluton than gven by the aperture eleents. Ths s llustrated n Fgure 7 below. Fgure 7. ecoposton o the ages ro two sensors when the sensor dstance s a non-nteger ultple o the sze o the aperture eleents. ( k = + ), k =,2. and are the coon age,, under a transor. V. PATAL ONATON n ths secton, we dscuss ssues arsng ro practcal pleentatons o the proposed archtecture. A. electon o aperture assebly The archtecture o ths work s lexble to allow a varety o pleentatons or the aperture assebly. For agng o vsble spectru, lqud crystal sheets [3] ay be used. Mcrorror arrays [8] ay be used or both vsble spectru agng and nrared agng. When a crorror array s used, the array s not placed n the drect path between the scene and the sensor, but rather t s placed at an angle so that the rays ro the scene s relected to the sensor when the crorrors are turned to a partcular angle, see [8] or an exaple o arrangeent. Further, when the crorror array s used, the transttance s bnary, takng the values o 0 and. The etallc asks o [0][] ay be used or Terahertz agng. For lleter wave agng, the ask o [2] can be used. n all these selectons, the aperture assebly s able to vary the transttance o ndvdual aperture eleent as nstructed by a prograable logc. B. ensor o nte sze n pleentatons, a sensor, such as a sngle photoconductve cell, has always a nte sze. We now consder the eect o a nte-sze sensor. For the purpose o coparson, we use ( x, y) to denote the vrtual age ro a sensor o nte sze, and use ( x, y ) to denote the vrtual age or an nntesal sensor whch s located at the center o ass o the nte-sze sensor. We wll establsh a relatonshp between ( x, y ) and ( x, y ). As beore, the age ro a sensor o nte sze s dened as the ntegraton o all rays reachng at the sensor that pass through a pont ( xy, ) on the aperture assebly, as llustrated n Fgure 8(A).

8 ubtted January, cene (A) ( xy, ) assebly ensor v ' cene F αv sensor assebly (B) Fgure 8. ensor o nte sze. (A) The age at pont (x,y) s dened as ntegraton o all rays wthn the cone passng through the pont (x,y) and arrvng at the sensor. (B) ervaton o the relatonshp between ( x, y ) and ( x, y ). The llustraton s ade on a plane perpendcular to the planes o aperture assebly and the sensor, whch both appear as a lne. Only upper hal o the nte-sze sensor (v) and the lower hal o the cone o the rays are shown. The botto o sensor (v) s where the nntesal sensor s located. n ths subsecton, we assue the scene s on a plane parallel to the aperture assebly and has a dstance o F or t. We also assue the nte-sze sensor has a two densonal shape, denoted by, on a plane parallel to the aperture assebly wth a dstance o ro t, see Fgure 8(B). We do not assue the senstvty o the nte-sze sensor s unor. Let ( uv, ) be a pont on, and ρ ( uv, ) be the senstvty o the sensor at pont ( uv, ). the sensor has unor senstvty, then ρ =, where s the area o. eerrng to Fgure 8(B), the upper hal o the nte-sze sensor s shown and labeled v, and the nntesal sensor s located at the botto o t. The lower hal o the cone o the rays ro the scene that can reach at the upper hal o the nte-sze sensor through pont ( xy, ) s labeled by v '. These rays or part o the age o the nntesal sensor, and t s labeled α v. The ntegraton o the ntensty o these rays s the value o ( x, y ). Fro the geoetry shown n Fgure 8(B), t can be easly vered that the actor α s gven by F α =. (2) + F At a pont ( uv, ) on, a ray on the regon labeled α v has ntensty ( x αu, y αv), but the senstvty o the nte-sze sensor at the pont s ρ ( uv, ), and thereore, the contrbuton o the ray to the ntegral s ρ( uv, ) ( x αuy, αv). Thus, the age o nte-sze sensor s gven by v ( x, y) = ρ( u, v) ( x αu, y αv) dudv, u v = ρ(, ) (, ), 2 x u y v dudv α α α (22) = ρ ( uv, ) ( x uy, vdudv ), α ( ρ ) = α ( x, y), where u v ρ ( uv, ) = α ρ(, ). 2 α α α (23) quaton (22) shows that the vrtual age or the nte-sze sensor s the convoluton o the age o nntesal sensor wth the pont spread uncton ρ α,.e., = ρ α. n other words, the vrtual age ( x, y ) o the nte-sze sensor s ( x, y ) o the nntesal sensor, blurred by ρ α, whose support s saller than the sze o the nte-sze sensor because the support o ρ s and α <. We now consder the pxelzaton o ( x, y ), whch s slar to (3) and gven below. (, ) = ( x, y) ( x, y) dxdy, = φ( x, y, u, v) dudvdxdy, (24) φ( xyuv,,, ) = ( xy, ) ρ ( uv, ) ( x uy, v). Ater a change o varables, equaton (24) becoes (, ) κ( x, y) ( x, y) dxdy. where ( x, y) = ( x + u, y + v) ( u, v) dudv, κ α = (25) = ρ, (26) α ρα( uv, ) = ρα( u, v). We now copare (, ) and (, ) when the ntesze sensor has the unor senstvty and when the scene s sucently ar away ro the aperture assebly. the scene s ar, then F s large copared to, so we can assue α =. Also, ρα = ρ =, the senstvty s unor. Next, we rewrte the equatons or (, ) and (, ), ro equatons (3) and (25), respectvely as (, ) = ( x, y) (, ), x y dxdy (27) (, ) = ( )(, ) (, ). x y x y dxdy t s now clear ro (27) that both (, ) and (, ) are pxelzaton o ( x, y ), the vrtual age ro the nntesal sensor, but the derence s that the orer s ρ α

9 ubtted January, ntegrated wth, and the later s ntegrated wth. n other words, whle (, ) s obtaned ro a pxelzaton o dsont regons dened by the aperture eleents,, (, ) s the result o pxelzaton by overlapped regons, resultng n blurrng. The overlapped regons are deterned by, the shape o the sensor. The blurrng s neglgble the area o s uch saller than that o, that s, the sensor s uch saller than the aperture eleent. The equaton or (, ) n (27) s useul n reconstructon o an age when usng easureents ro a nte-sze sensor. n the reconstructon, we should try to reconstruct, not (, ), but soe dscretzed verson o ( x, y ) by usng the constrants consstent to how the easureents are actually obtaned. For exaple, let z be the easureent vector obtaned wth a nte-sze sensor by usng the sensng atrx A. Then we reconstruct an age by solvng the ollowng proble o ndng the age ( x, y ) o the nntesal sensor: nze a cost uncton o ( x, y), subect to z = a, q ( )(, ) (, ), x y x y dxdy (28), as opposed to solvng the conventonal proble o ndng the age (, ) o the nte-sze sensor : nze a cost uncton o (, ), subect to (29) z = a, q (, )., The soluton to the conventonal proble (29) would result n a blurrng due to the nte sze o the sensor. However, by solvng the nzaton proble (28) n reconstructon, the eect o the nte-sze sensor s accounted or, and the blurrng s reoved. t s worthwhle to pont that the blurrng gven n (25) due to the nte sze o the sensor s derent ro the blurrng due to obects beng out o ocus o a lens. The blurrng n (25) does not exst an nntesal sensor can be bult, but t s stll possble or an obect to be out o ocus even a theoretcally perect lens s bult. The blur n (25) s caused by the nablty to ake an nntesal sensor, whch s analogous to the act that an age created by a realstc lens can never be perectly n ocus because t s possble to bult a theoretcally perect lens.. uper-resoluton When the sensor has an nntesal sze, the resoluton o the reconstructed age s the sae as the resoluton o the aperture assebly, as shown n (3) and (9). However, or a nte-sze sensor, an age ay be reconstructed wth a derent resoluton than that o the aperture assebly. n partcular, wth a nte-sze sensor, t s theoretcally possble to reconstruct an age o a resoluton uch hgher than the resoluton o the aperture assebly. quaton (28) provdes a ethod to reconstruct a vrtual age ( x, y ) whch can be consdered to have nnte resoluton, because t s a uncton o contnuous varables ( xy, ). However, t s not expected that the constrants n (28) are able to deterne a unque ( x, y ) or contnuous varables ( xy, ) n general, because there s only a nte nuber o constrants. On the other hand, there s soe pror knowledge o ( x, y ), super-resoluton reconstructon s possble. For exaple, t s known that the age s created by a pont lghtng source, and the sensor has the sae sze and shape as the aperture eleents, t s theoretcally possble to nd the exact locaton o the pont source by solvng proble (28). quaton (28) s also lexble n allowng pxelzaton o derent granularty n derent regons o an age, or exaple, ult-resoluton. The pxelzaton can be done by ) dvdng the age nto sall regons whch are called pxels (the regons ay have derent shapes or szes), and 2) assung that ( x, y ) s a constant n each o the pxel regons (the constant s the pxel value at the pxel). Then the ntegrals n (28) ay be calculated to yeld a set o constrants on the pxel values.. racton o aperture n pleentatons, when the aperture eleents are sall, the eect o dracton ust be consdered. For ths purpose, we consder an age ro an nntesal sensor, o a onochroatc wave wth the wave nuber k. Let d ( x, y ) be the vrtual age wth dracton eect when the aperture assebly has the transttance T( x, y ). As beore, let ( x, y ) be the vrtual age wthout the eect o dracton. Then d ( x, y ) can be wrtten n ters o ( x, y ) by Fraunhoer dracton equaton [4] as k( lu+ hv) d( x, y) = e T ( u, v) ( u, v) dudv. (30) n (30), lh, are the drecton cosnes o the pont ( x, y) wth respect to the orgn whch s located at the nntesal sensor. quaton (30) shows that the eect o dracton causes a blur n the age, uch as the eect o nte-sze sensor does n (22), but o course, wth a derent pont spread uncton. Furtherore, snce T( x, y ) s nvolved n (30), the blur caused by the dracton actually depends on the pattern o the aperture assebly. Now, let the transttance o aperture eleent be T and urther assue that or any two aperture eleents, and st, the ollowng ntegral s a constant over st :

10 ubtted January, k( lu+ hv) Tst e dxdy = const u v st b q (, ), qst (,),(,),. (3) Then we can derve a sple relatonshp between the pxelzed ages wth and wthout dracton. ndeed, the pxelzed age wth dracton s gven by (, ) = ( x, y) dxdy, d = = d e k( lu+ hv) k( lu+ hv) Tst e u v dudvdxdy st st, = b ( u, v) dudv, st, st, q (, ), qst (,) = b ( s, t). q (, ), qst (,) st T( u, v) ( u, v) dudvdxdy, (, ),(32) n the vector or, the pxelzed age can be wrtten as d = B, (33) where B s a square atrx wth entres b q (, ), qst (,) dened n (3). quaton (33) shows that the eect o the dracton s sply a blurrng wth the kernel atrx B whose entres are gven n (3). a easureent z s ade by usng a row ( ) vector a o a sensng atrx A, then n the reconstructon, the easureent needs to be consdered as ade by the oded row vector a ( ) B ( ) n order to account or the dracton eect. Note that the superscrpt s used n ( ) ( ) a B because atrx B n (33) actually depends on the pattern o the aperture assebly when easureent z s ade. V. POTOTYP n ths secton, we descrbe the prototype and present exaples ro the prototype devce. The agng devce conssts o a transparent onochroe lqud crystal dsplay (L) screen and a photovoltac sensor enclosed n a lght tght box, shown n Fgure 9. The L screen unctons as the aperture assebly whle the photovoltac sensor easures the lght ntensty. The photovoltac sensor s a trcolor sensor, whch outputs the ntensty o red, green and blue lghts. A coputer s used to generate the patterns or aperture eleents on L screen accordng to each row o the easureent atrx. The lght easureents are read ro the sensor and recorded or urther processng. The coputer s also responsble or synchronzaton between the creaton o patterns on the L and the tng o easureent capture, see Fgure 0. Fgure 9. Prototype devce. The top photo shows the laboratory set up to acqure the age o the books. The botto let s the L screen used as the aperture assebly, and the botto rght s a photo o the GB sensor board. Two sensors, ndcated by the red crcle, are ounted on the board. A. age acquston The L panel s congured to dsplay a axu resoluton o 302 x 27 = black or whte squares. nce the L s transparent and onochroe, a black square eans the eleent s opaque, and a whte square eans the eleent s transparent. Thereore, each square represents an aperture eleent wth transttance o a 0 (black) or (whte). For capturng copressve easureents, we use a sensng atrx whch s constructed ro rows o a Hadaard atrx o order N= ach row o the Hadaard atrx s peruted accordng to a predeterned rando perutaton. The rst eleents o a row are then sply apped to the aperture eleents o the L n a scan order ro the top to botto and then ro let to rght. An n the Hadaard atrx turns an aperture eleent transparent and a - turns t opaque. The easureents values or red, green and blue are taken by a sensor at the back o the enclosure box and recorded by the control coputer, as llustrated n Fgure 0. n experents reported n ths paper, only one sensor s used to take the easureents. esults or ult-vew agng wth two sensors wll be reported n a uture paper. A total nuber o 65534, whch corresponds to the total nuber o pxels o the age, derent easureents can be ade wth the prototype. n our experents, we only ake a ractonal o the total possble easureents. We express the nuber o easureents taken and used n reconstructon as a percentage o the total nuber o pxels. For exaple, 50% o

11 ubtted January, 203 easureents eans easureents are taken and used n reconstructon, whch s hal o the total nuber o pxels, larly, 25% eans 6384 easureents are taken and used n reconstructon. L creen nclosure Photovoltac ensor Pattern Measureents Fgure. econstructed ages o "occer". Top: 2.5% easureents. Botto: 50% easureents. Fgure 0. cheatc llustraton o the lensless copressve sensng age prototype. B. age econstructon We used varous stll le subects n the laboratory to deonstrate the concept o the agng devce. We rely on the standard reconstructon ethod coonly known as L nzaton o total varaton by solvng q. (9). The nuber o easureents needed or reconstructon o an age depends on any actors such as the coplexty (eatures) o the age and qualty o the reconstructed age. Fgure shows reconstructed ages o a soccer ball wth 2.5% and 50% easureents. Fgure 2 shows reconstructed ages wth relatvely ore eatures. The reconstructon o the ages used 25% and 50% o total easureents, respectvely. Fgure 3 shows reconstructed ages o a cat sleepng n a basket wth 25% and 50% o total easureents. We note that the color ages are reconstructed by usng drectly the easureents o the three color coponents ro the sensor. No calbratons were ade to balance the color coponents. Fgure 2. econstructed ages o "Books". Top 25% easureents. Botto 50% easureents

12 ubtted January, Fgure 3. econstructed ages o "leepng cat. Top: 25% easureents. Botto: 50% easureents. V. ONLUON An archtecture or lensless copressve sensng agng s proposed. The archtecture allows lexble pleentatons to buld sple, relable agng devces wth reduced sze, cost and coplexty. Furtherore, the ages ro the archtecture do not suer ro such artacts as blurrng due to deocus o the lens. evces based on ths archtecture ay be used n survellance applcatons or detectng anoales or extractng eatures such as speed o ovng obects. scusson and analyss were presented on how to handle ult-vew ages ecently, how to deal wth the eects o nte-szed sensor and dracton, and how to reconstruct ages wth hgher resoluton. A prototype devce was bult usng low cost, coercally avalable coponents to deonstrate that the proposed archtecture s ndeed easble and practcal. FN []. andès, J. oberg, and T. Tao, gnal recovery ro ncoplete and naccurate easureents, o. Pure Appl. Math. vol. 59, no. 8, 2005, pp [2]. onoho, opressed sensng, Trans. on noraton Theory, vol. 52 no. 4, 2006, pp [3] J. oberg, agng va copressve saplng, gnal Processng Magazne, vol 25, no 2, pp. 4-20, [4] hengbo L, Hong Jang and Paul Wlord and Yn Zhang and Mke cheutzow, A new copressve vdeo sensng raework or oble broadcast, to appear n the Transactons on Broadcastng, March, 203 [5] Hong Jang, hengbo L, azel Ha-ohen, Paul Wlord and Yn Zhang, calable Vdeo odng usng opressve ensng, Bell Labs Techncal Journal, Vol. 6, No. 4., pp , 202. [6] Hong Jang, We eng and Zuowe hen, urvellance vdeo processng usng copressve sensng, nverse Probles and agng, vol 6, no 2, pp 20-24, 202. [7] V. K. Goyal, A. K. Fletcher and. angan, opressve aplng and Lossy opresson, gnal Processng Magazne, vol 25, no 2, pp , [8] harpal Takhar, Jason N. Laska, Mchael B. Wakn, Marco F. uarte, ror Baron, hrra arvotha, Kevn F. Kelly, chard G. Baranuk, A New opressve agng aera Archtecture usng Optcal- oan opresson, Proc. &T/P oputatonal agng V, January [9] M. F. uarte, M. A. avenport,. Takhar, J. N. Laska, T. un, K. F. Kelly, and. G. Baranuk, ngle-pxel agng va copressve saplng, gnal Process. Mag., vol. 25, no. 2, pp. 83-9, [0] Wa La han, Krt haran, harpal Takhar, Kevn F. Kelly, chard G. Baranuk, and anel M. Mttlean, A sngle-pxel terahertz agng syste based on copressed sensng, Appled Physcs Letters, vol. 93, no. 2, pp , ept [] A. Hedar and. aeedka, A 2 caera desgn wth a sngle-pxel detector, n MMW-THz 2009., pp. 2, [2].. Babacan, M. Luess, L. pnoulas, A.K. Katsaggelos, N. Gopalsa, T. ler,. Ahern,. Lao, and A. apts, opressve passve lleter-wave agng, n age Processng (P), 20 8th nternatonal onerence on, sept., pp , 20 [3] Assa Zoet and hree K. Nayar, Lensless agng wth a ontrollable, onerence on oputer Vson and Pattern ecognton (VP), Jun, [4] Lpson A., Lpson G, Lpson H, Optcal Physcs, 4th ed., abrdge Unversty Press, 20, p 23. AKNOWLGMNT We thank Hock Ng and Bob Farah o Alcatel-Lucent or helpng wth the prototype enclosure. We also thank azel Ha-ohen, ongqng Zhao and Larry O'Goran o Alcatel-Lucent or ther nterests and rutul dscussons leadng to proveent o ths paper.