A Sngle-Image Super-Resoluton Method for Texture Interpolaton Yaron Kalt and Moshe Porat Abstract In recent years, a number of super-resoluton technques have been proposed. Most of these technques construct a hgh resoluton mage by ether combnng several lo resoluton mages at sub-pxel msalgnments or by learnng correspondences beteen lo and hgh resoluton mage pars. In ths paper e present a stochastc super-resoluton method for color textures from a sngle mage. The proposed algorthm takes advantage of the repettve nature of textures and the exstence of several smlar patches thn the texture, as ell as the color-ntensty correlaton that often exst n natural mages. In the frst step of the algorthm the ntensty component s nterpolated. For each pxel, the mssng value s chosen accordng to a probablty dstrbuton constructed from a measure of smlarty to other patches n the texture as ell as from local features and patch color smlarty. In the second stage, the color components are nterpolated n a smlar manner, usng patches of the color channels as ell as the already nterpolated ntensty values. Our concluson s that the proposed approach outperforms presently avalable methods. Index Terms Image processng, super resoluton, texture nterpolaton, color zoomng. I. INTRODUCTION Interpolaton s one of the fundamental tasks n mage processng. Its applcatons range from medcal and astronomcal mage processng to magnfcaton of detals n mages acqured from survellance cameras. The popular nterpolaton methods such as the b-lnear, b-cubc and nearest neghbor methods are convoluton based, space nvarant methods. They are dely used due to ther lo computatonal complexty and smple mplementaton. Snce these algorthms functon n essence as lo-pass flters, they tend to ntroduce blurrng and blockng artfacts, hch are vsble especally n areas contanng hgh frequency content such as edges as ell as n textures. Several classes of more sophstcated algorthms ere proposed n recent years [1]-[3]. Most of them adapt to the mage and attempt to reconstruct the hgh frequency content n a manner that preserves the edges and ntroduces as fe artfacts as possble. An mportant class of algorthms knon as super-resoluton (SR) uses multple lo resoluton mages n order to reconstruct a hgh resoluton mage [4]. The classcal SR technques combne several lo resoluton mages at sub-pxel msalgnments. Dfferent knds of SR algorthms Manuscrpt receved June 15, 2012; revsed July 12, 2012. Ths research as supported n part by Technon's fund #7110134 and by the Ollendorff Mnerva Center. Mnerva s funded through the BMBF. The authors are th the Department of Electrcal Engneerng, Technon, Hafa 32000, Israel (e-mal: ykalt@tx.technon.ac.l, mp@ee.technon.ac.l). have been proposed, follong approaches such as projecton onto convex sets [5], stochastc and determnstc regularzaton [6], and teratve back-projecton [7], among others. A dfferent class of SR technques s example-based SR, n hch correspondences beteen the lo and hgh resoluton mages are learned from a database of lo and hgh resoluton mage pars [8]. Another approach that has recently been proposed makes use of a sngle mage and s related to the Non-Local-Means (M) algorthm [9-10]. The majorty of mage nterpolaton and super-resoluton algorthms focus on the nterpolaton of gray-scale mages. In order to apply these algorthms to color mages, an extenson must be made n order to nclude all three color components. Separate nterpolaton of the channels has the potental of ntroducng color artfacts snce RGB ratos mght not be preserved. A common approach s to apply a sophstcated nterpolaton technque to the ntensty component, and a smpler scheme (such as b-cubc nterpolaton) to the color components. Ths s justfed by the fact that the human eye s more senstve to change of ntensty than t s to color. Whle ths approach acheves reasonable performance, some blurrng stll occurs at color edges due to the lnear nterpolaton. In addton, correlaton beteen the color and the ntensty s not used. Several algorthms ere desgned specfcally for color mages [11]. In ths ork e present a SR algorthm that up-scales color textures usng only nformaton from the nterpolated mage. Textures are an mportant part of natural mages, and ther percepton s beleved to have a sgnfcant role n the process of recognton of the human vsual system. Due to ther mportance, several approaches to texture analyss have been nvestgated over the years, such as statstcal based technques [12], technques based on the Markov random feld (MRF) model [13] and frequency doman technques [14]. The MRF model has also been used for texture nterpolaton [15]. A dfferent approach that has been used for texture flterng [16] and nterpolaton [10] s M. In ths ork e present a ne approach for texture nterpolaton that s based on a sngle mage. The proposed method has to steps. Frst, a stochastc nterpolaton of the ntensty s performed based on local features, non-local patch smlarty, as ell as nformaton from the color components. In the second stage, the color components are nterpolated n a stochastc manner, usng the already nterpolated ntensty values. The stochastc nterpolaton preserves the statstcal relatons beteen neghborng pxels and s shon to outperform exstng methods. The rest of the paper s organzed as follos: the proposed algorthm s presented n Secton II. In Secton III nterpolaton results are presented. Secton IV summarzes the proposed method and ts performance. DOI: 10.7763/IJFCC.2012.V2.120 54
II. THE PROPOSED ALGORITHM Unlke [18], e are consderng only same scale patch smlarty. For each nterpolated pxel, e compare the surroundng pxels, hch e ll refer to as the hgh resoluton patch, to patches n the lo resoluton mage. In order to perform the comparson correctly, e take a prelmnary scale adjustment step. In Fg. 1, an expanson of an mage by a factor of to s shon (gray pxels are the avalable lo resoluton pxels). In ths case, three types of pxels need to be nterpolated 'h', 'v' and 'd' type pxels. Fg. 1. Expanded mage: gray- orgnal (lo resoluton) pxels, hte mssng pxels Consderng for example the 'h' type pxels, a correspondng lo resoluton patch used for the comparson s shon n Fg. 2. Fg. 2. Lo resoluton patch for 'h' type pxels nterpolaton Comparng the patch proportons n Fgs. 1 and 2, a dfference n the vertcal drecton can be seen. In order to use lo and hgh resoluton patches of equal proportons, e estmate the ntermedate values by averagng every to ros as shon n Fg. 3. secton. A. Interpolaton of the Intensty Component Local - Non-Local Based Probablty Dstrbuton: Due to the repettve nature of textures, several lo resoluton patches smlar to the hgh resoluton patch typcally exst. The ntensty values of the central pxels of these patches could be consdered to be potental canddates for the ntensty value of the nterpolated pxel. In order for the nterpolaton process to preserve the statstcal relatons beteen neghborng pxels, e form a probablty dstrbuton for these values, accordng to hch the nterpolated value s chosen. The probablty dstrbuton takes nto account local as ell as non-local features of the texture. The non-local part ranks the canddate gray level values accordng to patch smlarty regardless of the patch locaton, hle the local part consders characterstcs of the surroundng pxels such as smoothness. The probablty dstrbuton thus combnes global statstcs ('typcal' ntensty values for the partcular pattern), th local features, hch are also ndcatve of the gray-level probablty dstrbuton. The probablty assgned to gray level k s- P L ( k ) L ( k ) ( ) ( ) here and are the local and non-local eghts that are assgned to each gray level k. Non-Local Weghts- In [15], Buades et al. used a non-local flter (M) for the purpose of mage denosng. We use a smlar expresson as the eght that s assgned to each lo resoluton patch reflectng ts smlarty to the hgh resoluton patch. When comparng a hgh resoluton patch to lo resoluton patches, e consder to cases - here and are the closest lo resoluton pxels (Fg. 1). In ths case, the hgh resoluton pxels,, are compared to pxels from the averaged ros (Fg. 3). The comparson s to lo resoluton mage patches (Fg. 2). The dstncton s made snce the ro averagng causes smoothng of edges. The second condton s ndcatve of an edge, n hch case the smoothed values are not used. For a gray scale mage I(, j) of sze, the eght that s assgned to the lo resoluton patch centered at (, j) s - L (1),,,, Fg. 3. Expanson of the lo resoluton mage. dark gray ros lo resoluton pxels, ntermdate ros ro averages A smlar procedure s used for the nterpolaton of 'v' type pxels. When comparng a hgh resoluton patch to the lo resoluton patches, ether the orgnal or the modfed lo resoluton mage s used, as descrbed n the follong 2 1,2 1 (2) here, s the group of pxels surroundng the nterpolated pxel at (l, m) n the hgh resoluton mage,, s the correspondng group of pxels of the lo resoluton patch centered at (, j), and s a lnear 55
quantzaton functon that s appled so that mperceptble dfferences have no effect on the assgned eghts. The eght assgned to gray level k s the follong sum: here,,. (, j ) I k (, j ) Local Weghts- The local eght L k s calculated usng a larger neghborhood, such as the one shon n Fg. 4 for 'h' type pxels. (3) The more smlar the color components are, the greater the eght that s gven to the ntensty value of the central pxel of the lo resoluton patch. As before, summaton over all of the patches gves the eght of gray scale level k - k. The probablty dstrbuton accordng to hch the ntensty value s chosen s p p ' L ' p L ( k ) ( ) color color ( k ) ( ) Ths completes the nterpolaton of the ntensty for 'h' and 'v' type pxels. The nterpolaton of 'd' type pxels proceeds n a smlar manner, except that the already nterpolated 'h' and 'v' pxel values (h,h,v,v n Fg. 5) are used as ell. (6) Fg. 4 Pxels used for the local eght calculaton of 'h' type pxels The local eght s a Gaussan hose mean and varance are determned by the neghborng lo resoluton pxel values (Fg. 4) - 1 2 2 Agan, e consder to cases: edge and non-edge. (4) In ths case - 2,, In ths case the Gaussan mean as calculated usng the nverse gradent approach, and the varance as proportonal to the edge slope. Usng color-ntensty correlaton- In many natural mages the ntensty and the color components are correlated. Ths correlaton could be used n order to further mprove the nterpolaton of the ntensty component. In addton, a large class of textures (e.g., carpets, fabrcs, etc.) contans only a small number of dstngushable colors, especally hen several threads of dfferent colors are used. For that reason, the color components, n the CIE-LA*B* space are frst jontly quantzed (usng the k-means or a smlar algorthm). A threshold functon s frst appled to the probablty dstrbuton obtaned from the ntensty values n (1), settng small probabltes to zero. 0 (5) Next, eghts are assgned to the gray level values based on the color components. The quantzed color components of the hgh resoluton patch are compared to those of the lo resoluton patches (n ths case usng Eucldan dstances). Fg. 5. Patch values used for nterpolatng 'd' type pxels B. Interpolaton of the Color Components In the second stage of the algorthm, the color components are nterpolated. A smlar methodology as before s used, consderng non-local smlarty and local features. The color nterpolaton, hoever, reles on the prevously nterpolated ntensty values. The color nterpolaton s based on a quantzed verson of the color components,.e. quantzed patches are compared. The color components, A*B*, are assocated th one of the values n the set-,,,,,,,,,. In addton, the values that are assgned to the mssng color components of the hgh resoluton pxels are also elements of C. The color that s assgned to each hgh resoluton pxel s chosen, as before, accordng to a probablty dstrbuton, constructed from to eghts, local and non-local, that are assgned to each value of C. The probablty dstrbuton s calculated from the eghts accordng to (7). p( c ) = r k = 0 color ( c ) ( c ) color L ( cr ), ( c ) color r color L 1 r k As before, e frst nterpolate 'h' and 'v' type pxels. Non-Local Weghts- Let us denote the group of pxels surroundng the pxel hose color s nterpolated by n,,n. We denote ther quantzed color components by c,,c. The hgh resoluton patch of quantzed color components s compared to correspondng patches of the quantzed colors n the lo resoluton mage. Ths tme, hoever, only exact matches are consdered. We form a set of the lo resoluton patches hose quantzed color components are,,. We denote ths group by,,, here J s the number of such patches. Due to the typcally small number of dstngushable colors n textures, and ther repettve nature, n most cases ths group ll be non-empty. In case the group s empty, the nterpolated color s one of the to quantzed colors of the nearest lo resoluton pxels. In ths case the color s chosen usng the color-ntensty correlaton. (7) 56
Let us no denote by,, the quantzed color components of the central pxels of the patches n P, and by,, the ntensty values of these pxels. In addton, for each quantzed color let us denote the group of ndces correspondng to patches n group P for hch the quantzed color of the central pxel s c by. The non-local eght s then defned by The PSNR values of the nterpolated "Wood" texture are presented n the follong table: PSNR(dB) Bcubc nterpolaton 14.1 Cubc splnes nterpolaton 15.8 The proposed algorthm 15.8 exp (8) here s the nterpolated ntensty value of the currently nterpolated pxel. It can be seen from (8), that n order for a color n to have a non-zero probablty, t must be the quantzed color of a central pxel n one of the patches. Local Weghts- The purpose of local eghts s to ensure that the nterpolaton process does not ntroduce any colors that are dfferent than the colors of the surroundng pxels. Let us denote the (quantzed) color components of the hgh resoluton patch by c,,c. The eght assgned to each possble color s As can be seen, the PSNR value that as obtaned usng the proposed nterpolaton method s the same as that obtaned usng cubc splnes. In Fgs. 6 and 7 the nterpolated "Wood" textures are presented. It can clearly be seen that the nterpolaton usng the proposed method produces a sgnfcantly less blurry texture n comparson to the other nterpolaton methods. The proposed nterpolaton algorthm as also appled to other textures from the Brodatz database, yeldng smlar results,.e. PSNR values smlar to those obtaned usng cubc splnes and superor to bcubc nterpolaton, hle beng less blurry than both. 1 c 1,,c 6 0 c 1,,c 6 (9) The nterpolaton of the 'd' type pxels proceeds n the exact same manner, except that the color components that have already been nterpolated are used as ell. III. EXPERIMENTAL RESULTS A. Performance Evaluaton of the Local-Non-local approach In order to evaluate the nterpolaton of the ntensty, a 100x100 porton of the (gray scale) ood texture from the Brodatz database [17] as nterpolated. The full (hgh resoluton) texture as frst donscaled by a factor of 2, and then up-sampled. The nterpolaton usng the proposed algorthm reled on nformaton from a larger porton of the mage (256x256 pxels). Fg. 7. Magnfed patches as marked n Fg. 6. Fg. 6. Texture 'ood'. top left hgh resoluton mage, top rght - bcubc nterpolaton, bottom left cubc splne nterpolaton, bottom rght the proposed method IV. SUMMARY AND CONCLUSIONS In ths paper e have ntroduced a sngle-mage super-resoluton method that s able to successfully nterpolate a very broad class of textures statonary as ell as non-statonary. Unlke other nterpolaton technques, the proposed method ncorporates the nformaton n the color channels n the nterpolaton of the ntensty values. As ndcated by the results, usng nformaton from local characterstcs as ell as from smlar patches acheves hgh 57
resoluton mages that are vsually superor to presently avalable nterpolaton technques. REFERENCES [1] X. L and M. T. Orchard, Ne edge-drected nterpolaton, IEEE Trans. Image Process, vol. 10, no. 10, pp. 1571 1527, Oct. 2001. [2] A. Belahmd and F. Guchard, "A partal dfferental equaton approach to mage zoom," In Proc. of the Int. Conf. on Image Processng, pp. 649-652, 2004. [3] S. Esedoglu and J. Shen, "Dgtal npantng based on the Mumford-Shah-Euler mage model," European Journal of Appled Mathematcs, Vol. 13, pp. 353-370, 2002. [4] S. Park, M. Park, and M. Kang, Super-resoluton mage reconstructon: a techncal overve, IEEE Sgnal Process. Mag, vol. 20, no. 3, pp. 21 36, May 2003. [5] H. Stark and P. Oskou, Hgh resoluton mage recovery from mage-plane arrays, usng convex projectons, J. Opt. Soc. Am. A, vol. 6, pp. 1715-1726, 1989. [6] R. C. Harde, K. J. Barnard, and E. E. Armstrong, Jont MAP regstraton and hgh-resoluton mage estmaton usng a sequence of undersampled mages, IEEE Trans. Image Processng, vol. 6, pp. 1621-1633, Dec. 1997. [7] M. Iran and S. Peleg, Improvng resoluton by mage regstraton, CVGIP: Graphcal Models and Image Proc, vol. 53, pp. 231-239, May 1991. [8] K. Km and Y. Kon. "Example-based learnng for sngle mage SR and JPEG artfact removal," Techncal Report 173. Max Plank Insttute, August 2008. [9] D. Glasner, S. Bagon, and M. Iran, Super-resoluton from a sngle mage, n Internatonal Conference on Computer Vson, 2009. [10] T. Wttman. "Mathematcal Technques for Image Interpolaton, Report Submtted for Completon of Mathematcs Department Oral Exam, Department of Mathematcs, Unversty of Mnnesota, USA, 2005. [11] Y. W. Ta, W. S. Tong, and C. K. Tang, "Perceptually-nspred and edge-drected color mage super-resoluton," In CVPR, vol. 2, pp. 1948 1955, 2006. [12] J. Weszka, C. Deya, and A. Rosenfeld, A Comparatve Study of Texture Measures for Terran Classfcaton, IEEE Trans. System, Man and Cybernetcs, vol. 6, pp. 269-285, 1976. [13] H. Yn and N. Allnson, Unsupervsed Segmentaton of textured Images Usng a Hererchcal Neural Structure, Electroncs Letters, vol. 30, no. 22, pp. 1842-1843, 1994. [14] M. Porat and Y. Zeev, "Localzed Texture Processng n Vson: Analyss and Synthess n the Gaboran Space," IEEE Trans. Bomedcal Eng, vol. 36, pp. 115-129, 1989. [15] S. Nemrovsky and M. Porat, On Texture and Image Interpolaton usng Markov Models, Sgnal Processng and Image Communcaton, vol. 24, pp. 139-157, 2008. [16] A. Buades, B. Coll, and J.M. Morel, "On mage denosng methods, "CMLA Preprnt, CMLA pp. 2004-2015, 2004. [17] P. Brodatz. Dover, Ne York, 1966. 58