An Enhanced Super-Resolution System with Improved Image Registration, Automatic Image Selection, and Image Enhancement

Transcription

1 An Enhanced Super-Resoluton System wth Improved Image Regstraton, Automatc Image Selecton, and Image Enhancement Yu-Chuan Kuo ( ), Chen-Yu Chen ( ), and Chou-Shann Fuh ( ) Department of Computer Scence and Informaton Engneerng, Natonal Tawan Unversty, Tape, Tawan Abstract In ths paper we propose a system that reconstructs hgh-resoluton mages wth mproved super-resoluton algorthms, whch s based on Iran & Peleg teratve method and employs our ntal nterpolaton, mage regstraton, automatc mage selecton, and mage enhancement methods. When the target of reconstructon s a movng object wth respect to a statonary camera, hgh-resoluton mages can stll be reconstructed, whereas prevous systems only wor well when we move the camera and the dsplacement of the whole scene s the same. Keywords mage mprovement, mage enhancement, super resoluton, mage regstraton, nterpolaton. Introducton Due to the envronmental constrant and the resoluton of the sensor, we can only get low qualty mages at tmes. In order to mprove the mage qualty and resoluton by human eyes, more than a sngle nput mage s requred. Wth mage sequences, a blurrng scene, a dm fgure, or an unclear object of poor qualty can be reconstructed to a super-resoluton output mage and can then be easly observed and recognzed. Prevous research regardng super resoluton s manly dvded nto teratve methods [], frequency doman methods [], and Bayesan statstcal methods [3]. In Secton we ntroduce an mproved super-resoluton method wth partcular choces of ntal guess and a better mage regstraton method. Then we propose a novel dea of mage selecton n Secton 3 so as to mae the system better and faster. In Secton 4 we apply a post-processng of mage enhancement to mae the output mage clearer. Experments and conclusons are descrbed n Sectons 5 and 6 respectvely.. Improved Iran & Peleg Iteratve Method. Bref Descrpton of Tradtonal Itran & Peleg Method Iran [] developed the teratve algorthm usng mage regstraton to reconstruct the super-resoluton mage n 99. The method manly conssts of three phases, ntal guess, magng process, and reconstructon process. At frst, a low-resoluton mage s taen as reference on whch we may reconstruct a guessed super-resoluton mage by nterpolaton technques. That s, drectly put extra pxels n between the orgnal reference mage and then nfer the pxel value wth respect to ts neghbor ntenstes. Wth the ntal guess, magng process s then appled accordng to the followng formula, g ( n) ( n) = ( T ( f )* h) s

2 where g s the th observed mage frame; f s the super-resoluton scene; h s the blurrng operator; s the transformaton operator that transforms other low-resoluton mages to the reference frame; and s s the down-samplng operator. The whole process represents the magng process that taes pctures wth a smulated camera. Then, we compare the result of the magng process wth the real low-resoluton mage we have n hand. The dfferences are used to mprove the reference mage n the current teraton. f ( n+ ) = f ( n) + K K T = ((( g g ( n) ) s)* p) where K s the total number of low-resoluton mages that are used; p s the de-blurrng operator; (n) f s the reconstructon result after nth teratons. Repeatedly apply the above process untl the reference frame converges to a satsfactory result after several teratons.. Improved Intal Guess When the magnfcaton factor and reconstructon mage szes get larger, the computaton tme becomes longer. Typcal runtme s on the order of hours and are machne-dependent. The ntal guess as descrbed above wll largely affect the performance of our result, and f a better ntal guess s appled, great amount of computaton tme wll be saved. Because ntal guess s done merely once at the begnnng of the process, the complexty of the whole Iran & Peleg method does not depend on the complexty of the ntal guess, whch s based on nterpolaton technques. Here we ntroduce only 3 rd order (cubc) nterpolaton that taes 4 neghborng pxels nto consderaton and then evaluate performances of st~5th order ntal guess by Pea Sgnal-to-Nose Rato (PSNR). T Thrd order, or cubc nterpolaton consders 4 unnown varables. Suppose the nterpolaton functon s 3 y = f ( x = ax + bx + cx + d, and nown 3 ) neghborng pxels nclude (, A), ( 0, B ), (, C ), and (, D ) ; then A B 0 = C D a b c d a b = c d = A B C D A 0 B C 0 D Smlarly, other orders of nterpolaton also solve for coeffcents of f n (x). Applyng f n (x) n -dmensonal nterpolaton algorthm, we can get all pxels n an ntegral row up-sampled frst by nterpolaton n x drecton, and then get all pxels by nterpolaton n y drecton. We observe that dfferent order of nterpolaton results n dfferent ntal-guess mages and dfferent convergence rates of mage qualty as the number of teraton grows. By choosng the most approprate order of nterpolaton, we wll get the best results of Iran & Peleg method, snce ntal guess has a great nfluence on the performance of mage regstraton and on the necessary number of teratons to acheve the pea mage result. In most stuatons, 3 rd order nterpolaton rans the best choce of ntal guess f both complexty and reconstructed mage qualty are concerned. We evaluate the performance of dfferent orders of nterpolaton by PSNR between the orgnal mage and reconstructed mages. The expermental results are

3 shown n Fgure. between pont ( x, and pont ( u, on the hgh-resoluton mage. LAD ( x, y; u, = LR ( x, = arg mn LAD ( x, y; u, ( u, I ( x + m, y + n) I ( u + m, v + n) ( m, n) w LT ( x, = LR ( x, ( x, * Magnfcat onfactor o Fgure. Performance wth st to 5 th order of nterpolaton appled for ntal guess. Usng ntal guess wth dfferent orders of nterpolaton has dfferent PSNR convergence rates. Blue, green, and cyan curves represent st, nd, and 4 th orders respectvely. Performance wth 3 rd and 5 th orders of nterpolaton acheves smlar results as the red curve shows..3 Improved Image Regstraton Image regstraton s crtcal n the performance of our algorthm snce each teraton refnes each pxel on the hgh-resoluton mage usng the nformaton of the correspondng pxel on the low-resoluton mages. We ntroduce two methods to acheve mage regstraton. The local matchng technque loos for a set of correspondng pars and the global matchng technque loos for the correspondng poston of the whole low-resoluton mage on the smulated hgh-resoluton mage..3. Local Matchng Technque For each nterestng pont ( x, on low-resoluton mage, the mappng functon LR ( x, loos for ts correspondng pont ( u, on the smulated hgh-resoluton mage. Functon LR ( x, mnmzes absolute dfference LAD ( x, y; u, wthn a local wndow w. Translaton LT ( x, s the translaton In order to get more accurate mage regstraton and then reconstruct the hgh-resoluton mage of a movng object, we choose nterestng ponts of correspondng pars under the followng constrants. a. The gradent at an nterestng pont should be larger than a threshold. For each nterestng pont on a low-resoluton mage, we loo for the correspondng pont on the smulated hgh-resoluton mage where hgher local-complexty around the pont s requred. b. The translaton between each correspondng par should not be zero. Our goal s to reconstruct a movng object on a statonary bacground so we consder the zero-translated ponts as bacground. These ponts should not be chosen as nterestng ponts. Under the constrants, we can fnd a set of correspondng pars. We use the mode translaton of the set to represent the translaton of mage. Set the set of nterestng ponts of mage. T = M ({ LT ( x, ( x, P }) where M (A) s the mode of lst A..3. Global Matchng Technque P s Global matchng functon GR () searches the correspondng poston ( u, of low-resoluton mage. Functon GR () mnmzes the absolute dfference wthn the whole mage GAD ( u,. GAD ( u, = GR( ) = arg mn GAD ( u, ( u, I ( x, I ( u + x, v + ( x, o 3

4 Then, the translaton.4 User-defned Boundary T of the mage s GR (). To mprove the speed and the accuracy of mage regstraton, we only loo for correspondng pars of the movng object. Thus we choose nterestng ponts nsde a user-defned boundary. For global matchng functon, the user-defned boundary should be bounded n the object,.e. each pxel on the area should belong to the object as well, so that the nterestng ponts wll not le on the bacground and ms-regstraton caused by occluson can be elmnated. For local matchng functon, the user-defned area could be larger than the object. The pont belongng to bacground can be gnored snce the relatve translaton s zero as descrbed n Secton.3.. For objects that we cannot use a rectangular area to bnd, we suggest applyng local matchng functon to calculate the translatons. 3. Automatc Selecton from Image Sequences Wth a large number of mage sequences, t not only costs much tme to reconstruct a hgh-resoluton mage but also reduces the qualty f some mages are ms-regstered. We propose a novel way to select a mnmal number of useful mages. To reconstruct a hgh-resoluton mage of magnfcaton factor of n, we only need one mage to get suffcent nformaton for each mod-translaton (modulus of translaton). Mod-translaton for mage s defned as T mod MagnfcatonFator. Our algorthm can select the best mage for each mod-translaton. Thus, we explot the most useful and mnmal number of mages to reconstruct hgh-resoluton mages. 3. Automatc Selecton wth Global Matchng Technque We propose two crtera to select the better mage from two mages wth the same mod-translaton. For two mages, j havng the same mod-translaton and select mage f GAD ( u, v ) < GAD j ( u j, v j ) ( u, v ) = T, we ( u, v ) = T, j j j Most regstraton has a mn GAD ( u, of nonzero because the ntenstes of smulated hgh-resoluton are produced by nterpolaton. If the ntal guess s reasonably correct, the real translaton of mage havng smaller GAD u, v ) wll be closer to an ( ntegral grd so the error would be mnmzed after the real translaton s rounded to T. 3. Automatc Selecton wth Local Matchng Technque Ms-regstered mages would reduce the qualty of hgh-resoluton mages. Therefore, we dscard these ms-regstered mages and select the most useful and mnmal number of low-resoluton mages by comparng the remanng mages wth the same mod-translaton. Image that should not be dscarded has the followng crtera. a. The number of nterestng ponts, # P, under the constrants descrbed n Secton.3. should be larger than a threshold. b. The rato of the mode of the translaton, #{( x, ( x, P, LT ( x, = T }/# P be larger than a threshold., should c. The rato of the second mode of the translaton, #{( x, LT ( x, = M ( LT ( p, q) T ), ( x, ( p, q) P }/# P should be smaller than a threshold. For two mages and j havng the same mod-translaton, and we select mage f a. σ < σ j ( u, v ) = T and j j j ( u, v ) = T, The varance of LT ( x, ( x, I } s defned { 4

5 as x y σ = σ + σ. Symbols σ x and σ y are the varances of the translaton values along x and y axes respectvely. When we calculate varances, the noses should not be taen nto consderaton. A nose s labeled f the number of the translaton s one. If the varance s smaller, the regstraton s more satsfactory for each nterestng pont and s closer to the real answer. Table ndcates the performance of our system wth and wthout automatc selecton. Table. The performance of our system wth and wthout automatc selecton. We use fve sets of 6x6 low-resoluton mages and the magnfcaton factor s 3. (Measured wth Intel Pentum III and 8MB RAM) Local Matchng Technque Global Matchng Technque Wth Selecton Wthout Selecton Wth Selecton Wthout Selecton Run Tme PSNR (seconds) (db) Image Enhancement Post-processng In order to mae the super-resoluton mages much clearer and more recognzable, we add a post-processng that apples some basc mage enhancement technques [5]. Edge sharpenng method mproves the resolvablty of the mage. In our system we apply Laplacan mas 8 for convoluton. After hgh-pass flterng, the mage becomes sharp-edged and the reconstructed mage s more easly recognzed (as shown n Fgure.). Besdes, local hstogram equalzaton s used to mae the mage more adaptve to human eyes and medan flter s appled so as to remove mpulse noses. Both of those mage enhancement technques are helpful for human recognton n our system. 5. Results 5. Reconstructng Hgh-resoluton Images wth Movng Smulated Camera We smulate a camera by tang an mage as orgnal scene and down-samplng the orgnal scene nto several pctures. Usng the smulated camera, we tae pctures begnnng at dfferent ponts,.e. the smulated camera moves when tang pctures. Then, our algorthm taes these pctures as nputs and reconstructs a hgh-resoluton mage teratvely and magnfcaton factor of length s 4. The am s to reconstruct hgh-resoluton mages of the whole scene so the user-defned area n regstraton should be the same wth the area of low-resoluton mages. The performance s good after suffcent teratons, as shown n Fgures 3 and Reconstructng Hgh-resoluton Objects from Image Sequences of Movng Object In secton 4., we smulate a camera tang pctures when movng on a statc scene. In ths secton, we tae 7 pctures of a movng object wth a real camera. On each pcture, only the object moves slghtly and the bacground stays mmoble. Our am s to reconstruct the hgh-resoluton mage of that object and magnfcaton factor of length s. To mprove the speed and accuracy of regstraton, we specfy an area wthn the object. As the number of teraton ncreases, on the hgh-resoluton mage, the object becomes clearer whle the bacground becomes blurry and words are more dscernble on the edge sharpened hgh-resoluton mage as Fgure 5 shows. 5

6 (a) (b) dramatcally faster. Besdes, because we dscard poor-qualty mages, fnal mage qualty wll be better. Fnally we add a post-processng of mage enhancement that contans edge crspenng and local hstogram equalzaton to mae the target objects n mage sequences more recognzable. Reference [] M. Iran and S. Peleg, Improvng Resoluton by Image Regstraton, CVGIP: Graphcal Models and Image Proc., Vol. 53, pp. 3-39, 99. (c) (d) Fgure. (a) One of low-resoluton mages. (b) Intal guess. (c) Reconstructed mage after 00 teratons. (d) Enhanced fnal output mage. [] R. Y. Tsa and T. S. Huang, Multframe Image Restoraton and Regstraton, n Advances n Computer Vson and Image Processng, Vol. (T. S. Huang, ed.), pp , Greenwch, CT: Ja Press, Conclusons We have developed an mage reconstructon system that consttutes mproved super resoluton teratve method, ntellgent selecton from mage sequences, and fnal mage enhancement process. Frst, we suggest a complex ntal guess usng 3 rd order nterpolaton n order to reduce the number of teratons requred and mprove the performance of mage regstraton. Second we propose a better mage regstraton method, ncludng usng gradent constrant, user-defned boundary, and translaton thresholdng, whch tends to capture only the nformaton of the movng object nstead of the statonary bacground and allows the reconstructon of mage sequences of a movng object n a scene. Then we ntroduce a novel dea of ntellgent mage selecton. By flterng out redundant and useless mages, the system runs [3] P. Cheeseman, B. Kanefsy, R. Kruft, J. Stutz, and R. Hanson, Super-Resolved Surface Reconstructon from Multple Images, NASA Techncal Report FIA-94-, 994. [4] A. M. Tealp, M. K. Ozan, and M. I. Sezan, Hgh-Resoluton Image Reconstructon for Lower-Resoluton Image Sequences and Space-Varyng Image Restoraton, IEEE Internatonal Conference on Acoustcs, Speech, and Sgnal Processng, Vol. III, pp. 69-7, San Francsco, CA, 99. [5] R. C. Gonzalez and R. E. Woods, Dgtal Image Processng, Addson-Wesley, Readng, MA, 99. [6] W. K. Pratt, Dgtal Image Processng, nd Ed., Wley, New Yor, 00. 6

7 (a) (b) (c) (d) Fgure 3. Results of our proposed method wth a fxed scene and a smulated movng camera. (a) One of low-resoluton mages. (b) Intal guess. (c) Reconstructed mage after 00 teratons. (d) Enhanced fnal output mage. Fgure 4. PSNR of teratvely output mages. As the number of teratons grows, the performance, evaluated by PSNR, converges. 7

8 (a) (b) (c) (d) Fgure 5. Results of our proposed method wth a movng scene and a fxed real camera. (a) One of low-resoluton mages. (b) Intal guess. (c) Reconstructed mage after 00 teratons. (d) Enhanced fnal output mage.. [ f (, j) F(, j)] MSE = N RMSE = MSE PSNR = 0log0(. Defne 55 RMSE A x = { a a A, a x} ) where n n A R R and a, x R R. For example, x, y ) ( x, y ) ( x, y ) ( x, y )} ( x, y ) = {( x, y ) ( x, )} {( y3 8