Attention Based Auto Image Cropping

Transcription

1 Attenton Based Auto Image Croppng Fred Stentford UCL Adastral Park Campus, Martlesham Heath, Ipswch, Suffolk, IP5 3RE, UK Abstract. Many mages contan salent regons that are surrounded by too much unnterestng background materal and are not as enlghtenng as a sensbly cropped verson. The choce of the best pcture wndow both at capture tme and durng subsequent processng s normally subjectve and a wholly manual task. Ths paper proposes a method of automatcally croppng vsual materal based upon a new measure of vsual attenton that reflects the nformatveness of the mage. 1 Introducton Hugh quanttes of dgtal mages and vdeo are beng created whch are not nformatve smply because the nterestng parts are not mmedately apparent to a human observer. Ths may be because there s an overwhelmng amount of dstractng background materal or because the mages are beng vewed on a small dsplay. In ether case there s a need to dentfy those regons n an mage whch are salent and to reshape the mage for maxmum mpact and mproved composton wthout tedous manual nvolvement. Ths problem has been addressed by Chen et al [1] for vewng large mages on small dsplays. Itt s ([2] 1998) vsual attenton measure was used n combnaton wth models for face and text detecton to measure the salence of Regons Of Interest (ROI) for ncluson n a cropped verson of the mage. Ths work was extended by Lu et al [3] to nclude varatons n tme wth applcaton to determnng optmal browsng paths across the ROIs. However, dfferent weghts were necessary for fusng the dfferent models nto a sngle attenton value for dfferent categores of mages. Zhang [4] used a croppng technque that made use of face detecton and an attenton model. The approach appeared to work well but employed many emprcally determned parameters. Dgtal camera mages were frst segmented by Ma et al [5] and then ROIs were assessed accordng to ther entropy, the sze and nearness to the centre of the mage. Agan emprcal parameters needed to be set accordng to the characterstcs of the camera and the habts of the user. Boutell [6] has shown that creatng more tranng data smply by blndly croppng 10% off sdes of mages sgnfcantly boosted scores on test sets. Ths provded ndrect evdence that cropped mages can be more nformatve. Suh et al [7] summed salency values wthn canddate cropped regons to determne the best selectons. Ths approach also used Itt s measure of salency and strateges for face detecton, but requred a threshold to lmt the sze of the cropped

2 2 Fred Stentford regon that vared from mage to mage. The approach taken n ths paper makes use of a measure of vsual attenton that was successfully appled to the control of the focusng of mage formng devces [8]. It was found that the subject was n focus when the average salency score was maxmsed over the whole mage. Ths measure could be consdered as an ndcaton of mage nformatveness and s appled here to the related problem of selectng an optmal croppng wndow. 2 Vsual Attenton Itt [2] makes use of colour, ntensty and orentaton flters followed by centresurround computatons to determne salency. Local contrast s employed by Ma et al [9] and Hu et al [10] together wth fuzzy growng to dentfy ROIs.. In ths paper salent regons are detected through a process that compares small regons wth others wthn the mage. A regon that does not match most other regons n the mage s very lkely to be anomalous and wll stand out as foreground materal. For example, the edges of large objects and the whole of small objects normally attract hgh attenton scores manly because of colour adjacences or textures that only occur rarely n the mage. Self smlar backgrounds that dsplay a translatonal symmetry are assgned low attenton scores. Regon matchng requres a few pxels (a fork) wthn that regon to match n a translated poston n another regon. If the dfference n colour of one pxel par exceeds a certan threshold a msmatch s counted and the attenton score s ncremented. X y Fg. 1. Fork at x msmatchng at y. Let a pxel x n a pattern correspond to colour components a where x = ( x 1, x 2 ) and a = ( a, a a ) (1) Let F ( x) = a Consder a neghbourhood N of x where x N ff x x ε 1 2, { } 3

3 Attenton Based Auto Image Croppng 3 Select a fork S A of m random pxels where S A = { x x x x 1, 2, 3,..., m}. (2) Construct a fork of m pxels S B where S B = { y 1, y2, y3,..., ym} (3) and x y = δ S B s a translated verson of S A. The fork F j S msmatches the fork A S f B ( x ) F ( y ) τ for any j (4) j j j, In general the colour component threshold τ s not a constant and wll be dependent j upon the colour space of the measurements under comparson.e. τ = f F x, F y (5) j j ( ( ) ( )) A locaton x wll be worthy of attenton f a sequence of t forks matches only a few other neghbourhoods n the space. In Fg. 1 a fork wth m = 3 pxels s selected n the neghbourhood of a pxel x and s shown msmatchng at y. The neghbourhood of the second pxel y matches the frst f the colour ntenstes of all the correspondng pxels have values wthn τ of each other. The attenton score V ( x) for each pxel x s ncremented each tme a msmatch occurs n the fork comparsons wth a sequence of y. Ths measure of vsual attenton has been appled to mage compresson Stentford [11] and focus control Shlston [12]. It was found that the maxmum M of the sum I of the attenton scores of all the pxels n an mage I at dfferent focal dstances tended to occur when the prncpal subject s n focus. M may be consdered to be a I measure of mage nformatveness. M I = V x (6) x I Ths paper makes use of the same measure to assess the nformatveness of cropped mages wth a vew to selectng the best verson. In ths case the optmum croppng wndow W s taken to be the one havng the hghest average pxel attenton score. I W I W I x W ( ) = arg max V ( x )/ W (7) Parameter values used n the experments below were t = 100, m = 3, ε = 1, τ = Results 3.1 Image Croppng V and then searchng for the wndow whch possesses the hghest average pxel attenton score accordng to The approach begns by generatng a salency map ( x)

4 4 Fred Stentford Fg. 2. Orgnal mage and salency map equaton (7). Fg. 2 shows an mage together wth ts salency map n whch the brghter colours ndcate hgher salency. The wndow W may be any shape or sze, but for many mage formng devces a fxed aspect rato may be requred and the method s llustrated here wth wndows restrcted to a zoom factor of 2. Fg. 3 shows a seres of mages together wth ther cropped versons. 3.2 Image Zoomng The results descrbed above open the possblty of an optmal zoom wndow for any vew usng the nformatveness measure n (7). However, ths would mean that all possble zoomed wndows for each scene would need to be evaluated whch s nether practcal nor probably necessary. The approach was appled to the problem of selectng the most nformatve sub-wndow wth the same aspect rato wthn a statc mage for a range of sub-wndow szes. Ths s equvalent to extractng the locaton of the most nformatve crop at each sze. Fg. 4 shows two orgnal mages I and a successon of reducng wndows W at ncreasng zoom factors selected accordng to equaton (7). 3.3 Zoom Factor The nformatveness of an mage of a dstant object should ncrease as the object comes closer and then decrease agan when perhaps a featureless surface of the object occupes a large proporton of the mage. A seres of 320x240 photos n Fg. 5 were taken of a red rectangle at varous focal dstances. The average pxel attenton V x / W s plotted n Fg. 6. A peak s obtaned when the rectangle x W score ( ) enlarged but wth background stll present and edges and corners clearly separated from the mage boundary.

5 Fg. 3. Orgnal mages and cropped versons Attenton Based Auto Image Croppng 5

6 6 Fred Stentford Fg. 4. Optmal croppng wndows at varous szes.. a) dog, b) boat z = 1.0 z = 1.20 z = 1.63 z = 2.44 z = 2.56 z = 2.64 z = 3.32 z = 3.68 Fg. 5. Images of a rectangle taken at dfferent zoom factors z.

7 Attenton Based Auto Image Croppng average attenton score zoom factor Fg. 6. Average pxel attenton scores for mages n Fg. 5. Closer vews of the object may expose more attentve detal and a sharp maxmum would not be reached. Fg. 7 shows a wde angle vew of a car and three successve zoomed x3 mages (300x225) controlled by the attenton measure n (7). In ths example the new wndows are derved from the prevous mage whch possesses greater resoluton n the regon of nterest than the orgnal. The fnal mage s a cropped verson of the prevous mage. Zoomng only on the orgnal mage does not produce the same results owng to the presence of small salent regons not relevant to the prncpal subject. Fg. 7. Zoomng x3 based upon attenton measure.

8 8 Fred Stentford 4 Dscusson The croppng results n Fg. 2 are satsfactory for a zoom factor of 2. However, an acceptable cropped mage may not exst for all wndow szes and therefore some judgment s necessary for selecton. The successon of wndows n Fg. 4 demonstrate a convergence towards the regons of hghest attenton value. The wndow at zoom factor 3 n Fg. 4a s not well cropped because the sngle wndow cannot encompass both the dog s head and the salent area of snow at the bottom of the mage. Ths may be handled n future by branchng and allowng more than one wndow to be formed at the same zoom factor. Ths would be a natural approach to mages that contaned more than one salent object. Although ths mechansm may be appled to pcture framng n cameras, the results n Fg. 4 are derved from a sngle mage and no addtonal defnton s obtaned n the smaller wndows. In the case of mages n Fg. 5 the resoluton ncreases but no new detal s exposed and the salency of the seres of mages passes a peak. On the other hand the mages n Fg. 7 show that the addtonal resoluton obtaned by zoomng reveals new salent materal that nfluences the wndow convergence path. Computaton requrements can be qute hgh especally f every pxel regon n the mage s scored. However, the attenton score only requres smple operatons of matchng and countng and may be carred out n parallel across the whole mage. The calculatons for controllng the focusng of a vdeo camera have been mplemented on a Texas Instruments DM642 DSP platform that operates at 25 fps. As the nformatveness measure n (7) s dentcal to that used to select the best focal dstance [12] t s concevable that the operatons of focusng and croppng could take place at the same tme. The focusng would naturally concentrate on the most salent regon wthn each wndow. Objects that were blurred, ether due to moton, camera shake, or the wrong focal dstance, would tend to dscourage focusng and croppng n those regons, but subjects that were at the correct focal dstance and not movng relatve to the camera would be targeted by the mechansm. 5 Conclusons Ths paper has descrbed the applcaton of an attenton measure to the automatc croppng of mages. The method may be appled to croppng sngle mages or gudng a seres of zoom operatons. More subjectve evaluaton s necessary on a greater range of mages and strateges developed for dentfyng multple croppng wndows. These experments could also compare human selected crops wth those obtaned by the model. Ths research has been conducted wthn the framework of the European Commsson funded Network of Excellence Multmeda Understandng through Semantcs, Computaton and Learnng (MUSCLE) [11].

9 Attenton Based Auto Image Croppng 9 References 1. Chen, L., Xe, X., Fan, X., Ma, W., Zhang, H., Zhou, H. : A Vsual Attenton Model for Adaptng Images on Small Dsplays. Multmeda Systems. 9 (2003) Itt, L., Koch, C., Nebur, E.: A Model of Salency-Based Vsual Attenton for Rapd Scene Analyss. IEEE Trans Pattern Analyss Mach Intell. 20 (1998) Lu, H., Xe, X., Ma, W., Zhang, H. : Automatc Browsng of Large Pctures on Moble Devces. 11 th ACM Int. Conf. on Multmeda. November (2003) 4. Zhang, M., Zhang, L., Sun, Y., Feng, L., Ma, W.: Auto Croppng for Dgtal Photographs. IEEE Conf. on Multmeda and Expo. July (2005) 5. Ma, M., Guo, J.K.: Automatc Image Croppng for Moble Devces wth Bult-n Camera. Proc. Consumer Communcaton & Networkng Conf. January (2004) Boutell, M., Luo, J., Gray, R.T.: Sunset Scene Classfcaton usng Smulated Image Recomposton. Proc. IEEE Conf. on Multmeda and Expo. 1 (2003) Suh, B., Lng, H., Bederson B.B., Jacobs, D.W.: Automatc Thumbnal Croppng and ts Effectveness. Proc. 16 th ACM Symposum on User Interface Software and Technology. (2003) Shlston, R., Stentford, F.W.M.: An Attenton-Based Focus Control System, Internatonal Conf. on Image Processng. October (2006) 9. Ma, Y., Zhang, H.: Contrast-Based Image Attenton Analyss by Usng Fuzzy Growng. Proc.11 th ACM Conf. on Multmeda. (2003) Hu, Y., Xe, X., Chen, Z., Ma, W. : Attenton Model Based Progressve Image Transmsson. Proc. IEEE Int. Conf. On Multmeda and Expo. (2004) Multmeda Understandng through Semantcs, Computaton and Learnng, Network of Excellence. EC 6 th Framework Programme. FP