Consigio Nazionae dee Ricerche Istituto di Cacoo e Reti ad Ate Prestazioni Restoration of bue scratches in digita image sequences Lucia Maddaena, Afredo Petrosino RT-ICAR-NA-05-2 December 2005 Consigio Nazionae dee Ricerche, Istituto di Cacoo e Reti ad Ate Prestazioni (ICAR) Sede di Napoi, Via P. Casteino, 803 Napoi, URL: www.na.icar.cnr.it
Consigio Nazionae dee Ricerche Istituto di Cacoo e Reti ad Ate Prestazioni Restoration of bue scratches in digita image sequences Lucia Maddaena 2, Afredo Petrosino 3 Rapporto Tecnico N.: RT-ICAR-NA-05-2 Data: dicembre 2005 Sottomesso per pubbicazione 2 Istituto di Cacoo e Reti ad Ate Prestazioni, ICAR-CNR, Sede di Napoi, Via P. Casteino, 803 Napoi 3 Università di Napoi Parthenope, Dipartimento di Scienze Appicate, Via A. De Gasperi 5, 8033 Napoi I rapporti tecnici de ICAR-CNR sono pubbicati da Istituto di Cacoo e Reti ad Ate Prestazioni de Consigio Nazionae dee Ricerche. Tai rapporti, approntati sotto escusiva responsabiità scientifica degi autori, descrivono attività di ricerca de personae e dei coaboratori de ICAR, in acuni casi in un formato preiminare prima dea pubbicazione definitiva in atra sede. 2
Restoration of bue scratches in digita image sequences LUCIA MADDALENA, ALFREDO PETROSINO 2 Nationa Research Counci, ICAR Via P. Casteino, 803 Napes, ITALY, Te.: +39-08 639522; fax: +39-08 63953 ucia.maddaena@na.icar.cnr.it 2 University Parthenope of Napes, Department of Appied Science Via A. De Gasperi 5, 8033 Napes, ITALY, Te.: +39-08 547660; fax: +39-08 5522293 afredo.petrosino@uniparthenope.it Abstract In this paper we consider the probem of detecting and removing bue scratches from digita image sequences. In particuar, we propose a detection method and a remova method that strongy rey on the specific features of such scratches. Evauation of the proposed methods, in terms of both accuracy and performance timings, and numerica experiments on rea images are reported. Keywords: Coour digita fim restoration; Bue scratch; Scratch detection; Scratch remova. Introduction Digita Fim Restoration is an evoving area of Image Processing aimed at studying methodoogies and techniques that aow to digitay restore damaged movies, in order to preserve their historica, artistic and cutura vaue and to faciitate their diffusion through modern communication media. Severa types of defects can be found in a damaged movie, such as dust and dirt, brightness and positiona instabiity, coour fading, scratches. We are specificay concerned with persistent scratches, intended as vertica ines appearing at the same ocation in subsequent frames of the image sequence. White or back scratches in od movies are mainy due to the abrasion of the fim caused by spurious partices present in the camera, during the sequence acquisition phase, or in the projector, during the fim projection. Instead, bue scratches, which are the subject of our interest, affect many modern coour movies and are due to spurious partices present in the transport mechanism of the equipment used for the deveopment of the fim. Most of the methods reported in iterature that afford this kind of probem are articuated in a detection phase and a remova phase. Submitted for pubication 3
The detection phase consists in searching, among a the vertica ines of the images, those that are not natura ines of the scene, which are characterized as defects. Severa methods have been adopted in the case of white or back scratches, such as those based on ow/high pass fiters [7, 8], morphoogica fiters [9, 2-5, 27], adaptive binarization [6], discrete waveet decomposition [5], statistics and MAP techniques [24, 25, 30], or oca gradient measures in the image [, 2] or in the image cross section [6], eventuay couped with techniques such as Hough transform [5, 8, 2] or Kaman fiter [2-5], and possiby foowed by Bayesian refinement strategies [8]. The resut of the detection phase over a sequence frame is a binary image, the scratch mask, of the same size, where white pixes are reated to scratch pixes in the corresponding sequence frame. The remova phase consists in reconstructing corrupted information in the defect area individuated by the scratch mask. Depending on the amount of the defect, information incuded in the scratch area can be either sighty or strongy affected by the defect; thus, the scratch remova probem can be approached either as a partiay corrupted data probem or as a missing data probem, respectivey. Foowing the partiay corrupted data approach, information incuded in the artifact area is taken into account for the remova. In the case of back or white scratches, some authors adopted such approach and obtained remova through morphoogica fiters [27], interpoation or approximation [4, 5, 24], eventuay foowed by the reconstruction of highfrequency components via Fourier series [4] or via MAP techniques [5]. On the other hand, in the missing data approach pixes in the artifact area are considered missing even if they are ony sighty atered. This approach has been adopted for back or white scratches by many authors, who obtained remova through interpoation or approximation [5, 22, 26], the adoption of autoregressive modes [7, 8], morphoogica fiters [9], or mean vector fiters [6], eventuay with the addition of east squares-based grain estimation [22]. Moreover, this approach is the one generay adopted for image inpainting, that is the set of techniques for making undetectabe modifications to images [28]; such techniques are generay used to fi-in missing data or to substitute information contained in sma image regions [32]. Inpainting has been pursued in iterature aso under different names, such as image interpoation (e.g. [29]) and fi-in (e.g. [2, 9]); the probem has been afforded aso as disoccusion, since missing data can be considered as occusions hiding the image region to be reconstructed (e.g. [3, 23]). Finay, inpainting is aso reated to texture synthesis, where the probem consists in generating, given a sampe texture, an unimited amount of image data which wi be perceived by humans as having the same texture []; specificay, inpainting can be considered as a constrained texture synthesis probem [4, 9, 20]. Even though the probem of detection and remova of white or back scratches in digita image sequences has been considered by so many authors and severa commercia software systems incude modues for their restoration (such as the DIAMANT Suite distributed by HS-ART Digita Service GmbH or the Reviva distributed by da Vinci Systems, Inc.), the specific case of bue scratches has not been specificay addressed. As aready mentioned, they generay affect modern coour movies and, therefore, before aunching a new motion picture, the fim must be digitay restored by companies speciaized in digita effects and postprocessing. The need for efficient and automatic toos abe to digitay remove bue scratches has been the 4
primary input for the reported research. Specificay, in this paper we propose a method for the detection and remova of bue scratches in digita images that takes into account the specific features of such kind of scratches. The contents of this paper are as foows. In Section 2 the features of bue scratches are anaysed, in order to device suitabe digita restoration techniques. Sections 3 and 4 outine the methods that we propose for bue scratch detection and remova, respectivey, giving detais of the reated agorithms and impementations. In Section 5 we describe quaitative and quantitative resuts achieved by the proposed approach on rea images. Concusions are reported in Section 6. 2. Bue scratch characterisation Bue scratches in a digita image sequence appear as bue strips ocated aong a thin area covering from top to bottom of each sequence frame. Exampes of bue scratches are given in Figs., 3 and 4, which are detais of 24 bits RGB coour images, originay of size 2880x2048, beonging to the movie Animai che attraversano a strada (2000). Contrary to white or back scratches appearing in dated movies, the direction of bue scratches does not deviate too much from the vertica direction, and their position aong the horizonta direction does not change too much (no more than few pixes) from one frame to the next. Therefore usuay bue scratches are not obique and have fixed position in consecutive frames of the image sequence. This is due to the fact that bue scratches are not caused by improper storage conditions or improper handing of the fim, as is usuay the case for ancient movies. They are rather caused by spurious partices present in the transport mechanism of the deveopment equipment; in the case of modern equipment, the transport mechanism stricty contros the sippage of the fim, which cannot move too much from its rectiinear trajectory. Due to this feature, restoration of bue scratches cannot rey on tempora discontinuity of the image intensity function aong the sequence; therefore, in the foowing we concentrate on purey spatia scratch detection and remova in each image. (a) (b) (c) (d) (e) Fig. : Exampe of a bue scratch: (a) coour image; (b) scratch detai; (c) red, (d) green, and (d) bue band of scratch detai. 5
Inside the bue scratch area, origina information has been substituted by more or ess intense bue coour. Specificay, considering the RGB coour space, in the bue band there are increased intensity vaues compared with the neighbourhood of the scratch; in the green band some of the pixes are atered in an unpredictabe way, usuay with a sight increase or decrease of intensity vaues; the red band is usuay uncorrupted, athough sometimes there coud be sma fuctuations of intensity vaues in pixes beonging to the scratch area. A detai of a bue scratch and its red, green and bue bands is given in Fig.. In order to have a better understanding of the scratch structure, we have anaysed a corrupted sequences of the above mentioned movie, identifying three types of bue scratches. The most common type incudes bue scratches such as the one appearing in Fig.. Looking at Fig. 2-(a), which shows the intensity curve of each coour band of the image of Fig., taken as horizonta section of the image intensity function at row 00, it ceary appears that the intensity curve of the bue band has a ridge in the scratch area. The described effect is sti more evident in Fig. 2-(b), where the horizonta projection of the intensity curve, taken as the mean over the image coumns of the intensity curve, is shown for the three coour bands. Specificay, in the scratch area the projection of the bue band has a ridge whose width w is about 9 pixes and whose height h is about 25 intensity vaues; the projection of the green band presents a sight decrease of about 5 intensity vaues around the center of the scratch. The projection of the red band does not show cear effects of the scratch, and red band can be therefore considered as uncorrupted. (a) (b) Fig. 2: Profies of the bue scratch in the image of Fig. : (a) intensity curves of the three bands, taken at row 00; (b) horizonta projection of the image intensity curves of the three bands. The second type incudes ess common bue scratches, as the one appearing in Fig. 3. Here we can observe that in the scratch area the projection of the bue band has a ridge accompanied by a shadow on the right; the tota scratch width w is about 5 pixes, whie the ridge height h is about 50 intensity vaues. The projections of the green and red bands show sma fuctuations of about 5 intensity vaues in the scratch area. The third type incudes ess common bue scratches that appear as two scratches cose together, as the one presented in Fig. 4. Here we can observe that in the scratch area the projection of the bue band has two neighboring ridges whose cumuative width is about 29 pixes, and whose heights are about 45 and 35 intensity vaues respectivey; the projection of the green band presents a sight increase of about 0 intensity 6
vaues around the center of the eft ridge. The projection of the red band does not show cear effects of the scratch, and red band can be therefore considered as uncorrupted. In Fig. 4 it is aso interesting to observe that the white scratch appearing on the eft of the bue one has coour band horizonta projections different from those of the bue scratch, since for white scratches the ridge affects a three coour bands in the same way. (a) (b) Fig.3: Exampe of a bue scratch: (a) coour image; (b) horizonta projection of the image intensity curves of the three bands. (a) (b) Fig.4: Exampe of a bue scratch: (a) coour image; (b) horizonta projection of the image intensity curves of the three bands. 3. Bue scratch detection 3.. Description of the method The idea at the basis of the bue scratch detection agorithm is that of searching, among a pixes beonging to vertica ines of the image, those having an intense bue coour. Specificay, our method consists in enhancing vertica edges of the image by appying a suitabe oca operator, and, due to the specific features of bue scratches, in restricting the search to vertica ridge edges, whose pixes are oca maxima for intensity curves of the bue band aong the horizonta direction. This 7
restriction aows to avoid considering contours of scene objects that appear as vertica ines but that are not image defects. The process eads to a modified version I E of the origina image, where bue vertica ines are particuary emphasized. In order to discriminate between pixes beonging to eventua bue vertica ines of the scene and pixes beonging to the bue scratch, we shoud be abe to determine the intense bue coour that is proper of bue scratches as emphasized in I E. The HSV coour space, which reies on the hue, saturation and vaue properties of each coour, aows to specify coours in a way that is cose to human experience of coours. Therefore, the conversion of the image to the HSV space can be hepfu in finding the bue coour range. Once the range of the bue coour searched has been determined, a binary image I B is obtained from the enhanced image I E, where pixes are marked if their coour is in this range. Finay, we identify the abscissae of vertica ines of image I B (and therefore those of vertica bue scratches of the origina image) as oca maxima of the horizonta projection of I B. Further improvement in the above described procedure can be obtained if the input image is suitaby preprocessed and if the resuting scratch mask is suitaby post-processed. The preprocessing is aimed at reducing noise that coud affect the input image, due to fim grain, dust and dirt, digitisation artifacts, etc.; the postprocessing is aimed at refining the scratch mask. 3.2. BSD agorithm Let I be the RGB image: { I( i, j,k), i =,,N; j =,,M;k,2,3} I = K K =, where N is the image height, M is the image width, and k=, 2, and 3 correspond to red, green, and bue bands respectivey. The proposed Bue Scratch Detection (BSD) agorithm for the detection of bue scratches in a digita image I is the foowing: BSD Agorithm Step. Pre-processing of the input image: noise reduction, with preservation of vertica edges; Step 2. Enhancement of vertica bue ines: a. enhancement of vertica edges; b. eimination of vertica edges not produced by vertica bue ines; Step 3. Binarisation: for each pixe of the image intensity matrix resuting from step 2: a. convert from RGB space to HSV space; b. if HSV vaues correspond to the intense bue coour, set to the corresponding pixe in the binary image I B ; Step 4. Refinement of the scratch mask: detection in the binary image I B of vertica ines that cover amost the whoe image height. 8
In our experiments, for Step we appy a one-dimensiona ow-pass fiter aong the coumns of the image intensity function, so that vertica edges are preserved. The fiter adopted is the mean in a pixes vertica neighbourhood of each pixe. The preprocessed image I P resuting from Step appied to the image of Fig. is shown in Fig. 5-(a). For Step 2.a we appy a one-dimensiona high-pass fiter aong the rows of the image intensity function. Supposing that w is the scratch width in the i-th row, for each pixe I P (i, j, ) the fiter adopted for our experiments is the fiter in a 3w pixes neighbourhood whose resut is described as: I j+ 3w / 2 E ( i, j, ) = a I P ( i,, ), = j 3w / 2 where: 2 = j w / 2,..., j + w / 2 a =. otherwise In Step 2.b we want to restrict our attention ony to vertica edges produced by bue vertica ines of width w; that is, we want to consider ony vertica edges whose bue band horizonta profie is a ridge edge of width w. For each pixe I P (i, j, ) we consider the three quantities: S L j w / 2 j+ w / 2 j+ 3w / 2 ( k) = I ( i,,k ), S ( k) = I ( i,,k ), S ( k) = I ( i,,k ), P = j 3w / 2 C P = j w / 2 R P = j+ w / 2+ and, for k=, 2, 3, set I E (i, j, k) = 0 if S C (3)< S L (3) or S C (3) < S R (3). This strong condition, in fact, ensures that the pixe I P (i, j, ) does not beong to a vertica ridge edge of width w of the bue band of image I P. Note that Steps 2.a and 2.b can be merged in a singe step, where for each pixe I P (i, j, ) we compute the above quantities S L (k), S C (k), and S R (k) and we set: I E ( i, j,k ) ( k) + 2 SC ( k) S R ( k) if SC ( 3) > S L ( 3) and SC ( 3) S R ( 3). S L > = 0 otherwise The resut of Step 2 on the image of Fig. is reported in Fig. 5-(b). The conversion from the RGB space to the HSV space adopted in Step 3.a is computed as foows: V = max(r, G, B), 0 S = [ V min( R, G, B)] / V V = 0, V 0 60 H = 60[2 60[4 0 ( G B) /( + ( B R) + ( R G) H + 360 S * V ) ] /( S * V ) ] /( S * V ) S = 0 V = R V = G V = B H < 0 where, for each pixe, the input vaues R, G, B are the pixe intensity vaues in the three bands, normaized in [0,], and the output vaues H, S, V are such that H [0,360], S [0,], and V [0,]. The bue coour is searched among pixes having hue H [80,300] (240 being the bue hue), saturation S > 0.45 and vaue V > 9
0.; these vaues take into account the transformations performed on the origina image I for obtaining the enhanced image I E, and have been experimentay chosen performing tests on severa different images. The binary image I B resuting from Step 3 on the image of Fig. is reported in Fig. 5-(c). In Step 4 we detect vertica ines of the binary image I B as oca maxima of the horizonta projection P of I B, whose j-th eement beonging to the generic band is defined as: P N ( j, ) = I ( i, j, ), j =, K,M. i= B Since bue scratches usuay cover most of the height of the image, a oca maximum for P in coumn j shoud be obtained for P(j, ) cose to the image height N. Therefore, we eiminate from the scratch mask the whoe coumn j as soon as P(j, ) is ower than a fixed percentage of N. We experimented that, in order to avoid deeting from the mask the scratch contours, obtaining a too sim mask, it is better to fix a percentage vaue ower than 00% of the height. In the genera case, a percentage equa to 50% is a good compromise between ack of fase positives and accurate detection of the bue scratch (see for instance Fig. 5-(d)). (a) (b) (c) (d) Fig. 5: BSD agorithm for the image of Fig. : Resuts of (a) Step ; (b) Step 2; (c) Step 3; (d) Step 4. It shoud be expicity observed that, in order to have an automatic restoration agorithm, the scratch width is preiminary computed using oca minima/maxima of the uminance cross-section, as in [6]. Other techniques, such as those used in [5, 8, 24, 30], coud be aternativey adopted. 0
4. Bue scratch remova 4.. Description of the method In anaysing the bue scratch features, we have aready observed in Section 2 that pixes beonging to the scratch have undergone an intensity vaue reduction or increase (depending on the considered coour band) compared with pixes in the scratch neighbourhood, but sti retain usefu information concerning the image structure. Therefore we approach the bue scratch remova probem as a partiay corrupted data probem. Looking more into detais at pots reported in Figs. 2, 3, and 4, we can observe that in uncorrupted areas of the image the dispacements of the bue band intensity vaues from those of the red band are ocay roughy constant; the same hods for dispacements of the green band from the red band. In the scratch area, instead, such dispacements appear strongy varying. Since, as aready observed in Section 2, the red band is usuay uncorrupted, we can restore the green and bue bands bringing their dispacement from the red band inside the scratch area to the same dispacement they have outside the scratch area. 4.2. BSR agorithm The Bue Scratch Remova (BSR) agorithm we have designed can be sketched as foows: BSR Agorithm For each row of the image: Step. preprocessing of the red band; Step 2. compute minimum, maximum and median dispacement of the green and bue bands from the red band in an uncorrupted neighbourhood of the scratch; Step 3. add median dispacement to a pixes of the green and bue bands beonging to the scratch area whose dispacement from the red band is beow minimum or above maximum dispacement. Step, here accompished with rank-order fiters, is required to take into account cases where the red band appears sighty corrupted. For Step 2 of BSR agorithm in the i-th row the neighbourhood N i,k for band k chosen in our experiments consists of three uncorrupted pixes beonging to the same row on the right of the scratch and three on the eft: { I(i, j,k) : j = b 3,b 2,b,b+ w,b+ w+,b+ w 2} N i, k + =, where w is the scratch width and b indicates the first coumn of the scratch. Defining the dispacement in the i-th row of the band k from the red band as: in Step 2 we compute: D { d(i, j,k) = I(i, j,) I(i, j,k) : I(i, j,k) } =, i, k N i,k
D max i,k = max d(i, j,k) Di,k min med { d(i, j,k) },D = min { d(i, j,k) },D = median{ D }, i,k d(i, j,k) Di,k i,k i, k and in Step 3 we restore the k-th coour component I(i, j, k) of a pixe as: min max if its vaue is not incuded in [ D, D ]. i,k i,k I med ( i, j,k) = I( i, j,) D i,k 5. Experimenta resuts 5.. Evauation of BSD agorithm BSD agorithm has been tested on severa rea images. From the visua inspection standpoint, the accuracy of the achieved resuts appears quite high, as it is shown by the scratch mask reported in Fig. 5-(d) for the image of Fig.. (a) (b) (c) Fig. 6: Compete cubic interpoating spine modes for bue band horizonta projection of the images in:(a) Fig. ; (b) Fig. 3; (c) Fig. 4. Anyway, the vaidity of the method caims for a more quantitative evauation. To this aim, we have artificiay corrupted rea images with bue scratches. We modeed the horizonta projection of the bue band 2
); in the scratch with a compete cubic spine interpoating extrema of the projection and its maximum point. Such mode is quite adequate for the genera bue scratch, as it is shown in Fig. 6-(a), where the compete cubic spine interpoating points marked as * is superimposed to the rea bue band projection of the image in Fig. -(a). Different bue scratch profies, such as those presented in Figs. 3-(b) and 4-(b), can be anaogousy modeed with a compete cubic spine interpoating suitabe points, as it is shown in Figs. 6-(b) and 6-(c). Moreover, since the behaviour of the green band projection cannot be modeed a priori, to create more reaistic artificia bue scratches for the green band projection we appy a simiar mode, scaed by a factor f. Specificay, we considered L=20 uncorrupted origina RGB images I, =,, L, each of size N M,; they incude we known images (e.g. Lena, Tiffany ) obtained by [7, 8, 3] as we as images taken from uncorrupted areas of aready digitised images of the movie Animai che attraversano a strada (2000). The corresponding images with an artificia bue scratch of odd width w and height h, denoted as I w,h, =,, L; w=5,7,, 5, h=50, 60, 70, have been obtained as: T [ ] if ( i, j) T w w,h T I ( ) ( i, j) + 0,sw,h ( j) /f,sw,h ( j) I i, j = () T I ( i, j) otherwise T where I ( i, j) = [ I ( i, j,),i ( i, j,2),i ( i, )] T w,h T w,h w,h w,h, I ( i, j) = I ( i, j,),i ( i, j,2),i ( i, ) j,3 [ ] T j,3 denotes the scratch domain, that is the rectanguar subset of the image domain of size N w having as first coumn the centre coumn b=m /2 of the image: Ω w ={ (i, j): i = b,, b +w-; j=,, N }, and s w,h (j) denotes the compete cubic spine interpoating points (b-,0), (b+w/2,h), (b+w,0). An exampe of an image I w,h artificiay corrupted with a bue scratch of width w=5 and height h=70 is given in Fig. 7, together with the horizonta projection of the intensity curves for its three bands; a the other artificiay corrupted images I w,h are avaiabe at web page [0], together with corresponding resuts obtained with the proposed agorithms. Knowing a priori the scratch mask for such images, we can then appy BSD agorithm to the corrupted images and have an error estimate. For each mask B w,h computed with BSD agorithm for the artificiay scratched image I w,h described in (), with size N M, we count: C w,h = number of correct detections (scratch pixes that are incuded in the computed scratch mask); F w,h = number of fase aarms (pixes not beonging to the scratch that are incuded in the computed scratch mask), and their rates RC w,h and RF w,h over their respective domains: RC w,h = C w,h /( N w), N w being the number of corrupted pixes (i.e. the dimension of the set w RF w,h = F w,h /(N M - N w)., w 3
(a) (b) (c) (d) Fig. 7: Exampe of artificia bue scratch: (a) origina image; (b) horizonta projection of the intensity curves of the three bands of origina image; (c) image corrupted with bue scratch of width w=5 and height h=70; (b) horizonta projection of the intensity curves of the three bands of corrupted image. Given the scratch width w and the height h, the measures adopted for the objective estimation of BSD agorithm are: L w,h mean correct detection rate: RC = RC L = the vaue of RC w,h, the better the detection resut; L w,h mean fase aarm rate: RF = RF L = w,h w,h. Such measure gives vaues in [0,]; the higher. Such measure gives vaues in [0,]; the ower the vaue of RF w,h, the better the detection resut. Vaues for RC w,h obtained with BSD agorithm appied to images I w,h described in (), varying the scratch width w and height h, are reported in Fig. 8. Here we can observe that they are generay quite high, even if they tend to decrease increasing the scratch width w and decreasing height h, in accordance with the increasing difficuty in detecting bue scratches as the width widens and as the height decreases. Corresponding RF w,h vaues are aways cose to zero. 4
Fig. 8: Error estimates for BSD agorithm appied to images described in (): mean correct detection rate. The computationa compexity of BSD agorithm, in terms of comparisons and arithmetic operations invoved, for an image of size N M affected by a bue scratch of width w is O(N*M*w). Just to give an idea, execution times of BSD agorithm, impemented in ANSI C on a Pentium IV, 2GHz, 256Mbytes RAM, for 24 bits RGB coour images of size 256*256, 576*720, and 2048*2880, affected by a bue scratch of width w ranging from 5 to 5 pixes are neary 0.03 s, 0.2 s, and 6.9 s, respectivey. We concude that execution times are quite ow for reduced size images; however, they are not sufficienty ow for rea time bue scratch detection in the case of movie resoution images. Paraeisation strategies for BSD agorithm are currenty under examination. 5.2. Evauation of BSR agorithm The resut of BSR agorithm appied to the naturay corrupted images of Figs., 3, and 4 and to the artificiay corrupted image of Fig. 7 is shown in Figs. 9, 0,, and 2, respectivey, together with the horizonta projection of the intensity curves of their three bands. Here we can observe that BSR agorithm performs in a quite satisfactory way from the subjective visua point of view. (a) (b) Fig 9: BSR agorithm for the image of Fig. : (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. 5
(a) (b) Fig 0: BSR agorithm for the image of Fig. 3: (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. (a) (b) Fig : BSR agorithm for the image of Fig. 4: (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. (a) (b) Fig 2: BSR agorithm for the image of Fig. 7: (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. 6
., Our aim now is to evauate the restoration quaity attained by BSR agorithm in terms of some objective measure. Therefore, we have again considered the artificiay corrupted images I w,h of size N M described by () used for the evauation of BSD agorithm. Given the scratch width w and the height h, et be, for =,, L: o the subimage of origina image I containing ony pixes in r the subimage of the restored image R w,h, obtained with BSR agorithm, containing ony pixes in w We consider the foowing objective measures, a computed as the mean over the three bands of each image: MeanMSE: mean, over the L images, of the Mean Square Error (MSE) between the origina and the restored images: w MeanMSE = L L = N w o r 2, where. is intended as vector norm. Such measure gives a nonnegative vaue; the smaer the vaue of MeanMSE, the better the restoration resut; MeanPSNR: mean, over the L images, of the Peak-to-Noise-Ratio between the origina and the restored images obtained considering the MSE: MeanPSNR = L L = 0* og 0 255 o N w 2 r 2. Such measure gives a nonnegative vaue; the higher the vaue of MeanPSNR, the better the restoration resut; MeanSSIM: mean, over the L images, of the Structura Simiarity Index [33] appied to the origina and the restored images: MeanSSIM = L L ( 2 µ o µ r + C )( o r C ) 2 σ + 2 ( C )( C ), 2 2 2 2 = µ + µ + σ + σ + o r o r 2 where C =(K *A) 2, C 2 =(K 2 *A) 2, K =0.0, K 2 =0.03, and A=255. Such measure gives vaues in [0,]; the higher the vaue of MeanSSIM, the better the restoration resut. Resuts in terms of the described measures obtained by BSR agorithm varying the scratch width w and height h are reported in Fig. 3, and show that statistica properties of the origina images are quite we 7
restored. Moreover, it can be observed that resuts obtained with a the considered measures show ower accuracy increasing the scratch width w and height h, in accordance with the increasing reconstruction difficuty as the reconstruction area widens and as the scratch contrast grows. (a) (b) (c) Fig 3: Error measures for BSR agorithm appied to images described in (): (a) MeanMSE; (b) MeanPSNR; (c) MeanSSIM. Such resuts have aso been compared with anaogous resuts obtained with an impementation of the inpainting agorithm (missing data approach) presented in [4], shown in Fig. 4. Here we can observe that a the considered error measures attain vaues worse than those obtained by BSR agorithm. (a) (b) (c) Fig 4: Error measures for the inpainting agorithm presented in [4] appied to images described in (): (a) MeanMSE; (b) MeanPSNR; (c) MeanSSIM. Conscious that, due to the specific features of bue scratches, the defect cannot be perfecty simuated on an uncorrupted image, we performed aso different accuracy measurements. Having at our disposa amost uniform rea images affected by bue scratches (reported in Figs. 5 and 6), we have taken the above measures on subbocks of such images. Specificay, for the image of Fig. 5-(a) showing a bue scratch of average width 9 (from coumn 27 to coumn 35), we have considered as corrupted image, I C, the subimage of the origina image containing a bock of coumns that incude the bue scratch (from coumn 2 to coumn 4), and we have considered two uncorrupted images, I UL and I UR, the first containing a bock of uncorrupted coumns on the eft of I C (from coumn 00 to 20) and the second containing a bock of uncorrupted coumns on the right of I C (from coumn 42 to 62). Appying BSR agorithm to the corrupted image I C, we have obtained the restored image I R. Sub-images I C, I UL,I UR, and I R, of the image of Fig. 5-(a) are reported in Fig. 8
5-(e). The mean, the standard deviation, and the L 2 norm for the corrupted image I C, for the uncorrupted images I UL and I UR and for the restored image I R are compared and reported in Tabe. The resuts confirm that BSR agorithm performs quite we for bue scratches of standard width. (a) (b) (c) (d) (e) Fig 5: Exampe of a bue scratch on a uniform background: (a) origina image; (b) horizonta projection of the intensity curves of the three bands of the origina image; (c) restored image; (d) horizonta projection of the intensity curves of the three bands of the restored image; (e) subimages considered for the measures reported in Tabe. 9
(a) (b) (c) (d) (e) Fig 6: Exampe of a wide bue scratch on a uniform background: (a) origina image; (b) horizonta projection of the intensity curves of the three bands of the origina image; (c) restored image; (d) horizonta projection of the intensity curves of the three bands of the restored image; (e) subimages considered for the measures reported in Tabe 2. 20
Sub-image Mean Std. Dev. L 2 norm I C 58.72 8.04 4352 I UL 58.22 6.96 4302 I UR 57.7 6.92 4225 I R 57.25 7.03 4232 Tabe : Mean, standard deviation and L 2 norm for the corrupted image I C, for the uncorrupted images I UL and I UR, and for the restored image I R reported in Fig. 5. Sub-image Mean Std. Dev. L 2 norm I C 26.48 3.7 2525 I UL 24.64 2.87 2349 I UR 26.07 2.63 248 I R 25.38 2.57 245 Tabe 2: Mean, standard deviation and L 2 norm for the corrupted image I C, for the uncorrupted images I UL and I UR, and for the restored image I R reported in Fig. 6. Anaogous measures for the amost uniform image of Fig. 6-(a) are reported in Tabe 2. In this case the average scratch width is 23 pixes; the corrupted image I C, containing a bock of coumns of the image incuding the bue scratch (from coumn to coumn 45), and the two uncorrupted images I UL and I UR, containing the bock from coumn 76 to 0 and from coumn 46 to 80, respectivey, are shown in Fig. 6- (e), together with the restored image I R obtained appying BSR agorithm to I C. The resuts confirm that BSR agorithm performs quite we aso for very arge bue scratches. The computationa compexity of BSR agorithm is quite ow, incuding a number of comparisons ineary proportiona to the size of the image and a number of arithmetic operations ineary proportiona to the number of rows of the image and the scratch width. Execution times of BSR agorithm, in ANSI C on a Pentium IV, 2GHz, 256Mbytes RAM, for 24 bits RGB coour images of size 256*256, 576*720, and 2048*2880, affected by a bue scratch of width w ranging from 5 to 5 pixes are neary 0.002 s, 0.0 s, and 0.55 s, respectivey. Therefore, we can concude that execution time is generay sufficienty ow for rea time bue scratch remova, even for movie resoution images. 6. Concusions We considered the probem of detecting and removing bue scratches from digita image sequences. In particuar, we anaysed in detai the specific features of such kind of scratches and proposed a detection method and a remova method that strongy rey on these features. A thorough anaysis of the agorithms accuracy, accompanied by severa numerica experiments carried out on both naturay and artificiay corrupted images, show that the proposed detection and remova agorithms produce satisfying resuts. The performance of the agorithms, in terms of execution times, is quite good for TV resoution images; however, for the case of movie resoution images the detection agorithm does not aow rea time computation, 2
requiring execution times in the order of tens of seconds. Paraeisation strategies for the detection agorithm are currenty under examination. 7. Acknowedgements The authors woud ike to express their gratitude to the anonymous referees for their usefu suggestions. This work has been partiay supported by the Regiona Competence Centre for the Deveopment and Transfer of Innovation Appied to Cutura and Environmenta Heritage (INNOVA) funded by Regione Campania, Itay. References [] A. Anzaone, A. Machì, A Method for Accurate Detection of Lineary Scratched Areas in Motion Pictures, Proceeding of IASTED-VIIP0, Spain (200) 565-570. [2] C. Baester, M. Bertamio, V. Casees, G. Sapiro, J. Verdera, Fiing-in by joint interpoation of vector fieds and grey eves, IEEE Transactions on Image Processing 0 (200) 200-2. [3] C. Baester, V. Casees, J. Verdera, Disoccusion by joint interpoation of vector fieds and grey eves, SIAM Journa Mutiscae Modeing and Simuation 2 (2003) 80-23. [4] R. Bornard, E. Lecan, L. Laborei, J.-H. Chenot, Missing data correction in sti images and image sequences, Proceedings of ACM Mutimedia (2002) 355-36. [5] T. Bretscheneider, O. Kao, P.J. Bones, Remova of Vertica Scratches in Digitised Historica Fim Sequences Using Waveet Decomposition, Proceedings of Image and Vision Computing Conference, Hamiton, New Zeaand (2000) 38-43. [6] V. Bruni, D. Vituano, A Generaized Mode for Scratch Detection, IEEE Transactions on Image Processing 3 (2004) 44-50. [7] Coor Texture Anaysis, Institut fur Computervisuaistik, Universitat Kobenz-Landau, http://www.unikobenz.de/fb4/institutes/icv/agpriese/research/coor%20texture%20anaysis. [8] Computer Vision Laboratory, Computer Science Department, University of Massachussets, http://viswww.cs.umass.edu/visib/houses/im4/images.htm. [9] E. Decenciere Ferrandiere, Restauration automatique de fims anciens, PhD Thesis, Ecoe Nationae Superieure des Mines de Paris (997). [0] DFR Laboratory, ICAR-CNR, Napes, www.na.icar.cnr.it/~maddaena./dfrlab/buescratches.htm. [] A.A. Efros, W.T. Freeman, Image Quiting for texture synthesis and transfer, Proceedings of 28th Annua Conference on Computer Graphics and Interactive Techniques (200) 34-346. [2] L. Joyeux, O. Buisson, B. Besserer, S. Boukir, Detection and Remova of Line Scratches in Motion Picture Fims, Proceedings of IEEE Internationa Conference on Computer Vision and Pattern Recognition, Fort Coins, Coorado (999) 548-553. [3] L. Joyeux, Reconstruction de sequences d'images haute resoution. Appication a a restauration de fims cinematographiques, PhD Thesis, Universitè de La Rochee (2000). 22
[4] L. Joyeux, S. Boukir, B. Besserer, O. Buisson, Reconstruction of Degraded Image Sequences. Appication to Fim Restoration, Image and Vision Computing 9 (200) 503-56. [5] L. Joyeux, S. Boukir, B. Besserer, Tracking and MAP reconstruction of ine scratches in degraded motion pictures, Machine Vision and Appications 3 (2002) 9-28. [6] O. Kao, J. Engehausen, Scratch Remova in Digitised Fim Sequences, Proceedings of Internationa Conference on Imaging Science, Systems, and Technoogy, CSREA Press (2000) 7-79. [7] A.C. Kokaram, R. Morris, W. Fitzgerad, P. Rayner, Detection/Interpoation of Missing Data in Image Sequences, IEEE Transactions on Image Processing 4 (995) 496-59. [8] A.C. Kokaram, Motion Picture Restoration: Digita Agorithms for Artefacts Suppression in Archived Fim and Video, Springer-Verag (998). [9] A.C. Kokaram, A statistica framework for picture reconstruction using AR modes, Proceedings of European Conference of Computer Vision, Workshop on Statistica Methods for Time Varying Image Sequences (2002) 73-78. [20] A.C. Kokaram, Parametric texture synthesis for fiing hoes in pictures, Proceedings of IEEE Internationa Conference in Image Processing (2002) 325-328. [2] A. Machì, F. Coura, F. Nicotra, Detection of irreguar inear scratches in aged motion picture frames and restoration using adaptive masks, Proceedings of IASTED Internationa Conference on Image Processing (2002) 254-259. [22] L. Maddaena, Efficient Methods for Scratch Remova in Image Sequences, Proceedings of th Internationa Conference on Image Anaysis and Processing, IEEE Computer Society (200) 547-552. [23] S. Masnou, J.-M. More, Leve-ines based disoccusion, Proceedings of 5th IEEE Internationa Conference on Image Processing 3 (998) 259-263. [24] R.D. Morris, Image Sequence Restoration Using Gibbs Distributions, PhD Thesis, University of Cambridge (995). [25] R.D. Morris, W.J. Fitzgerad, A.C. Kokaram, A samping based approach to ine scratch remova from motion picture frames, Proceedings of IEEE Internationa Conference on Image Processing (996) 80-804. [26] L. Rosenthaer, A. Wittmann, A. Gunz, R. Gschwind, Restoration of Od Movie Fims by Digita Image Processing, Proceedings of IMAGÈCOM 96, Bordeaux, France (996) 20-22. [27] T. Saito, T. Komatsu, T. Ohuchi, T. Seto, Image Processing for Restoration of Heaviy-Corrupted Od Fim Sequences, Proceedings of Internationa Conference on Pattern Recognition, Barceona, IEEE Computer Society (2000) 7-20. [28] G. Sapiro, Image inpainting, SIAM News 35 (2002). [29] J. Shen, Inpainting and the fundamenta probem of image processing, SIAM News 36 (2003). 23
[30] D. Tegoo, F. Isgrò, A Genetic Agorithm for Scratch Remova in Static Images, Proceedings of Internationa Conference on Image Anaysis and Processing, IEEE Computer Society (200) 507-5. [3] USC-SIPI Image Database, Eectrica Engineering Department, Signa & Image Processing Institute, University of Southern Caifornia, http://sipi.usc.edu/database. [32] J. Verdera, V. Casees, M. Bertamio, G. Sapiro, Inpainting surface hoes, Proceedings of IEEE Internationa Conference on Image Processing (2003), 4-7. [33] Z. Wang, L. Lu, A.C. Bovik, Video quaity assessment based on structura distortion measurement, Signa Processing: Image Communication 9 (2004) 2-32. 24