Restoration of blue scratches in digital image sequences

Avaiabe onine at www.sciencedirect.com Image and Vision Computing 26 (2008) 1314 1326 www.esevier.com/ocate/imavis Restoration of bue scratches in digita image sequences Lucia Maddaena a, *, Afredo Petrosino b,1 a Nationa Research Counci, Institute for High-Performance Computing and Networking, ICAR, Via P. Casteino 111, 80131 Napes, Itay b Department of Appied Science, University of Napes Parthenope, Via A. De Gasperi 5, 80133 Napes, Itay Received 24 January 2006; accepted 28 Apri 2006 Abstract In this paper, we consider the probem of detecting and removing bue scratches from digita image sequences. In particuar, we propose a detection method and a remova method that strongy rey on the specific features of such scratches. Evauation of the proposed methods, in terms of both accuracy and performance timings, and numerica experiments on rea images are reported. Ó 2006 Esevier B.V. A rights reserved. Keywords: Coour digita fim restoration; Bue scratch; Scratch detection; Scratch remova 1. Introduction Digita fim restoration is an evoving area of image processing aimed at studying methodoogies and techniques that aow to digitay restore damaged movies, in order to preserve their historica, artistic and cutura vaue and to faciitate their diffusion through modern communication media. Severa types of defects can be found in a damaged movie, such as dust and dirt, brightness and positiona instabiity, coour fading, scratches. We are specificay concerned with persistent scratches, intended as vertica ines appearing at the same ocation in subsequent frames of the image sequence. White or back scratches in od movies are mainy due to the abrasion of the fim caused by spurious partices present in the camera, during the sequence acquisition phase, or in the projector, during the fim projection. Instead, bue scratches, which are the subject of our interest, affect many modern coour movies and are due to spurious partices present in the transport mechanism of the equipment used for the deveopment of the fim. * Corresponding author. Te.: +39 081 6139522; fax: +39 081 6139531. E-mai addresses: ucia.maddaena@na.icar.cnr.it (L. Maddaena), afredo.petrosino@uniparthenope.it (A. Petrosino). 1 Te.: +39 081 5476601; fax: +39 081 5522293. Most of the methods reported in iterature that afford this kind of probem are articuated in a detection phase and a remova phase. The detection phase consists in searching, among a the vertica ines of the images, those that are not natura ines of the scene, which are characterized as defects. Severa methods have been adopted in the case of white or back scratches, such as those based on ow/high pass fiters [1,2], morphoogica fiters [3 8], adaptive binarization [9], discrete waveet decomposition [10], statistics and MAP techniques [11 13], or oca gradient measures in the image [14,15] or in the image cross-section [16], eventuay couped with techniques such as Hough transform [2,7,15] or Kaman fiter [4 7], and possiby foowed by Bayesian refinement strategies [2]. The resut of the detection phase over a sequence frame is a binary image, the scratch mask, of the same size, where white pixes are reated to scratch pixes in the corresponding sequence frame. The remova phase consists in reconstructing corrupted information in the defect area individuated by the scratch mask. Depending on the amount of the defect, information incuded in the scratch area can be either sighty or strongy affected by the defect; thus, the scratch remova probem can be approached either as a partiay corrupted data probem or as a missing data probem, respectivey. Foowing the partiay corrupted data approach, information 0262-8856/$ - see front matter Ó 2006 Esevier B.V. A rights reserved. doi:10.1016/j.imavis.2006.04.013

L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 1315 incuded in the artefact area is taken into account for the remova. In the case of back or white scratches, some authors adopted such approach and obtained remova through morphoogica fiters [8], interpoation or approximation [6,7,11], eventuay foowed by the reconstruction of high-frequency components via Fourier series [6] or via MAP techniques [7]. On the other hand, in the missing data approach pixes in the artefact area are considered missing even if they are ony sighty atered. This approach has been adopted for back or white scratches by many authors, who obtained remova through interpoation or approximation [10,17,18], the adoption of autoregressive modes [1,2], morphoogica fiters [3], or mean vector fiters [9], eventuay with the addition of east squares-based grain estimation [17]. Moreover, this approach is the one generay adopted for image inpainting, that is the set of techniques for making undetectabe modifications to images [19]; such techniques are generay used to fi-in missing data or to substitute information contained in sma image regions [20]. Inpainting has been pursued in iterature aso under different names, such as image interpoation (e.g. [21]) and fi-in (e.g. [22,23]); the probem has been afforded aso as disoccusion, since missing data can be considered as occusions hiding the image region to be reconstructed (e.g. [24,25]). Finay, inpainting is aso reated to texture synthesis, where the probem consists in generating, given a sampe texture, an unimited amount of image data, which wi be perceived by humans as having the same texture [26]; specificay, inpainting can be considered as a constrained texture synthesis probem [23,27,28]. Even though the probem of detection and remova of white or back scratches in digita image sequences has been considered by so many authors and severa commercia software systems incude modues for their restoration (such as the DIAMANT Suite distributed by HS-ART Digita Service GmbH or the Reviva distributed by da Vinci Systems, Inc.), the specific case of bue scratches has not been specificay addressed. As aready mentioned, they generay affect modern coour movies and, therefore, before aunching a new motion picture, the fim must be digitay restored by companies speciaized in digita effects and post-processing. The need for efficient and automatic toos abe to digitay remove bue scratches has been the primary input for the reported research. Specificay, in this paper we propose a method for the detection and remova of bue scratches in digita images that takes into account the specific features of such kind of scratches. The contents of this paper are as foows. In Section 2 the features of bue scratches are anaysed, in order to device suitabe digita restoration techniques. Sections 3 and 4 outine the methods that we propose for bue scratch detection and remova, respectivey, giving detais of the reated agorithms and impementations. In Section 5 we describe quaitative and quantitative resuts achieved by the proposed approach on rea images. Concusions are reported in Section 6. 2. Bue scratch characterisation Bue scratches in a digita image sequence appear as bue strips ocated aong a thin area covering from top to bottom of each sequence frame. Exampes of bue scratches are given in Figs. 1, 3 and 4, which are detais of 24 bits RGB coour images, originay of size 2880 2048, beonging to the movie Animai che attraversano a strada (2000). ccontrary to white or back scratches appearing in dated movies, the direction of bue scratches does not deviate too much from the vertica direction, and their position aong the horizonta direction does not change too much (no more than few pixes) from one frame to the next. Therefore, usuay bue scratches are not obique and have fixed position in consecutive frames of the image sequence. This is due to the fact that bue scratches are not caused by improper storage conditions or improper handing of the fim, as is usuay the case for ancient movies. They are rather caused by spurious partices present in the transport mechanism of the deveopment equipment; in the case of modern equipment, the transport mechanism stricty contros the sippage of the fim, which cannot move too much from its rectiinear trajectory. Due to this feature, restoration of bue scratches cannot rey on tempora discontinuity of the image intensity function aong the sequence; therefore, in the foowing we concentrate on purey spatia scratch detection and remova in each image. Fig. 1. Exampe of a bue scratch: (a) coour image; (b) scratch detai; (c) red, (d) green, and (e) bue band of scratch detai.

1316 L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 Inside the bue scratch area, origina information has been substituted by more or ess intense bue coour. Specificay, considering the RGB coour space, in the bue band there are increased intensity vaues compared with the neighbourhood of the scratch; in the green band some of the pixes are atered in an unpredictabe way, usuay with a sight increase or decrease of intensity vaues; the red band is usuay uncorrupted, athough sometimes there coud be sma fuctuations of intensity vaues in pixes beonging to the scratch area. A detai of a bue scratch and its red, green and bue bands is given in Fig. 1. In order to have a better understanding of the scratch structure, we have anaysed a corrupted sequences of the above mentioned movie, identifying three types of bue scratches. The most common type incudes bue scratches such as the one appearing in Fig. 1. Looking at Fig. 2(a), which shows the intensity curve of each coour band of the image of Fig. 1, taken as horizonta section of the image intensity function at row 100, it ceary appears that the intensity curve of the bue band has a ridge in the scratch area. The described effect is sti more evident in Fig. 2(b), where the horizonta projection of the intensity curve, taken as the mean over the image coumns of the intensity curve, is shown for the three coour bands. Specificay, in the scratch area the projection of the bue band has a ridge whose width w is about 9 pixes and whose height h is about 25 intensity vaues; the projection of the green band presents a sight decrease of about five intensity vaues around the centre of the scratch. The projection of the red band does not show cear effects of the scratch, and red band can be therefore considered as uncorrupted. The second type incudes ess common bue scratches, as the one appearing in Fig. 3. Here, we can observe that in the scratch area the projection of the bue band has a ridge accompanied by a shadow on the right; the tota scratch width w is about 15 pixes, whie the ridge height h is about 50 intensity vaues. The projections of the green and red bands show sma fuctuations of about 5 intensity vaues in the scratch area. The third type incudes ess common bue scratches that appear as two scratches cose together, as the one presented in Fig. 4. Here, we can observe that in the scratch area the projection of the bue band has two neighboring ridges whose cumuative width is about 29 pixes, and whose heights are about 45 and 35 intensity vaues, respectivey; the projection of the green band presents a sight increase of about 10 intensity vaues around the centre of the eft ridge. The projection of the red band does not show cear effects of the scratch, and red band can be therefore considered as uncorrupted. In Fig. 4, it is aso interesting to observe that the white scratch appearing on the eft of the bue one has coour band horizonta projections different from those of the bue scratch, since for white scratches the ridge affects a three coour bands in the same way. 3. Bue scratch detection 3.1. Description of the method Fig. 2. Profies of the bue scratch in the image of Fig. 1: (a) intensity curves of the three bands, taken at row 100; (b) horizonta projection of the image intensity curves of the three bands. The idea at the basis of the bue scratch detection agorithm is that of searching, among a pixes beonging to vertica ines of the image, those having an intense bue coour. Specificay, our method consists in enhancing vertica edges of the image by appying a suitabe oca operator, and, due to the specific features of bue scratches, in restricting the search to vertica ridge edges, whose pixes are oca maxima for intensity curves of the bue band aong the horizonta direction. This restriction aows to avoid considering contours of scene objects that appear as vertica ines but that are not image defects. The process eads to a modified version I E of the origina image, where bue vertica ines are particuary emphasized. In order to discriminate between pixes beonging to eventua bue vertica ines of the scene and pixes beonging to the bue scratch, we shoud be abe to determine the intense bue coour that is proper of bue scratches as emphasized in I E. The HSV coour space, which reies on the hue, saturation and vaue properties of each coour, aows to specify coours in a way that is cose to human experience of coours. Therefore, the conversion of the image to the HSV space can be hepfu in finding the bue coour range.

L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 1317 Fig. 3. Exampe of a bue scratch: (a) coour image; (b) horizonta projection of the image intensity curves of the three bands. Fig. 4. Exampe of a bue scratch: (a) coour image; (b) horizonta projection of the image intensity curves of the three bands. Once the range of the bue coour searched has been determined, a binary image I B is obtained from the enhanced image I E, where pixes are marked if their coour is in this range. Finay, we identify the abscissae of vertica ines of image I B (and therefore those of vertica bue scratches of the origina image) as oca maxima of the horizonta projection of I B. Further improvement in the above described procedure can be obtained if the input image is suitaby pre-processed and if the resuting scratch mask is suitaby post-processed. The pre-processing is aimed at reducing noise that coud affect the input image, due to fim grain, dust and dirt, digitisation artefacts, etc.; the post-processing is aimed at refining the scratch mask. 3.2. BSD agorithm Let I be the RGB image I ¼fIði;j;kÞ; i ¼ 1;...;N; j ¼ 1;...;M; k ¼ 1;2;3g; where N is the image height, M is the image width, and k = 1, 2, and 3 correspond to red, green, and bue bands, respectivey. The proposed bue scratch detection (BSD) agorithm for the detection of bue scratches in a digita image I is the foowing: BSD Agorithm. Step 1. Pre-processing of the input image: noise reduction, with preservation of vertica edges; Step 2. Enhancement of vertica bue ines: a. enhancement of vertica edges; b. eimination of vertica edges not produced by vertica bue ines; Step 3. Binarisation: for each pixe of the image intensity matrix resuting from step 2: a. convert from RGB space to HSV space; b. if HSV vaues correspond to the intense bue coour, set to 1 the corresponding pixe in the binary image I B ; Step 4. Refinement of the scratch mask: detection in the binary image I B of vertica ines that cover amost the whoe image height. In our experiments, for Step 1 we appy a one-dimensiona ow-pass fiter aong the coumns of the image intensity function, so that vertica edges are preserved. The fiter adopted is the mean in a 11 pixes vertica neighbourhood of each pixe. The preprocessed image I P resuting from Step 1 appied to the image of Fig. 1 is shown in Fig. 5(a). For Step 2a, we appy a one-dimensiona high-pass fiter aong the rows of the image intensity function. Supposing that w is the scratch width in the ith row, for each pixe

1318 L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 I P (i,j,æ) the fiter adopted for our experiments is the fiter in a 3w pixes neighbourhood whose resut is described as: I E ði;j;þ ¼ jþ3w=2 X ¼j 3w=2 where: 2 ¼ j w=2;...;j þ w=2 a ¼ : 1 otherwise In Step 2b we want to restrict our attention ony to vertica edges produced by bue vertica ines of width w; that is, we want to consider ony vertica edges whose bue band horizonta profie is a ridge edge of width w. For each pixe I P (i,j,æ) we consider the three quantities: S L ðkþ ¼ S R ðkþ ¼ j w=2 1 X ¼j 3w=2 jþ3w=2 X ¼jþw=2þ1 Fig. 5. BSD agorithm for the image of Fig. 1: resuts of (a) Step 1; (b) Step 2; (c) Step 3; (d) Step 4. 8 >< S L ðkþþ2s C ðkþ S R ðkþ if S C ð3þ > S L ð3þ and I E ði;j;kþ¼ S C ð3þ > S R ð3þ : >: 0 otherwise a I P ði;;þ; The resut of Step 2 on the image of Fig. 1 is reported in Fig. 5(b). The conversion from the RGB space to the HSV space adopted in Step 3a is computed as foows: jþw=2 I P ði;;kþ; S C ðkþ ¼ X I P ði;j;kþ; ¼j w=2 I P ði;;kþ; and, for k =1, 2, 3, set I E (i,j,k) =0 if S C (3) < S L (3) or S C (3) < S R (3). This strong condition, in fact, ensures that the pixe I P (i,j,æ) does not beong to a vertica ridge edge of width w of the bue band of image I P. Note that Steps 2a and 2b can be merged in a singe step, where for each pixe I P (i, j,æ) we compute the above quantities S L (k), S C (k), and S R (k) and we set: V ¼ maxðr;g;bþ; S ¼ 0 ifv ¼ 0 ; ½V minðr;g;bþš=v otherwise 8 0 if S ¼ 0 >< 60ðG BÞ=ðSV Þ if V ¼ R H ¼ 60½2 þðb RÞ=ðSV ÞŠ if V ¼ G ; 60½2 þðr GÞ=ðSV ÞŠ if V ¼ B >: H þ 360 if H < 0 where for each pixe, the input vaues R, G, B are the pixe intensity vaues in the three bands, normaized in [0,1], and the output vaues H, S, V are such that H2[0,360], S2[0,1], and V2[0,1]. The bue coour is searched among pixes having hue H2[180,300] (240 being the bue hue), saturation S > 0.45 and vaue V > 0.1; these vaues take into account the transformations performed on the origina image I for obtaining the enhanced image I E, and have been experimentay chosen performing tests on severa different images. The binary image I B resuting from Step 3 on the image of Fig. 1 is reported in Fig. 5(c).

L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 1319 In Step 4, we detect vertica ines of the binary image I B as oca maxima of the horizonta projection P of I B, whose jth eement beonging to the generic band is defined as: Pðj;Þ ¼ XN i¼1 I B ði;j;þ; j ¼ 1;...;M: Since bue scratches usuay cover most of the height of the image, a oca maximum for P in coumn j shoud be obtained for P(j,Æ) cose to the image height N. Therefore, we eiminate from the scratch mask the whoe coumn j as soon as P(j,Æ)is ower than a fixed percentage of N. We experimented that, in order to avoid deeting from the mask the scratch contours, obtaining a too sim mask, it is better to fix a percentage vaue ower than 100% of the height. In the genera case, a percentage equa to 50% is a good compromise between ack of fase positives and accurate detection of the bue scratch (see for instance Fig. 5(d)). It shoud be expicity observed that, in order to have an automatic restoration agorithm, the scratch width is preiminary computed using oca minima/maxima of the uminance cross-section, as in [16]. Other techniques, such as those used in [2,7,11,13], coud be aternativey adopted. 4. Bue scratch remova 4.1. Description of the method In anaysing the bue scratch features, we have aready observed in Section 2 that pixes beonging to the scratch have undergone an intensity vaue reduction or increase (depending on the considered coour band) compared with pixes in the scratch neighbourhood, but sti retain usefu information concerning the image structure. Therefore, we approach the bue scratch remova probem as a partiay corrupted data probem. Looking more into detais at pots reported in Figs. 2 4, we can observe that in uncorrupted areas of the image the dispacements of the bue band intensity vaues from those of the red band are ocay roughy constant; the same hods for dispacements of the green band from the red band. In the scratch area, instead, such dispacements appear strongy varying. Since, as aready observed in Section 2, the red band is usuay uncorrupted, we can restore the green and bue bands bringing their dispacement from the red band inside the scratch area to the same dispacement they have outside the scratch area. 4.2. BSR agorithm The bue scratch remova (BSR) agorithm we have designed can be sketched as foows: BSR Agorithm. For each row of the image: Step 1. Preprocessing of the red band; Step 2. Compute minimum, maximum and median dispacement of the green and bue bands from the red band in an uncorrupted neighbourhood of the scratch; Step 3. Add median dispacement to a pixes of the green and bue bands beonging to the scratch area whose dispacement from the red band is beow minimum or above maximum dispacement. Step 1, here accompished with rank-order fiters, is required to take into account cases where the red band appears sighty corrupted. For Step 2 of BSR agorithm in the ith row the neighbourhood N i,k for band k chosen in our experiments consists of three uncorrupted pixes beonging to the same row on the right of the scratch and three on the eft N i;k ¼fIði;j;kÞ: j ¼ b 3; b 2; b 1; b þ w; b þ w þ 1; b þ w þ 2g; where w is the scratch width and b indicates the first coumn of the scratch. Defining the dispacement in the ith row of the band k from the red band as: D i;k ¼fdði;j;kÞ ¼ Iði;j;1Þ Iði;j;kÞ:Iði;j;kÞ 2N i;k g; in Step 2 we compute D max i;k ¼ max fdði;j;kþg; dði;j;kþ2d i;k D med i;k ¼ medianfd i;k g; D min i;k ¼ min dði;j;kþ2d i;k fdði;j;kþg; and in Step 3 we restore the kth coour component I(i, j, k) of a pixe as: Iði;j;kÞ ¼Iði;j;1Þ D med i;k ; if its vaue is not incuded in ½D min i;k ; Dmax i;k Š: 5. Experimenta resuts 5.1. Evauation of BSD agorithm BSD agorithm has been tested on severa rea images. From the visua inspection standpoint, the accuracy of the achieved resuts appears quite high, as it is shown by the scratch mask reported in Fig. 5(d) for the image of Fig. 1. Anyway, the vaidity of the method caims for a more quantitative evauation. To this aim, we have artificiay corrupted rea images with bue scratches. We modeed the horizonta projection of the bue band in the scratch with a compete cubic spine interpoating extrema of the projection and its maximum point. Such mode is quite adequate for the genera bue scratch, as it is shown in Fig. 6(a), where the compete cubic spine interpoating points marked as * is superimposed to the rea bue band projection of the image in Fig. 1(a). Different bue scratch profies, such as those presented in Figs. 3(b) and 4(b), can

1320 L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 31] as we as images taken from uncorrupted areas of aready digitised images of the movie Animai che attraversano a strada (2000). The corresponding images with an artificia bue scratch of odd width w and height h, denoted as I w;h ; ¼ 1;...;L; w ¼ 5;7;...;15; h ¼ 50;60;70; have been obtained as I ~w;h ði;jþ : I! jði;jþ : þ½0;s w;h ðjþ=f ;s w;h ðjþš : if ði;jþ 2X w I! ði;jþ : ; otherwise where! I ði;jþ : ¼½I ði;j;1þš;i ði;j;2þ;i ði;j;3þš : ; I ~w;h X w ði;jþ : ¼½I w;h ði;j;1þ; I w;h ði;j;2þ;i w;h ði;j;3þš : ; denotes the scratch domain, that is the rectanguar subset of the image domain of size N w having as first coumn the centre coumn b = M /2 of the image: X w ¼fði;jÞ:i ¼ b;...;b þ w 1; j ¼ 1;...;N g; and s w,h (j) denotes the compete cubic spine interpoating points (b 1,0), (b + w/2,h), (b + w,0). An exampe of an image I w;h artificiay corrupted with a bue scratch of width w = 15 and height h = 70 is given in Fig. 7, together with the horizonta projection of the intensity curves for its three bands; a the other artificiay corrupted images I w;h are avaiabe at web page [32], together with corresponding resuts obtained with the proposed agorithms. Knowing a priori the scratch mask for such images, we can then appy BSD agorithm to the corrupted images and have an error estimate. For each mask B w;h computed with BSD agorithm for the artificiay scratched image I w;h described in (1), with size N M, we count: C w;h = number of correct detections (scratch pixes that are incuded in the computed scratch mask); F w;h = number of fase aarms (pixes not beonging to the scratch that are incuded in the computed scratch mask), and their rates RC w;h domains: and RF w;h over their respective Fig. 6. Compete cubic interpoating spine modes for bue band horizonta projection of the images in: (a) Fig. 1; (b) Fig. 3; (c) Fig. 4. be anaogousy modeed with a compete cubic spine interpoating suitabe points, as it is shown in Fig. 6(b) and (c). Moreover, since the behaviour of the green band projection cannot be modeed a priori, to create more reaistic artificia bue scratches for the green band projection we appy a simiar mode, scaed by a factor f. Specificay, we considered L = 20 uncorrupted origina RGB images I, =1,...,L, each of size N M ; they incude we known images (e.g. Lena, Tiffany ) obtained by [29 RC w;h ¼ C w;h =ðn wþ;n w being the number of corrupted pixes (i.e. the dimension of the set X w ); RF w;h ¼ F w;h =ðn M N wþ: Given the scratch width w and the height h, the measures adopted for the objective estimation of BSD agorithm are: mean correct detection rate: RC w;h ¼ 1=L XL ¼1 RC w;h : Such measure gives vaues in [0,1]; the higher the vaue of RC w,h, the better the detection resut; mean fase aarm rate:

L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 1321 Fig. 7. Exampe of artificia bue scratch: (a) origina image; (b) horizonta projection of the intensity curves of the three bands of origina image; (c) image corrupted with bue scratch of width w = 15 and height h = 70; (b) horizonta projection of the intensity curves of the three bands of corrupted image. RF w;h ¼ 1=L XL ¼1 RF w;h : Such measure gives vaues in [0, 1]; the ower the vaue of RF w,h, the better the detection resut. Vaues for RC w,h obtained with BSD agorithm appied to images I w;h described in (1), varying the scratch width w and height h, are reported in Fig. 8. Here we can observe that they are generay quite high, even if they tend to decrease increasing the scratch width w and decreasing height h, in accordance with the increasing difficuty in detecting bue scratches as the width widens and as the height decreases. Corresponding RF w,h vaues are aways cose to zero. The computationa compexity of BSD agorithm, in terms of comparisons and arithmetic operations invoved, for an image of size N M affected by a bue scratch of width w is O(N M w). Just to give an idea, execution times of BSD agorithm, impemented in ANSI C on a Pentium IV, 2 GHz, 256 Mbytes RAM, for 24 bits RGB coour images of size 256 256, 576 720, and 2048 2880, affected by a bue scratch of width w ranging from 5 to 15 pixes are neary 0.03, 0.2, and 6.9 s, respectivey. We concude that execution times are quite ow for reduced size images; however, they are not sufficienty ow for rea time bue scratch detection in the case of movie resoution images. Paraeisation strategies for BSD agorithm are currenty under examination. Fig. 8. Error estimates for BSD agorithm appied to images described in (1): mean correct detection rate. 5.2. Evauation of BSR agorithm The resut of BSR agorithm appied to the naturay corrupted images of Figs. 1, 3, and 4 and to the artificiay corrupted image of Fig. 7 is shown in Figs. 9 12, respectivey, together with the horizonta projection of the intensity curves of their three bands. Here, we can observe that BSR agorithm performs in a quite satisfactory way from the subjective visua point of view.

1322 L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 Fig. 9. BSR agorithm for the image of Fig. 1: (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. Fig. 10. BSR agorithm for the image of Fig. 3: (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. Fig. 11. BSR agorithm for the image of Fig. 4: (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. Our aim now is to evauate the restoration quaity attained by BSR agorithm in terms of some objective measure. Therefore, we have again considered the artificiay corrupted images I w;h of size N M described by (1) used for the evauation of BSD agorithm. Given the scratch width w and the height h, et be, for =1,...,L: o the subimage of the origina image I containing ony pixes in X w, r the subimage of the restored image R w;h, obtained with BSR agorithm, containing ony pixes in X w. We consider the foowing objective measures, a computed as the mean over the three bands of each image: MeanMSE: mean, over the L images, of the mean square error (MSE) between the origina and the restored images: MeanMSE ¼ 1 L X L ¼1 1 N w ko r k 2 ;

L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 1323 Fig. 12. BSR agorithm for the image of Fig. 7: (a) restored image; (b) horizonta projection of the intensity curves of the three bands of the restored image. Fig. 13. Error estimates for BSR agorithm appied to images described in (1): (a) MeanMSE; (b) MeanPSNR; (c) MeanSSIM. Fig. 14. Error estimates for the inpainting agorithm presented in [27] appied to images described in (1): (a) MeanMSE; (b) MeanPSNR; (c) MeanSSIM.

1324 L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 where kæk is intended as vector norm. Such measure gives a nonnegative vaue; the smaer the vaue of MeanMSE, the better the restoration resut; MeanPSNR: mean, over the L images, of the peak-signa-to-noise-ratio between the origina and the restored images obtained considering the MSE:!! MeanPSNR ¼ 1 X L 255 2 10 og L 10 1 ¼1 ko : N w r k 2 Such measure gives a nonnegative vaue; the higher the vaue of MeanPSNR, the better the restoration resut; MeanSSIM: mean, over the L images, of the structura simiarity index [33] appied to the origina and the restored images: MeanSSIM ¼ 1 L X L ¼1 ð2 o r þ C 1 Þð2 r o r þ C 2 Þ ð 2 o þ 2 r þ C 1 Þðr 2 o þ r 2 r þ C 2 Þ ; where C 1 =(K 1 *A) 2, C 2 =(K 2 *A) 2, K 1 = 0.01, K 2 = 0.03, and A = 255.Such measure gives vaues in [0,1]; the higher the vaue of MeanSSIM, the better the restoration resut. Resuts in terms of the described measures obtained by BSR agorithm varying the scratch width w and height h are reported in Fig. 13, and show that statistica properties of the origina images are quite we restored. Moreover, it can be observed that resuts obtained with a the considered measures show ower accuracy increasing the scratch width w and height h, in accordance with the increasing Fig. 15. Exampe of a bue scratch on a uniform background: (a) origina image; (b) horizonta projection of the intensity curves of the three bands of the origina image; (c) restored image; (d) horizonta projection of the intensity curves of the three bands of the restored image; (e) subimages considered for error estimates reported in Tabe 1.

L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 1325 reconstruction difficuty as the reconstruction area widens and as the scratch contrast grows. Such resuts have aso been compared with anaogous resuts obtained with an impementation of the inpainting agorithm (missing data approach) presented in [27], shown in Fig. 14. Here we can observe that a the considered error measures attain vaues worse than those obtained by BSR agorithm. Conscious that, due to the specific features of bue scratches, the defect cannot be perfecty simuated on an uncorrupted image, we performed aso different accuracy measurements. Having at our disposa amost uniform rea images affected by bue scratches (reported in Figs. 15 and 16), we have taken the above measures on subbocks of such images. Specificay, for the image of Fig. 15(a) showing a bue scratch of average width 9 (from coumn 127 to coumn 135), we have considered as corrupted image, I C, the subimage of the origina image containing a bock of coumns that incude the bue scratch (from coumn 121 to 141), and we have considered two uncorrupted images, I UL and I UR, the first containing a bock of uncorrupted coumns on the eft of I C (from coumn 100 to 120) and the second containing a bock of uncorrupted coumns on the right of I C (from coumn 142 to 162). Appying BSR agorithm to the corrupted image I C, we have obtained the restored image I R. Subimages I C, I UL, I UR, and I R,of the image of Fig. 15(a) are reported in Fig. 15(e). The mean, the standard deviation, and the L 2 norm for the corrupted image I C, for the uncorrupted images I UL and I UR and for the restored image I R are compared and reported in Tabe 1. The resuts confirm that BSR agorithm performs quite we for bue scratches of standard width. Anaogous measures for the amost uniform image of Fig. 16(a) are reported in Tabe 2. In this case, the average scratch width is 23 pixes; the corrupted image I C, containing a bock of coumns of the image incuding the bue scratch (from coumn 111 to 145), and the two uncorrupted images I UL and I UR, containing the bock from coumn 76 to 110 and from coumn 146 to 180, respectivey, are shown in Fig. 16(e), together with the restored image I R obtained appying BSR agorithm to I C. The resuts confirm that BSR agorithm performs quite we aso for very arge bue scratches. The computationa compexity of BSR agorithm is quite ow, incuding a number of comparisons ineary proportiona to the size of the image and a number of arithmetic operations ineary proportiona to the number of rows of the image and the scratch width. Execution times of BSR agorithm, in ANSI C on a Pentium IV, 2 GHz, 256 Mbytes RAM, for 24 bits RGB coour images of size 256 256, 576 720, and 2048 2880, affected by a bue scratch of width w ranging from 5 to 15 pixes are neary 0.002, 0.01, and 0.55 s, respectivey. Therefore, we can concude that execution time is generay sufficienty ow for rea time bue scratch remova, even for movie resoution images. Tabe 1 Mean, standard deviation and L 2 norm for the corrupted image I C, for the uncorrupted images I UL and I UR and for the restored image I R reported in Fig. 15 Sub-image Mean Std. dev. L 2 norm IC 58.72 8.04 4.352 IUL 58.22 6.96 4.302 IUR 57.17 6.92 4.225 IR 57.25 7.03 4.232 Fig. 16. Exampe of a wide bue scratch on a uniform background: (a) origina image; (b) horizonta projection of the intensity curves of the three bands of the origina image; (c) restored image; (d) horizonta projection of the intensity curves of the three bands of the restored image; (e) subimages considered for error estimates reported in Tabe 2. Tabe 2 Mean, standard deviation and L 2 norm for the corrupted image I C, for the uncorrupted images I UL and I UR and for the restored image I R reported in Fig. 16 Sub-image Mean Std. dev. L 2 norm IC 26.48 3.17 2.525 IUL 24.64 2.87 2.349 IUR 26.07 2.63 2.481 IR 25.38 2.57 2.415

1326 L. Maddaena, A. Petrosino / Image and Vision Computing 26 (2008) 1314 1326 6. Concusions We considered the probem of detecting and removing bue scratches from digita image sequences. In particuar, we anaysed in detai the specific features of such kind of scratches and proposed a detection method and a remova method that strongy rey on these features. A thorough anaysis of the agorithms accuracy, accompanied by severa numerica experiments carried out on both naturay and artificiay corrupted images, show that the proposed detection and remova agorithms produce satisfying resuts. The performance of the agorithms, in terms of execution times, is quite good for TV resoution images; however, for the case of movie resoution images the detection agorithm does not aow rea time computation, requiring execution times in the order of tens of seconds. Paraeisation strategies for the detection agorithm are currenty under examination. Acknowedgements The authors woud ike to express their gratitude to the anonymous referees for their usefu suggestions. This work has been partiay supported by the Regiona Competence Centre for the Deveopment and Transfer of Innovation Appied to Cutura and Environmenta Heritage (INNOVA) funded by Regione Campania, Itay. References [1] A.C. Kokaram, R. Morris, W. Fitzgerad, P. Rayner, Detection/ interpoation of missing data in image sequences, IEEE Transactions on Image Processing 4 (1995) 1496 1519. [2] A.C. Kokaram, Motion Picture Restoration: Digita Agorithms for Artefacts Suppression in Archived Fim and Video, Springer, Berin, 1998. [3] E. Decenciere Ferrandiere, Restauration automatique de fims anciens, PhD Thesis, Ecoe Nationae Superieure des Mines de Paris, 1997. [4] L. Joyeux, O. Buisson, B. Besserer, S. Boukir, Detection and remova of ine scratches in motion picture fims, Proceedings of IEEE Internationa Conference on Computer Vision and Pattern Recognition, Fort Coins, Co, 1999, pp. 548 553. [5] L. Joyeux, Reconstruction de sequences d images haute resoution. Appication a a restauration de fims cinematographiques, PhD Thesis, Universite de La Rochee, 2000. [6] L. Joyeux, S. Boukir, B. Besserer, O. Buisson, Reconstruction of degraded image sequences, appication to fim restoration, Image and Vision Computing 19 (2001) 503 516. [7] L. Joyeux, S. Boukir, B. Besserer, Tracking and MAP reconstruction of ine scratches in degraded motion pictures, Machine Vision and Appications 13 (2002) 119 128. [8] T. Saito, T. Komatsu, T. Ohuchi, T. Seto, Image processing for restoration of heaviy-corrupted od fim sequences, Proceedings of Internationa Conference on Pattern Recognition, Barceona, IEEE Computer Society, 2000, pp. 17 20. [9] O. Kao, J. Engehausen, Scratch remova in digitised fim sequences, Proceedings of Internationa Conference on Imaging Science, Systems, and Technoogy, CSREA Press, 2000, pp. 171 179. [10] T. Bretscheneider, O. Kao, P.J. Bones, Remova of vertica scratches in digitised historica fim sequences using waveet decomposition, Proceedings of Image and Vision Computing Conference, Hamiton, New Zeaand, 2000, pp. 38 43. [11] R.D. Morris, Image sequence restoration using Gibbs distributions, PhD Thesis, University of Cambridge, 1995. [12] R.D. Morris, W.J. Fitzgerad, A.C. Kokaram, A samping based approach to ine scratch remova from motion picture frames, Proceedings of IEEE Internationa Conference on Image Processing, vo. 1, 1996, pp. 801 804. [13] D. Tegoo, F. Isgro, A genetic agorithm for scratch remova in static images, Proceedings of Internationa Conference on Image Anaysis and Processing, IEEE Computer Society, 2001, pp. 507 511. [14] A. Anzaone, A. Machi, A method for accurate detection of ineary scratched areas in motion pictures, Proceeding of IASTED-VIIP01, Spain, 2001, pp. 565 570. [15] A. Machí, F. Coura, F. Nicotra, Detection of irreguar inear scratches in aged motion picture frames and restoration using adaptive masks, Proceedings of IASTED Internationa Conference on Image Processing, 2002, pp. 254 259. [16] V. Bruni, D. Vituano, A generaized mode for scratch detection, IEEE Transactions on Image Processing 13 (2004) 44 50. [17] L. Maddaena, Efficient methods for scratch remova in image sequences, Proceedings of 11th Internationa Conference on Image Anaysis and Processing, IEEE Computer Society, 2001, pp. 547 552. [18] L. Rosenthaer, A. Wittmann, A. Gunz, R. Gschwind, Restoration of od movie fims by digita image processing, Proceedings of IMAGCOM 96, Bordeaux, France, 1996, pp. 20 22. [19] G. Sapiro, Image inpainting, SIAM News (2002) 35. [20] J. Verdera, V. Casees, M. Bertamio, G. Sapiro, Inpainting surface hoes, Proceedings of IEEE Internationa Conference on Image Processing, 2003, pp. 14 17. [21] J. Shen, Inpainting and the fundamenta probem of image processing, SIAM News (2003) 36. [22] C. Baester, M. Bertamio, V. Casees, G. Sapiro, J. Verdera, Fiingin by joint interpoation of vector fieds and grey eves, IEEE Transactions on Image Processing 10 (2001) 1200 1211. [23] A.C. Kokaram, A statistica framework for picture reconstruction using AR modes, Proceedings of European Conference of Computer Vision, Workshop on Statistica Methods for Time Varying Image Sequences, 2002, pp. 73 78. [24] C. Baester, V. Casees, J. Verdera, Disoccusion by joint interpoation of vector fieds and grey eves, SIAM Journa Mutiscae Modeing and Simuation 2 (2003) 80 123. [25] S. Masnou, J.-M. More, Leve-ines based disoccusion, Proceedings of Fifth IEEE Internationa Conference on Image Processing, vo. 3, 1998, pp. 259 263. [26] A.A. Efros, W.T. Freeman, Image quiting for texture synthesis and transfer, Proceedings of 28th Annua Conference on Computer Graphics and Interactive Techniques, 2001, pp. 341 346. [27] R. Bornard, E. Lecan, L. Laborei, J.-H. Chenot, Missing data correction in sti images and image sequences, Proceedings of ACM Mutimedia, 2002, pp. 355 361. [28] A.C. Kokaram, Parametric texture synthesis for fiing hoes in pictures, Proceedings of IEEE Internationa Conference in Image Processing, 2002, pp. 325 328. [29] Coor Texture Anaysis, Institut fur Computervisuaistik, Universitat Kobenz-Landau, http://www.uni-kobenz.de/fb4/institutes/icv/ AGPriese/Research [30] Computer Vision Laboratory, Computer Science Department, University of Massachussets, http://vis-www.cs.umass.edu/visib/houses/ IM4/images.htm [31] USC-SIPI Image Database, Eectrica Engineering Department, Signa and Image Processing Institute, University of Southern Caifornia, http://sipi.usc.edu/database [32] DFR Laboratory, ICAR-CNR, Napes, www.na.icar.cnr.it/maddaena./dfrlab/buescratches.htm [33] Z. Wang, L. Lu, A.C. Bovik, Video quaity assessment based on structura distortion measurement, Signa Processing: Image Communication 19 (2004) 121 132.