Detection and Restoration of Vertical Non-linear Scratches in Digitized Film Sequences Byoung-moon You 1, Kyung-tack Jung 2, Sang-kook Kim 2, and Doo-sung Hwang 3 1 L&Y Vision Technologies, Inc., Daejeon, Republic of Korea 2 National Archives of Korea, Seongnam, Republic of Korea 3 Department of Computer Science, Dankook University, Cheonan, Republic of Korea Abstract - Vertical scratches are common artifacts in digitized film sequences. Conventional scratch restoration methods assume that vertical scratches are linear and cover the entire height of frame. Little has studied in regard to detecting vertical non-linear scratches and using temporal relation in two consecutive frames. The proposed method makes use of line-shaped templates and projections to detect vertical linear and non-linear scratches, and in-between frames to find the continuity of the vertical scratches in two consecutive frames. Weighted median filters are implemented to restore pixels damaged by the vertical scratches. Test results with Korean historic film sequence shows effective restoration of the vertical linear and non-linear scratches. Keywords: Vertical non-linear scratch, line-shaped template, in-between frame, restoration 1 Introduction Motion picture films are susceptible to being easily scratched. Any form of mechanical viewing of film relies on the fact that the film is pulled down by sprockets engaging in perforations in the film. The scratches, caused by the mechanical rubbing of film against a projector, are narrow, bright, vertical or nearly vertical lines, lasting for several frames along the film direction. A variety of researches [1-5] have proposed vertical scratch detection and removal methods using intensity of pixels, followed by Hough transform to reduce false detection alarms. Kokoram [1] proposed a Hough transform, following vertical sub-sampling and horizontal median filtering, to detect vertically linear scratches. Shih et al. [2] subdivided the frame into horizontal bands. The intensity of pixels was projected along the vertical directions to obtain the value of histogram from each band and accumulated scratch continuations at the same horizontal position through different bands. Joyeux et al. [3] presented a method based on Radon projection along the y-axis and temporal information using all frames of the degraded sequences to minimize noise influence. Khriji et al. [5] used a horizontal segment with a fixed width to find potential pixels belong to the scratches. However, motion picture films tend to shrink (sometimes stretch) with age. If the film shrinks radically from its original shape, the film will no longer run smoothly on the projector since the distance between perforations has changed too much, causing the pull down mechanism in the projector to tear the perforations. Film perforations are also damaged by the repeated use of the film. In such cases, the vertical scratches may appear non-linear due to the improper synchronization between the sprocket and the perforation in film. This paper presents a new method to restore vertical linear and non-linear scratches based on the assumption that the vertical scratches are not always linear. The method uses line-shaped templates to detect linear segments from each frame, and in-between frames to find the continuity of the linear segments between the two consecutive frames. Vertical linear and non-linear scratches are found by projecting the linear segments in the in-between frames and measuring the length of linear segments connected by the projections. This approach was tested with Korean historic film sequence, with effective restoration of the vertical linear and non-linear scratches in the frame. 2 The Proposed Method The proposed method is divided into four steps: finding linear segments from the original frames, generating inbetween frames from two successive linear-segment frames, detecting the vertical linear and non-linear scratches from the in-between frames, and restoring the vertical scratches from the original frames. 2.1 Detection of linear segments Conventional methods define the vertical scratches as a set of continuous pixels, intensity of which is a local extreme along the horizontal direction. The works proposed in [1] and [2] use a cross-section filter to identify pixels defected by vertical scratches, suffering from false alarms due to noises such as dirt, dust and sparks. In the proposed method, the vertical scratches are considered a stream of linear segments. A linear-segment frame is created from the original frame as follows. The lineshaped segment template of size h * w is defined and moved over the original frame. Let O is the original frame, L a set of pixels in the original frame covered by the template (1).
(1) Where L j,i is the pixel in the original frame covered by the pixel (j, i) of the template. I(L j,i ) is the intensity of the pixel L j,i and I(L) the average intensity of the pixels in the set L. (2) I(L j,0) and I(L j,w+1 ) are intensities of the left and right neighbor of the pixels covered by the pixel (j, 1) and (j, w) of the template. (Fig. 1) (a) Fig. 1: Indices of the template and the original frame. The gray area represents the template. The pixels of the linear-segment frame are marked as the linear segment (for simplicity, 1 ) if the pixels of the original frame covered by the template meet the two conditions: the intensity of the pixels remains almost constant (3), and the average intensity of the pixels is brighter than that of the neighbor pixels of the template along the horizontal direction (4). where t 1 and t 2 are the thresholds to determine linear segments. The height of the template is related with the angle of vertical linear scratches to the vertical line and the degree of curvature of vertical non-linear scratches. If the height is set to 1, the proposed method is the same as the conventional method looking for a local maximum of the intensity along the horizontal direction. The width of the template decides the width of linear segments, which is the same as the scratch width. Fig. 2 (a) and show the original images with vertical linear and non-linear scratches. 2.2 Generation of in-between frames The vertical scratches, caused by the mechanical contact of film against the projector or the improper synchronization between the sprocket and the perforation in film, run for (3) (4) Fig. 2: Original frames with vertical scratches (a) linear scratch, non-linear scratches. The scratched area(white) are enhanced for display. several frames along the film direction. Scratch detection methods that use a frame-by-frame based approach does not reflect the continuous characteristic of the scratches and fail to detect the beginning and ending of the scratches. Thus, the proposed method considers the sequence as a whole and uses the in-between frames, as a scratch detection searching window, to detect disconnected scratches and the beginning and ending segments of the scratches. The in-between frames are obtained from two consecutive linear-segment frames and can be generated as many as the height (m) of the original frame (Fig. 3). The first in-between frame is the first linear-segment frame and the last in-between frame the second linear-segment frame. The other in-between frames are combinations of the two consecutive linearsegment frames. Fig. 3 shows the end segment of the vertical scratch in the previous fame, Fig. 3 (a). The frameby-frame based methods fail to find the ending part because the linear segment is too short to be considered as a defect. However, in the k th in-between frame, the ending segment becomes a part of the long vertical scratch, and is considered a defect. In Fig. 3 (d), for example, the k th in-between frame consists of the bottom (m - k) rows of the first linear-segment frame and the top k rows of the second linear-segment frame.
(c) (a) (d) Fig. 3: In-between frames (a b) two consecutive frames with linear segments, (c) the first in-between frame, (d) the k th inbetween frame, (e) the last in-between frame. (e) 2.3 Detection of vertical scratches The in-between frames are examined for detection of the vertical scratches. A horizontal profile of the linear segments is obtained by vertically projecting the in-between frame with assumption that the vertical scratches are not overlapped each other. A morphological operator, closing, is applied to the profile to fill the small gaps caused by disconnected scratches. The clusters of continuous non-zero columns in the profile are considered the horizontal location of scratches. The shapes of clusters for each type of scratches are illustrated in Fig. 4. A vertical linear scratch consists of a single column or a set of columns with the same value, and the value of each column is the same (or less but almost the same) as the height of the frame. A diagonal linear scratch makes a set of columns with the same value, but the sum of the value of these columns is the same (or less but almost the same) as the height of the frame. A vertical non-linear scratch consists of a set of columns with different values, and the sum of the values is greater than or almost the same as the height of the frame. In the case that the cluster is the diagonal linear or vertical non-linear scratches, the columns of the cluster is horizontally projected to get its vertical profile. The gapfilling processing for the vertical profile is performed again and the vertical length of the cluster is determined by finding a longest cluster of non-zero rows. If the vertical length of the cluster is greater than a threshold, the linear segments in the cluster are determined as diagonal or vertical non-linear scratches (Fig. 5) 2.4 Removal of vertical scratches A pixel of the original frame is marked as a scratched pixel if at least one pixel among corresponding pixels of the in-between frames is flagged as a defected pixel (5). (5)
where m is the height of the frame, the pixel at location (j,i) of the n th original frame, the pixel at location (j,i) of the p th in-between frame generated from the previous two original consecutive {(n-1) th, n th } frames, the pixel (j,i) of q th inbetween frame from the {n th, (n+1) th } frames. The marked pixels of the vertical scratches were restored with a weighted median filtering. Median filtering methods were used to remove the scratches in the previous works ([6] and [7]). However, the median filters use only rank-order information of the pixel value within the filter window and cause edge jitter, streaking and may remove important image details [8]. (a) (c) Fig. 4: Illustration of the types of the vertical scratches and these horizontal profiles (a) vertical linear scratch, diagonal linear scratch, (c) vertical non-linear scratch. (a) (c) Fig. 5: Profiles for a vertical non-linear scratch (a) original frame with vertical non-linear scratches, horizontal profile of the original frame, (c) vertical non-linear scratch, (d) vertical profile of the vertical non-linear scratch (c). (d) In the proposed method, a weighted median filtering uses the rank-order and the spatial information between pixels to restore the damaged pixels. A four point 1-D horizontal window is defined and weights to each point in the window are assigned in a reciprocal proportion to the distance between the damaged pixel and the points. In the filtering process, the defected pixels are excluded and only normal ones are considered in finding the median value. Example results of the median filtering and the proposed weighted median filtering are shown in Table 1, proving the weighted median filter is more sensitive to the local intensity variations than the median filter. 3 Experimental Results Experiments were performed on a subset of 1,523 frames, size of which is 740 x 480 pixels, from a Korean digitized film sequence composed of 18,832 frames. The proposed method was implemented in C++ and run on Windows 7 system with Intel core i5-2400 3.1GHz and 4GB RAM. With the 740 in-between frames and three liner-shaped template (9x1, 9x2, 9x3 pixels), the proposed method restored about 7 frames/second. To avoid any bias, Students who do not have any knowledge on image processing or picture restoration were recruited and instructed to count any noticeable vertical defects as vertical scratches. Human detected scratches provide the ground truth (the second column in the table 2). The proposed method detected 58.4% of vertical scratches without in-between frames and 63.2% of vertical scratches with in-between frames. The restoration percentages were 54.5% and 60.4% without/with in-between frames respectively. The original frames with vertical scratches, liner-segment frames overlapped on the original frames, and its restored frames are shown in Fig. 6 4 Conclusion A fully automatic detection and restoration method for the vertical linear and non-linear scratches was presented based on the line-shaped templates and in-between frames. The lineshaped template matching approach to detect linear segments was robust to impulsive noises and local brightness variation. In-between frames using temporal relations between two consecutive frames were able to detect the broken vertical scratches, and also the beginning and ending parts of the vertical scratches. Experimental results showed that the proposed method gives effective detection and restoration of the vertical scratches in the natural scenes, but have some false alarms from the scenes with complex structures, such as buildings and bookcases. The proposed method will be improved by implementing a linear segment tracking approach for the scenes with complex structures. Acknowledgement This work was supported by National Archives of Korea in 2011.
References [1] A. C. Kokaram, "Detection and Removal of Line Scratches in Degraded Motion Picture Sequences," in Proc. Signal Processing VIII : Theories and Application, Trieste, Italy, pp. 5-8, Sept. 1996. [2] T. K. Shih, L. H. Lin and W. J. Lee, Detection and Removal of Long Scratch Lines in Aged Films, in Proc. IEEE, pp. 477-480, 2006. [3] L. Joyeux, S. Boukir and B. Besserer, Film Line Scratch Removal Using Kalman Filtering and Bayesian Restoration, in Proc. WACV 2000, IEEE Workshop on the Application of Computer Vision, Palm Springs, CA, Dec. 2000. [4] E. Ardizzone, H. Dindo, O. Gambino, G. Mazzola, Scratches Removal in Digitised Aerial Photos Concerning Sicilian Territory, in Proc. 14 th International Workshop on Multimedia Communications and Services, Maribor, German, pp. 393-396, Jun. 27-30, 2007. [5] L. Khriji, M. Meribout, M. Gabbouj, Detection and Removal of Video Defects Using Rational-based Techniques, Advances in Engineering Software 36, pp. 487-495, 2005. [6] K. Chishima and K. Arakawa, A Method of Scratch Removal from Old Movie Film Using Variant Window by Hough Transform, in Proc. 2009 IEEE, pp. 1559-1563, 2009. [7] S.W. Kim and K. H. Ko, Efficient Optimization of Inpainting Scheme and Line Scratch Detection for Old Film Restoration, in Proc. PRICAI 2006, LNAI 4099, pp. 623-631, 2006. [8] L. Yin, R. Yang, M. Gabbouj, Y. Neuvo, Weighted Media Filters: a Tutorial, IEEE Trans. on Circuits and Systems-II: Analog and Digital Signal Processing, Vol. Scratched pixel(x) and its four neighbor pixels Table 1: Example Results by Median Filters Results by median Results by weighted median filtering filtering Weight window [2,3,3,2] [40, 60, X, 60, 40] [40, 60, 60, 60, 40] [40, 60, 60, 60, 40] [60, 40, X, 40, 60] [40, 60, 60, 60, 40] [60, 40, 40, 40, 60] Scratched pixels(x 1, X 2 ) and its four neighbor pixels Weight window [2,3,2,1] for X 1 Weight window [1,2,3,2] for X 2 [50, 60, X 1, X 2, 40, 40] [50, 60, 50, 50, 40,40] [50, 60, 50, 40, 40, 40] Scratch restoration without inbetween frames Number of scratches Table 2: Detection/Restoration Results Ground truth Number of detected scratches The proposed method Fully restored Partially restored Missed 1,523 889 829 60 634 % 100 58.4 54.5 3.9 41.6
(a) (d) (g) (e) (h) (c) (f) (i) Fig. 6 : The original and restored frames (a-c) original frames, (d-f) linear segment frames overlapped with the original frames, (g-i) restored frames, and (c) are two consecutive original frames and the short scratch in upper right side of the (c) frame was detected as a part of the scratch in the frame.