Automated metal surface inspection through machine vision

Transcription

1 1 Automated metal surface inspection through machine vision W-Y Wu* and C-C Hou Department of Industrial Engineering and Management, I-Shou University, Kaohsiung, Taiwan Abstract: This paper proposes a method for automated visual inspection of metal surfaces. Firstly, the modified grey-level co-occurrence matrices of metal images are used to access the information of metal surfaces. Secondly, the difference moment and the entropy of the greylevel co-occurrence matrices are extracted as the features of the metal surfaces. Finally, the features of the inspecting images are then compared with the preset confidence interval to determine whether the inspecting metal is defective or not. Some combinations of relative positions between two positioning pixels and feature descriptors were tested in the experiments to find the best one. The experimental results show that the proposed method can detect the defects effectively and has better correct detection rates than the conventional method. Keywords: machine vision, metal surface inspection, feature descriptor, co-occurrence matrix 1 INTRODUCTION 16 contrast measures were used for classification. Kassim et al. analysed images of workpiece surfaces Industrial inspection plays a very important role in to monitor machine tools [10]. Mannan et al. present achieving quality assurance. Many inspection tasks techniques for monitoring cutting tools [11]. They require substantial visual abilities and flexibility. It is used spectral analysis and wavelet decomposition known that humans are prone to making mistakes, methods to access the characteristic of tool wear. and they are slow and less consistent. The visual Okawa proposed a method for detecting two major inspection tasks have been automated owing to the types of defect appearing on the surface of cast metal: needs of 100 per cent inspection, high efficiency, lower fins and notches [12]. Five features extracted from production cost and logging defect rates and types in the histogram were used to determine whether the manufacturing processes. Many machine vision sys- inspecting metal was defective or not. Piironen et al. tems for automated visual inspection have been pre- developed a prototype for an automated visual sented in the literature [1 8]. This paper focuses on on-line metal strip inspection system [13]. They used the automated visual inspection of metal surfaces. morphological operations to enhance the defective Many methods have been proposed for automated parts. The feature data of inspecting metals were metal surface inspection, such as X-ray, histogram of classified by using a binary decision-tree classifier. images, image transformation and statistical features Suresh et al. segmented the grey-level image into an of images. edge-enhanced image [14]. After the threshold oper- Don et al. used a tree classifier to classify metal ation, a connectivity checking process was applied to samples into six classes according to their surface link the same objects. Finally, the defects could be roughness [9]. The co-occurrence matrix was used in found by comparing their extracting statistical feacomputing the metal roughness. In their experiments, tures. Wong et al. used the co-occurrence matrix and the fuzzy set theorem to detect casting surface defects The MS was received on 25 April 2002 and was accepted after automatically [15]. Xian et al. developed an illuminatrevision for publication on 8 October * Corresponding author: Department of Industrial Engineering and ing method for detecting defects in bearing rollers Management, I-Shou University, Kaohsiung, 84008, Taiwan. [16]. Laws used the texture energy approach to class- IMAG RPS 2003 Imaging Science J. Vol. 51

2 2 W-Y WU AND C-C HOU ify textures [17]. In his experiment, the texture energy approach is able to classify eight textures with 94 per cent accuracy. In common with a number of other texture segmentation schemes, the texture energy approach gives significant errors on boundaries between different textures. Grey-level co-occurrence matrices have been widely used for texture description, texture classification, and segmentation [18 20]. Co-occurrence matrices are second-order statistics, and they can describe twodimensional relations for pairs of grey levels of pixels in a digital image. They are sampled by using different relative positions of the two positioning pixels. In general, co-occurrence matrices are not used directly in Fig. 1 Plot of grey-level co-occurrence matrix (side view) classification and segmentation. The feature descriptors computed from co-occurrence matrices are often used [21, 22]. It is found that the feature descriptors of co-occurrence matrices will differ from each other Let I be a two-dimensional image with n grey levels, in distinct textures. Therefore, co-occurrence matrices and g(x, y) be the grey level of the pixel P(x, y). The play an important role in texture analysis. value of g(x, y) will range from 0 to n 1. In order This paper proposes a modified method in computto find the grey-level co-occurrence matrix of the ing grey-level co-occurrence matrices for performing image, the positioning operator (PO) must be defined. automated metal surface inspection. In the next sec- The positioning operators that define the relative postion the grey-level co-occurrence matrices will be itions between the two positioning pixels can be deterintroduced as well as the feature descriptors. The mined by two factors: angles and distances. Therefore, modified method and the defect detection procedures a positioning operator can be presented as PO= will also be demonstrated. The experimental results (h, d ), where h is the angle and d is the distance are presented in section 3. Section 4 gives some con- between the two positioning pixels respectively. cluding remarks. According to the PO, the matrix M=(m ) is i,j n n defined as follows [23]: y) g(x, y)=i, g(x+d, y)= j, Y x, y}, y) g(x, y) =i, g(x+d, y+d ) = j, Y x, y}, m =G {P(x, i,j {P(x, y) g(x, y)=i, g(x, y+d )= j, Y x, y}, {P(x, y) g(x, y) =i, g(x d, y+d ) = j, Y x, y}, if h=0 if h=45 if h=90 if h=135 (1) 2 METAL SURFACE INSPECTION METHOD where i, j=0, 1,..., n 1, and { } is the number of the set { }. 2.1 Grey level co-occurrence matrices 2.2 Modified grey-level co-occurrence matrices Grey-level co-occurrence matrices show the spatial The grey-level co-occurrence matrices present the relationships of pairs of grey levels of pixels in images. relative frequency of grey-levels of two pixels in spat- They are simple and effective texture descriptors and ial relation. In order to enhance the features of the are widely used in the texture discrimination problem. defective parts of the image, it is better to use the One of the principles of grey-level co-occurrence information both of grey levels and of positioning matrices is that the given texture patterns can effec- operators in computing the co-occurrence matrices. tively be extracted by adjusting the positioning Only the pixels with a difference in grey levels between operators [23]. A side-view plot of the grey-level the two positioning pixels that is greater than or equal co-occurrence matrix is shown in Fig. 1. to a preset value r will be accumulated in the Imaging Science J. Vol. 51 IMAG RPS 2003

3 AUTOMATED METAL SURFACE INSPECTION THROUGH MACHINE VISION 3 co-occurrence matrices. The determination of r depends on the texture feature of the inspected metal surface. Therefore, the modified matrix M =(m i,j ) n n is obtained as follows: if i j r m = i,j Gm i,j 0, otherwise (2) Furthermore, the grey-level co-occurrence matrix C= (c i,j ) n n can then be obtained by normalizing the matrix M : c = m i,j (3) i,j W W m s t s,t for i, j=0, 1,..., n 1. It is indicated in equation (2) that, if r=0, then M=M. That is, the original grey-level co-occurrence matrix can be expressed as a special case of the modified grey-level co-occurrence matrix. Furthermore c ij =0 for i j <r (4) Figures 2a and b show top-view plots of the greylevel co-occurrence matrices of perfect and defective metal surfaces respectively. The darker the region, the greater is the value. It can be seen that the defective metal surface has a more scattered grey-level co-occurrence matrix than the perfect surface. As stated in equation (4), the modified co-occurrence matrices will have values on the diagonal block. A plot with r=3 is presented in Fig. 3. From these plots it can be seen that the modified grey-level co-occurrence matrices have a better ability to reveal the difference between perfect and defective metal surfaces. Fig. 2 Top-view grey-level co-occurrence matrices: (a) perfect metal surface; (b) defective metal surface difference moment can be defined as 2.3 Feature descriptors DM= (i j)2c (5) i,j i j In most cases it is hard to compare the two images 2. Entropy (EN). The entropy represents the ranfrom their grey-level co-occurrence matrices directly. domness of elements in the matrix, and can be Instead, the feature descriptors computed from defined as co-occurrence matrices are used in texture analysis. EN= c log c (6) Two features of grey-level co-occurrence matrices i,j i,j were tested in the experiments. They are the difference i j moment and entropy [22]. 1. Difference moment (DM). The difference moment 2.4 Defect decision rule is used to represent the distribution of elements in The statistical decision method is used in defect the matrix. It will be large when the distribution decision and is shown in Fig. 4. Given a 1 a confi- of the matrix is far from the main diagonal. The dence level, the confidence interval (CI) for inspection IMAG RPS 2003 Imaging Science J. Vol. 51

4 4 W-Y WU AND C-C HOU can be established by sampling k images of perfect metals. It is defined as s CI= Cf : z a/2 k, f s :+z (7) a/2 kd where z a/2 is the z value for a standard Gaussian distribution, and f: and s are the mean and standard deviation of feature descriptors of k template images respectively. The metal surface under inspection can then be classified as perfect or defective by the following rule: Rule 1. If the feature descriptor of the inspecting image falls in the confidence interval, then it is classified as a perfect metal. Otherwise, it is said to be defective. 2.5 Defect detection algorithm It is not easy for the human eye to tell the difference between images of 64 and 256 grey levels. In order to reduce the processing time as well as the memory in computing the grey-level co-occurrence matrices, the grey level is reduced from 256 to 64. The size of the grey-level co-occurrence matrices will be reduced from to It is known that the defects of metal surfaces occur locally. In order to extract the local textures, each image is divided into 64 (=8 8) subimages. Chow and Kaneko used overlapping subimages in the adaptive thresholding approach [24]. For each subim- Fig. 3 Top-view modified grey-level co-occurrence age, if the histogram is bimodal, a threshold is selected matrices: (a) perfect metal surface; (b) defective for the subimage. Nevertheless, defects may be found metal surface on boundaries of two subimages. Therefore, the neighbouring subimages overlap by 5 pixels in each Fig. 4 Defect decision for inspecting metal surfaces by using the statistical decision method Imaging Science J. Vol. 51 IMAG RPS 2003

5 AUTOMATED METAL SURFACE INSPECTION THROUGH MACHINE VISION 5 direction in this paper. In addition, the subimages belonging to the background can be identified very easily and will not be tested in the inspection. Figure 5 shows an image of a metal surface. In order to access the information of texture, it is necessary to increase the image resolution. Figures 6a and b are enlarged images of the perfect and defective metal surface respectively. They contain about one-quarter of the metal inspected. Thus, there will be four images for each inspection. The original image has 256 grey levels. The grey level is reduced to 64 and is partitioned into 64 overlapping subimages ( Fig. 7). Two of the overlapping subimages are highlighted in Fig. 7. Overall, the proposed method for automated metal surface inspection can be summarized as consisting of the following steps (see Fig. 8): Step 0. For each subimage, find the confidence intervals of the feature descriptors from k template images. Step 1. Grab images. Step 2. Reduce grey level of images to 64. Step 3. Partition images into 64 overlapping subimages. Step 4. Find the modified grey-level co-occurrence matrices for each subimage. Step 5. Find the feature descriptors of the modified grey-level co-occurrence matrices for each subimage. Step 6. Detect defects by rule 1 for each subimage. Fig. 6 Test images of about one-quarter of the metal surface: 3 EXPERIMENTAL RESULTS (a) perfect metal surface; (b) defective metal surface The proposed method was tested on a PC with an Intel Pentium IV 1.8G microprocessor and a Matrox Meteor II frame grabber. The illuminating device was of the annular fluorescent type on account of the fact that annular fluorescence lights have better balanced output. Moreover, the lighting and focuses of the charge-coupled device (CCD) were adjusted to have the best clearness before grabbing the metal surface. By controlling the lighting direction, the defective regions could be enhanced, thereby reducing the complexity of the inspection. The tested images were in size. Each image was partitioned into 64 overlapping subimages. The grey level of images was first reduced from 256 to 64. Thus, the size of each subimage was Subimages belonging to the background could be easily identified by means of the grey levels. To reduce the processing time, only the 40 subimages belonging Fig. 5 Image of tested metal to the metal surface were inspected. Moreover, four IMAG RPS 2003 Imaging Science J. Vol. 51

6 6 W-Y WU AND C-C HOU Fig. 7 Test image with 64 grey levels and 64 overlapping subimages images were grabbed for each inspection. Therefore, 160 (=40 4) subimages were inspected for each metal. In the first experiment, four angles (0, 45, 90 and 135 ) and four distances (1, 2, 3 and 4) were tested in the experiment. Therefore, there were 16 (=4 4) combinations of positioning operators in the experiments. Three feature descriptors were used in the experiment: the difference moment, the entropy and the composite of difference moment and entropy. In the case of the composite of difference moment and entropy, if one of these two feature descriptors did not fall within the confidence interval, it was said to be defective. In addition, the confidence level was set to 95 per cent. The experimental results are listed in Table 1. From the experimental results it can be seen that the composite of difference moment and entropy has the best correct detection rates among all the feature Fig. 8 Flow chart for automated metal surface inspection Imaging Science J. Vol. 51 IMAG RPS 2003

7 AUTOMATED METAL SURFACE INSPECTION THROUGH MACHINE VISION 7 Table 1 Correct detection rates for different positioning original matrices. In order to find the best value of r for operators and feature descriptors metal surface inspection, several values are tested in the experiments. Positioning Difference Difference operator moment Entropy moment and The plots of correct detection rates for difference (h, d ) (%) (%) entropy (%) moment, entropy and composite of difference moment and entropy are shown in Figs 10 to 12 (0, 1) (0, 2) respectively. In each plot, the value of r is set from 0 (0, 3) to 10, where r=0 represents the results of the original (0, 4) grey-level co-occurrence matrices. (45, 1) From Fig. 10 it can be seen that, for the difference (45, 2) (45, 3) moment feature, the highest correct detection rates (45, 4) occur when r=5, 4, 5 and 7 for defect levels of 1/6, (90, 1) /10, 1/15 and 1/20 respectively. The correct detection (90, 2) (90, 3) rates can be improved by the modified method for all (90, 4) degrees of defect. (135, 1) For the entropy feature, the original grey-level (135, 2) co-occurrence matrix has small correct detection rates (135, 3) (135, 4) when defect levels are 1/15 and 1/20 (Fig. 11). The correct detection rates can be improved significantly by setting r>0. In addition, the modified method has better performance than the original method for all descriptors. The reason for this is that small defects defect levels, and it has the largest correct detection may be ignored when the feature descriptor is set to rates when r is 4. difference moment or entropy. A per cent detechas The composite of difference and entropy feature tion rate is achieved when the composite feature is a high correct detection rate for both the conven- applied with positioning operators (h, d ) =(90, 1). tional and modified grey-level co-occurrence matrices Furthermore, it is desirable to compare the modiimproved by the modified method especially for a (Fig. 12). However, the correct rates can also be fied grey-level co-occurrence matrices with the condefect level of 1/6. From Fig. 12 it can be seen that it ventional one. In order to evaluate the proposed method, four degrees of defects (1/6, 1/10, 1/15 and has the highest correct detection rates for all defect 1/20 of subimages) are sampled for experiments. The levels when r is 4. area of each subimage is about 0.3 cm2. Therefore, The above discussions indicate that the proposed the area of 1/6, 1/10, 1/15 and 1/20 of a subimage will method can improve correct detection rates for metal be 0.053, 0.032, and cm2 respectively. In surface inspection. In the above experiment, r=4 seems to be a good choice for the samples tested. the experiments, the positioning operator will be set However, for other samples and applications the best to (90, 1), since this had the best correct detection values of r must be found by a pretest. rate in the above experiment. In addition, the confi- Table2 gives the correct detection rates for different dence level will be set to 95 per cent again. levels of defects. For feature descriptors it can be seen As indicated before, the determination of r depends that the composite feature descriptor has the best peron the feature of the co-occurrence matrices. That is, formance for all degrees of defects. In particular, the the best value of r for the modified co-occurrence modified grey-level co-occurrence matrices signifimatrices should be determined case by case. Therefore, cantly improve the detection rates for the entropy care must be taken in setting the value of r. Figure 9 feature. When the areas of defects are as small as shows the plots of the modified grey-level co-occurrence cm2, the modified method still has per matrices obtained from Fig. 2a with different values of cent correct rates. r. By setting r>0, the differences between the perfect and the defective metal surfaces can be enhanced. However, it can be seen that a large value of r will have 4 CONCLUSIONS a large blank diagonal block. Almost all c are zero for ij r>6, as shown in Figs 9f to h. This indicates that a In this paper, a method is proposed for inspecting large value of r will lose most of the information of the metal surfaces by means of grey-level co-occurrence IMAG RPS 2003 Imaging Science J. Vol. 51

8 8 W-Y WU AND C-C HOU Fig. 9 Modified grey level co-occurrence matrices with different values of r Imaging Science J. Vol. 51 IMAG RPS 2003

9 AUTOMATED METAL SURFACE INSPECTION THROUGH MACHINE VISION 9 Fig. 10 Correct detection rates with different values of r for difference moment Table 2 Comparison between modified co-occurrence matrices and conventional matrices for different defective ratios and feature descriptors Defective ratio 1/6 1/10 1/15 1/20 (area in cm2) (0.053) (0.032) (0.021) (0.016) Grey-level co-occurrence matrices (%) Difference moment Entropy Difference moment and entropy Modified grey-level co-occurrence matrices (r=4) (%) Difference moment Entropy Difference moment and entropy Fig. 11 Correct detection rates with different values of r for entropy matrices. Two descriptors are used to extract features from grey-level co-occurrence matrices. The statistical decision approach is used to determine whether the inspected metal surface is defective or not. Furthermore, modified grey-level co-occurrence matrices for computing features of image are proposed. The difference in grey levels between two positioning pixels is considered to enhance defects in this paper. The experimental results show that it can detect defects with reliable correct detection rates. In addition, by using modified grey-level co-occurrence matrices, the method can significantly improve correct detection rates by comparison with the conventional method. ACKNOWLEDGEMENT This paper is partially supported by the National Science Council, China, under Grant NSC E REFERENCES 1 Chan, J. P. and Palmer, G. S. Machine vision applications in industry. In IEEE Colloquium on Application of Machine Vision, 1995, pp. 1/1 1/6. 2 Iivarinen, J. Heikkinen, K., Rauhamaa, J., Vuorimaa, P. and Visa, A. A defect detection scheme for web surface inspection. Int. J. Pattern Recognition and Artif. Intell., 2000, 14(6), Fig. 12 Correct detection rates with different values of r 3 Katafuchi, N., Sano, M., Ohara, S. and Okudaira, M. for composite of difference moment and entropy A method for inspecting industrial parts surface based IMAG RPS 2003 Imaging Science J. Vol. 51

10 10 W-Y WU AND C-C HOU on an optics model. Mach. Vision Applic., 2000, 12, system for hot steel slabs. IEEE Trans. Pattern Analysis and Mach. Intell., 1983, 4, Lahajnar, F., Bernard, R., Pernus, F. and Kovacic, S. 15 Wong, B. K., Elliott, M. P. and Rapley, C. W. Automatic Machine vision system for inspecting electric plates. casting surface defect recognition and classification. In Computers in Industry, 2002, 47, IEEE Colloquium on Application of Machine Vision, 5 Lee, M. F., de Silva, C. W., Croft, A. and Wu,Q.M.J. 1995, pp. 10/1 10/5. Machine vision system for curved surface inspection. 16 Xian, W., Zhang, Y., Tu, Z. and Hall, E. L. Automated Mach. Vision Applic., 2000, 12, visual inspection of the surface appearance defects 6 Newman, T. S. and Jain, A. K. A survey of automated of bearing roller. In Proceedings of 1990 IEEE visual inspection. CVGIP: Computer Vision and Image International Conference on Robotics and Automation, Understanding, 1995, 61, , Tsai, D. M. and Hsieh, C. Y. Automated surface inspecof 17 Laws, K. I. Rapid texture identification. In Proceedings SPIE Conference on Image Processing for Missile tion for directional textures. Image and Vision Computing, 1999, 18, Guidance, San Diego, 1980, pp Tsai, D. M. and Hsiao, B. Automatic surface inspection 18 Kovalev, V. and Petrou, M. Multidimensional using wavelet reconstruction. Pattern Recognition, 2001, co-occurrence matrices for object recognition and matching. CVGIP: Graphical Models and Image 34, Processing, 1996, 58, Don, H. S., Fu, K. S., Liu, C. R. and Lin. W. C. Metal 19 Oja, E. and Valkealahti, K. Co-occurrence map: quantizsurface inspection using image processing techniques. ing multidimensional texture. Pattern Recognition Lett., IEEE Trans. Syst., Man, and Cybernetics, 1984, 3, 1996, 17, Valkealahti, K. and Oja, E. Reduced multidimensional 10 Kassim, A. A., Mannan, M. A. and Jing, M. Machine co-occurrence histogram in texture classification. IEEE tool condition monitoring using workpiece surface tex- Trans. Pattern Analysis and Mach. Intell., 1983, 20, ture analysis. Mach. Vision Applic., 2000, 11, Mannan, M. A., Kassim, A. A. and Jing, M. Application 21 Gotlieb, C. C. and Kreyszig, H. E. Texture descriptors of image and sound analysis techniques to monitor the based on co-occurrence matrices. CVGIP: Graphical condition of cutting tools. Pattern Recognition Lett., Models and Image Processing, 1990, 51, , 21, Haralick, R. M., Shanmugam, K. and Dinstein, I. 12 Okawa, Y. Automatic inspection of the surface defects Textural features for image classification. IEEE Trans. of cast metal. Computer Vision, Graphics, and Image Syst., Man, and Cybernetics, 1973, 3, Processing, 1984, 25, Haralick, R. M. and Shapiro, L. G. Computer and Robot 13 Piironen, T., Silven, O., Pietikäinen, M., Laitinen, T. and Vision, 1992, Vol. 1 (Addison-Wesley, Reading, Massa- Strömmer, E. Automated visual inspection of rolled chusetts). metal surfaces. Mach. Vision Applic., 1990, 3, Chow, C. W. and Kaneko, T. Boundary detection of 14 Suresh, B. R., Fundakowski, R. A., Levitt, T. S. and radiographic images by a threshold method. In Frontiers Overland, J. E. A real-time automated visual inspection of Pattern Recognition, 1972, pp Imaging Science J. Vol. 51 IMAG RPS 2003