The application of image division method on automatic optical inspection of PCBA

1 1 1 1 0 The application of image division method on automatic optical inspection of PCBA Min-Chie Chiu Department of Automatic Control Engineering Chungchou Institute of Technology, Lane, Sec. 3, Shanchiao Rd. Yuanlin, Changhua 503 Taiwan, R.O.C. Long-Jye Yeh Che-Jung Hsu Department of Mechanical Engineering Tatung University Taiwan, R.O.C. Abstract Because of the vigorous growth in the electronic industry, the quantity and variety of products has risen enormously. In order to pursue profits, a strategy of cost-reduction is necessary. Automatic optical inspection (AOI) has been widely used in the inspection process of printed circuit board assembles (PCBAs); however, unrecognized deficiencies still occur. In order to increase precision in recognizing a PCBA s deficiencies by using current inspection techniques, a huge quantity of samples used in off-line training is obligatory. Unfortunately, it is not suitable for an industry which produces a variety of products with a smaller quantity. Traditional AOI methods have been investigated and substantially tested in this paper. Results reveal that many image errors which haven t been identified may raise maintenance cost of PCBAs. To overcome the above drawbacks, a new and efficient algorithm, an image division method (IDM), is proposed. Consequently, the experimental results using the IDM reveal that recognition efficiency can be improved. Keywords and phrases : Machine version, image division method, AOI, PCBA, SMT. E-mail: minchie.chiu@msa.hinet.net Journal of Information & Optimization Sciences Vol. ( ), No., pp. 1 1 c Taru Publications

M. C. CHIU, L. J. YEH AND C. J. HSU 1. Introduction 1 1 1 1 0 30 3 3 3 To improve life, many electronic products have been developed. And now, the electronic industry is playing an essential role in the world. The printed circuit board assembly (PCBA) is one of the most important components installed inside an electronic product. To promote quality and increase electronic performance, many electronic elements have been installed onto the shape-minimized board; therefore, the surface mount technology (SMT) used to fasten and assemble an element onto the printed circuit board s surface becomes important. The SMT locates the surface mounting device (SMD) on the planned printed circuit board (PCB) in which the solder is sprayed. Thereafter, the solder flows between the SMD and PCB. Traditional inspection of the PCBA is performed by visual examination followed by an electrical instrumental test; however, the inspection is time-consuming, resulting in eye fatigue and leading to an inferior product. Therefore, the automatic optical inspection (AOI), which may reduce labor expenses, improve inspection levels, and increase product quality, has been developed and widely used in the inspection process. To successfully identify the deficient elements inside a PCBA, a superior inspection algorithm is important. To improve the inspection efficiency with respect to the deficiencies, the AOI will be equipped with a different algorithm. Teoh proposed the histogram method [1] which can establish a related chart between pixel-frequency and gray-value for a specified image zone; moreover, a parameter index in calculating all the values of gray pixels within the zone will be summed up. The situation of misalignment or a missing element will then be judged by the above parameter index. In Lin s report [], the identification work will be maintained even though there is a change of light. However, the selection of the gray level will be decided in advance by the background color of the PCBA s missing element before the off-line training is performed. When the background color is similar to that of the electronic element, the recognition work by the histogram method will be inefficient. The white point statistic method is mainly used to identify the printed character. In Teoh s research [1], the above image was classified as two values (black and white) by a threshold value which is decided by a specified region with contra colors. By summing up the total number of

AUTOMATIC OPTIMAL INSPECTION OF PCBA 3 1 1 1 1 0 30 3 3 3 white points, the deficiency on the opposite side of the element can be picked up. Because of the white color on the opposite side [], the white point statistic method is superior in identifying the deficiency of that side. However, precision will be decreased when the ratios of the white point in the testing image are similar to that of the standard image (qualified image). Similarly, if there is a missing element, accuracy will also decrease. To overcome this drawback, manipulating the image zone is required. Loh [3] proposes a run-length encoding method in which the image is classified as two values (black and white) in a horizontal direction. By comparing the value between the standard and the testing image, the corresponding deficiency can be examined; however, it is not easy to recognize if there is a slightly shifted or misaligned condition. The projection method is often used to identify an inferior solder such as a solder bridge phenomena. By using a threshold value, the integrated gray value along the horizontal or vertical image is judged for the deficiency of the solder bridge. The threshold value plays an essential role which will tremendously influence precise inspection. This method is suitable for the inspection of the connector of a small outline J-lead (SOJ); however, it not suitable for both the resister and capacitor. Pern proposed the coefficient correlation method [] to recognize the deficiency in the PCBA s image. By comparing the averaged gray value and variation between standard image and testing image, calculating their relationship, and determining a threshold, the deficiency can be distinguished. The coefficient correlation method is easy to use without presetting a threshold value; in addition, precision will not be influenced by various testing images. However, it will be highly influenced when light intensity is changed or misalignment occurs. The total gray error index method is proposed by Lin []. The total error index is calculated by subtracting all the gray values of the testing image from the standard image and then summing up the absolute variation. The deficiency can be distinguished by using the above index. This method has the advantage of not influencing the various images of the PCAB too much; however, the PCBA image will be highly influenced when the light intensity is slightly changed or misalignment occurs. The gray zone division/statistic method proposed by Lin [] divides the gray zones of the standard image and testing image into five regions 0 9, 50 99, 0 19, 150 199, and 00 55. By using the bar

M. C. CHIU, L. J. YEH AND C. J. HSU 1 1 chart to analyze the number of pixels with respect to the five regions, the bar with the biggest deviation will be selected as the characterized zone. A selected threshold value is taken to evaluate the deficiency of the testing image. The gray zone division/statistic method is superior when the light intensity is slightly changed or the location of image is slightly shifted []. The high gray variation/pixel ratio method ( T1 method) is similar to the total gray error index method [5]. The threshold (T1) is calculated by dividing the total gray error by the total image points and multiplying by 1.5. By investigating the number (N1) of pixels in which the gray value is greater than T1, the new indicator used to identify the deficiency is then obtained by dividing N1 by the total image points. As investigated above, not all the deficiencies can be recognized by a single algorithm. To overcome the above drawbacks, a new and efficient algorithm an image division method (IDM) is proposed in this paper.. Nomenclature 1 1 0 f This paper is constructed on the basis of the following notations: the maximal common factor of an image s length (m) and width (n). Is(x, y) the gray value of the pixel (x, y) in the standard image. It(x, y) the gray value of the pixel (x, y) in the testing image. E n E max the total gray value s variation between standard and testing images at the nth zone. the maximum total gray value s variation between standard and testing images for all zones. µ s the mean gray value of the standard image after re-division. σ s the variance of the standard image after re-division. µ T the mean gray value of the testing image after re-division. σ T the variance of the testing image after re-division. 3. Classification of the deficiency in a PCBA Because of technical improvements for semi-conductors, many electronic elements attached to the PCBA are miniaturized; therefore, several deficiencies often exist in a completed PCBA. In order to assure product quality after the re-flow of a PCBA, the AOI is used to find deficiencies in electronic elements which are fastened onto the surface of the printed

AUTOMATIC OPTIMAL INSPECTION OF PCBA 5 circuit board. The manufacturing process of the PCBA illustrated in Figure 1 includes a PCB loader, a printing machine, a mounting machine, a re-flow, and a PCB un-loader. 1 1 Figure 1 The manufacturing process of a PCBA The general deficiencies which occasionally occurred in an AOI are classified as the following: A. Wrong element: The misplacement of an electronic element in the PCBA is possible during an incorrect assembly process which will result in a tremendous rise in cost. B. Missing element: A missing electronic element caused by collision and vibration can happen during the assembly process. This will ruin the PCBA s performance. The related deficient images after the graying process are shown in Figure. 1 Figure The deficiency of missing elements in a PCBA

M. C. CHIU, L. J. YEH AND C. J. HSU C. Misalignment: Incorrect placement of an element will happen when the machine operation is not precise. The related deficient images after the graying process are shown in Figure 3. Figure 3 The deficiency of misalignment in a PCBA D. Reverse: The influence of reverse is huge for the directional electronic elements such as capacitor and integrated circuit. E. Opposite side: If the bottom of the electronic element is turned up, it will result in an incorrect performance in the PCBA. The related deficient images are shown in Figure. 1 1 Figure The opposite deficiency in a PCBA F. No-solder: The solder for the electronic element is insufficient or terminated when the soldering process is incomplete. The related deficient images after the graying process are shown in Figure 5.

AUTOMATIC OPTIMAL INSPECTION OF PCBA 7 Figure 5 The deficiency of no solder in a PCBA G. Bridge: Over-soldering will happen when the soldering process is imperfect. This will result in an extra connection between electronic elements. The related deficiency images after the graying process are shown in Figure. Figure The bridge deficiency (solder overflow) in a PCBA. AOI index in a PCBA 1 1 1 In order to evaluate the availability of the AOI algorithm, four kinds of AOI indexes including (1) false-alarm rate, () fault-miss rate, (3) incorrect-flaw-classification rate, and () inspection time are considered. The false-alarm rate is an incorrect judgment that occurs by identifying a qualified product as an unqualified product. A higher false-alarm rate will increase a product s inspection and maintenance load.

M. C. CHIU, L. J. YEH AND C. J. HSU 1 1 1 1 0 30 3 A fault-miss rate is a misjudgment that occurs by identifying the unqualified product as the qualified product. The higher fault-miss rate will influence the quality of the product; in addition, the unqualified product which has not been picked up will be sent to the next manufacturing process which may consequently cause the product to be voided. This will lead to an increase in the cost of the product. The incorrect flaw-classification rate is the miss-classification of a deficiency. For example, a missing deficiency is regarded as a misalignment. A higher incorrect-flaw-classification rate which happens because of an inappropriate inspection algorithm will lead to misjudgments about the products deficiencies and highly influence an improvement strategy during the manufacturing and soldering process. The inspection time in an AOI system is essential. The maximum allowable time is no more than the operation time of the previous equipment. In this paper, the above AOI indexes in conjunction with various algorithms are programmed by JAVA. 5. Image division method When the ratio of the image deficiency to the full image is small enough, the gray value which is lower than that of the threshold value will lead to an incorrect recognition. In order to improve this drawback, the image division method is adopted by dividing the testing image into several regions. Subsequently, the individual image comparison for each region will be carried out to identify the deficiency by using the specified threshold value. Obviously, the required inspection time will be increased if the number of regions increases. The f, a maximum common factor of the image s length (m) and width (n), is adopted for dividing the full image. Here, the standard image and inspection image are divided as and respectively. E 1 = E = E n 1 = f f x=0 y=0 f f x= f +1 y=0 m f 1 x=(m f ) Is(x, y) It(x, y), Is(x, y) It(x, y), n 1 y=(n f ) Is(x, y) It(x, y), (1a) (1b) (1c)

AUTOMATIC OPTIMAL INSPECTION OF PCBA 9 1 1 1 1 0 E n = m 1 x=(m f ) n 1 y=(n f ) Is(x, y) It(x, y). (1d) To find the location of the primary deficiency, the maximum total variation is selected. E max = max(e 1, E, E 3,..., E n 1, E n ). () Because of the deficiency located along the edge of the region, and in order to shift the deficiency to the center of the region, information from the deficient center is required in advance. With the coordinates of the deficiency at the down/left corner and the upper/right corner [ (x min, y min ) and (x max, y max ) ], the center (x c, y c ) of the deficiency can be obtained. ( xmax + x (x c, y c ) = min, y ) max + y min, (3) where (x max, y max ) is the corresponding (x, y) that causes the maximum variation of (Is(x, y) It(x, y)) at the zone with E max and (x min, y min ) is the corresponding (x, y) that causes the minimum variation of (Is(x, y) It(x, y)) at the zone with E max After shifting the center of the specified region to the center of the primary deficiency, the mean gray values (µ s, µ T ) and variances (σs,σt ) with respect to both the standard and inspection images are calculated as µ s = 1 f f σ s = 1 f f µ T = 1 f f σ T = 1 f f y c + f / y=y c f / y c + f / y=y c f / y c + f / y=y c f / y c + f / y=y c f / x c + f / x=x c f / x c + f / x=x c f / x c + f / x=x c f / x c + f / x=x c f / Is(x, y), () [Is(x, y) µ s ], (5) It(x, y), () [It(x, y) µ T ]. (7) By using Eqs. ()-(7), a new index (I) for the IDM (image division method) is defined as I = 1 f f y c+ f / y=y c f / x c+ f / x=x c f / [Is(x, y) µ s] [It(x, y) µ T ] σs σt, 0 r 1. ()

M. C. CHIU, L. J. YEH AND C. J. HSU. Results and discussion.1 Results In this paper, two hundred inspection pictures used in a practical PCBA s inspection process have been adopted. One hundred and seventyfour pictures are qualified: ten pictures are missing a component, eight are misaligned, and eight are on the opposite side. The related images and various AOI algorithms can be obtained and assigned by the interface window programmed by JAVA run on a notebook (INTEL PENTIUM 1.5GHz&7MB RAM). The selected range of an image is 0 pixels. The flow diagram of the AOI is shown in Figure 7. 1 1 Figure 7 The flow diagram of an AOI in a PCBA As indicated in Figure 7, both the standard image and inspection image are captured. The AOI system is initialized by starting the JAVA s interface window shown in Figure. 1 Figure The start up of JAVA s interface window of an AOI system in a PCBA. As indicated in Figure 9, the standard image will be put inside the left window. The testing image will be put inside the middle window. The discrepancy between these images will be shown in the right window

AUTOMATIC OPTIMAL INSPECTION OF PCBA 11 Figure 9 Loading the images in the JAVA s interface window. To avoid the influence of light intensity, the transformation of color to a gray image shown in Figures and 11 is required Figure The transformation of color to a gray image in the JAVA s interface window of an AOI system in a PCBA (before a gray transformation). Figure 11 The transformation of color to a gray image in the JAVA s interface window of an AOI system in a PCBA (after a gray transformation) Figure 1 The selection of algorithm of an AOI algorithm

1 M. C. CHIU, L. J. YEH AND C. J. HSU Figure 13 The result of an AOI algorithm by the coefficient correlation method. Discussion After using various AOI algorithms, the related results with respect to each algorithm are listed in Tables 1- and Figures 1-19, respectively. Table 1 Recognition result for the coefficient correlation method Item Coefficient Pass Missing Opposite Misalignnumber number component side ment number number number 1 1 0.9990 3 0 0 0.999 0.990 1 1 3 0.9979 0.9970 9 5 1 1 0.999 0.990 3 1 0 5 0.9959 0.9950 1 1 1 0 <0.999 0 0 1 0 Table Recognition result for the total gray error index method Item Coefficient Pass Missing Opposite Misalignnumber number component side ment number number number 1 1 0.0 0 1 0 0 0.011 0.00 0 0 0 0 3 0.01 0.030 3 0.031 0.00 59 3 0 5 0.01 0.050 3 1 0 0 0.051 0.00 1 0 0 0 7 0.01 0.070 15 0 0 0 0.071 0.00 1 0 0 9 0.01 0.090 0 0 0 >0.091 5 0 0 0

AUTOMATIC OPTIMAL INSPECTION OF PCBA 13 1 1 Table 3 Recognition result for the gray zone division/statistic method Item Coefficient Pass Missing Opposite Misalignnumber number component side ment number number number 1 1 0.0 0 1 1 0 0.011 0.00 0 0 0 0 3 0.01 0.030 57 5 7 0.031 0.00 7 1 0 0 5 0.01 0.050 3 1 0 0 0.051 0.00 13 0 0 0 7 0.01 0.070 1 0 0 0.071 0.00 1 0 0 0 9 0.01 0.090 1 0 0 0 >0.091 5 0 0 0 Table Recognition result for the white point statistic method Item Coefficient Pass Missing Opposite Misalignnumber number component side ment number number number 1 <0.999 0 0 0.9991 1 1 0 3 1 1.001 3 1.0011 1.00 9 1 0 0 5 1.001 1.003 0 0 1.0031 1.00 7 1 0 0 7 1.001 1.005 1 0 0 0 >1.005 1 0 0 0 As indicated in Table 1 and Figure 1, most of the qualified images have index values which are larger than 0.9970 when the coefficient correlation method is used; therefore, an assumption is made that the image will be qualified when the index is larger than 0.9970. It is obvious that the coefficient correlation method has a good effect on the false-alarm rate. However, three kinds of deficiencies are diversely allocated. The effect of an flaw-classification rate is insufficient. To proceed with the AOI inspection, an algorithm selection is required. As indicated in Figure 1, the coefficient correlation method is selected. The result and mean gray value are shown in Figure 13.

1 M. C. CHIU, L. J. YEH AND C. J. HSU 1 Table 5 Recognition result for the high gray variation/pixel ratio method ( T1 method) Item Coefficient Pass Missing Opposite Misalignnumber number component side ment number number number 1 <0.1 0 1 1 0 0.1 0.11 5 3 3 3 0.11 0.1 1 0 5 0.1 0.13 13 0 0 0 5 0.13 0.1 0 0 0 0 01 0.15 50 1 0 0 7 0.15 0.1 7 1 0 >0.1 0 3 0 Table Recognition result for the IDM Item Coefficient Pass Missing Opposite Misalignnumber number component side ment number number number 1 1 0.99 5 0 1 0 0.99 0.9 15 1 3 0.979 0.97 1 0 0 1 0.99 0.9 1 1 1 5 0.959 0.95 0 1 1 0 0.99 0.9 0 1 0 0 7 0.939 0.93 0 0 0 0 0.99 0.9 0 1 0 9 0.919 0.91 0 1 0 0 <0.91 0 3 0 As indicated in Table and Figure 15, both the qualified and unqualified images are diversely distributed along the index axis when the total gray error index method is applied on an AOI. It is possible that the number of samples is insufficient. As indicated in Table 3 and Figure 1, the character of the qualified image can be roughly identified when the gray zone division/statistic method is used. However, the ability of the flaw-classification rate is insufficient because of the diverse distribution of the deficiencies on the index s axis.

AUTOMATIC OPTIMAL INSPECTION OF PCBA 15 Figure 1 The result of an AOI by the coefficient correlation method Figure 15 The result of an AOI by the total gray error index method Figure 1 The result of an AOI by the gray zone division/statistic method

1 M. C. CHIU, L. J. YEH AND C. J. HSU Figure 17 The result of an AOI by the white point statistic method Figure 1 The result of an AOI by the high gray variation/pixel ratio method (T1 method) Figure 19 The result of an AOI by the image division method

AUTOMATIC OPTIMAL INSPECTION OF PCBA 17 1 1 1 As indicated in Table and Figure 17, the distinction between the qualified and unqualified images is not very clear when using the white point statistic method. Moreover, the ability of the flaw-classification rate is insufficient because of the diverse distribution of the deficiencies on the index s axis. As indicated in Table 5 and Figure 1, the deficiency of misalignment is grouped at the index of 0.1 0.1 when using the high gray variation/pixel ratio method. However, the distinction between the qualified and unqualified images is not clear. Therefore, the effect of the false-alarm rate is insufficient. As indicated in Table and Figure 19, most of the qualified images have index values which are larger than 0.9 when the IDM is used. It is obvious that the effect of the false-alarm rate in the IDM is superior to the coefficient correlation method. Consequently, the image division method proposed in this paper promotes the effectiveness of the false-alarm rate during the AOI process. 7. Conclusion 1 0 Five kinds of traditional AOI methods the coefficient correlation method, the total gray error index method, the gray zone division/statistic method, the white point statistic method, and the high gray variation/pixel ratio method have been substantially applied in the AOI process. The results reveal that only the coefficient correlation method is effective in the false alarm rate. When a new algorithm (IDM) is adopted in the AOI process, the experiment proves that its effectiveness in the falsealarm rate is superior to the coefficient correlation method. Consequently, the IDM proposed in this paper promotes the efficiency of the false alarm rate in an AOI system. 30 3 3 References [1] E. K. Teoh, D. P. Mital, B. W. Lee and L.K. Wee, Automated visual inspection of surface mount PCBs, Industrial Electronics Society, 1th Annual Conference of IEEE, Vol. 1 (1) (1990), pp. 57 50. [] H. H. Loh and M. S. Lu, Printed circuit board inspection using image analysis, IEEE Transactions on Industry Applications, Vol. 35 () (1999), pp. 3.

1 M. C. CHIU, L. J. YEH AND C. J. HSU [3] K. Y. Pern, The PCBA s AOI on Surface Mount Element by Computer s Vision Technique, Master thesis, Chao-Tung University, Taiwan, 000. [] Y. S. Lin, The Evaluation and Comparison of Various Machine Vision Algorithm Used in AOI of PCBA Surface Mount Element, Master thesis, Tsing Hua University, Taiwan, 003. [5] D. L. Tsai, The Inspection of PCBA Surface Mount Element by Using Machine Vision, Master thesis, Tsing Hua University, Taiwan, 00. Received August, 00