PRACTICAL ISSUES IN PIXEL-BASED AUTOFOCUSING FOR MACHINE VISION

Proceedigs of the 200 IEEE Iteratioal Coferece o Robotics & Automatio Seoul, Korea May 2-26, 200 PRACTICAL ISSUES IN PIXEL-BASED AUTOFOCUSING FOR MACHINE VISION Ng Kuag Cher,Nathaiel Poo Au Neow* Marcelo H. Ag Jr.* Ceter for Itelliget Products ad Maufacturig Systems Departmet of Mechaical Egieerig Natioal Uiversity of Sigapore *{mpepooa,mpeagh}@us.edu.sg Abstract Differet autofocusig methods exist for may cameras today. While ot igorig commercially available methods requirig specialized hardware, this paper focuses maily o pixel based autofocusig algorithms as applied to CCD camera systems. Differet measures of image sharpess are compared. For each of these, differet algorithms for searchig the best les settig are assessed i terms of performace as well as their applicability to various situatios. I additio, several other factors potetially affectig camera focusig are also discussed. Based o the iformatio obtaied, this research attempts to formulate a robust autofocusig algorithm. Keywords Autofocusig, focusig, machie visio, image processig.. Itroductio Focusig is a importat aspect i may applicatios ivolvig machie visio. The degree of focus i a image is a factor i determiig image quality. For example, a focused image might cotai some details ot preset i a ufocused image of the same scee. Not every applicatio requires automatic focusig. Fixed focus settigs may be used whe the depth of field is large or the camera to object distace is kow. Ofte, a camera s built-i autofocusig system is adequate for the task ivolved. However, certai applicatios may require a greater degree of cotrol for focusig to obtai a sharp image. It may be ecessary to have a focusig widow that may be selected or chaged dyamically. A good focusig system is thus required to esure the reliability of the images obtaied. Most autofocusig systems ivolve hardware built ito the camera, which may ofte be difficult to customize. Nevertheless, more flexible techiques exist, allowig a scee to be focused based solely o the iformatio obtaied from a camera s CCD array. Oly the (x, positio ad the red, gree ad blue itesities of each pixel is kow. I this paper, this type of focusig shall be referred to as pixel-based focusig. Despite the disadvatages of this techique to traditioal focusig methods, such as speed ad cost, occasios arise where the added flexibility ad degree of cotrol of pixelbased focusig make it the preferred focusig method. Pixel-based focusig ivolves several parameters. The first cosists of the focusig widow, or the regio of the scee that is to be focused. Next, a quatity idicative of the image sharpess, or a sharpess fuctio, is required. Followig this, a searchig algorithm to fid the global maximum of the sharpess fuctio must be chose. I additio, several other factors may eed to be cosidered fluctuatios i scee illumiatio, the depth of field, as well as les aberratios. I the case of color CCD cameras, the optio arises of choosig the red, gree, blue chael of a image for focusig, as well as usig either grayscale or all three chaels. This research covers the practical issues i implemetig a pixel-based autofocusig algorithm. Each of the parameters used for focusig, such as the sharpess fuctio, the searchig algorithm, or the focusig widow, may be customized for a particular applicatio. Makig appropriate decisios regardig each of these parameters requires some backgroud ad experiece. I this paper, results are preseted to eable oe to make a more iformed choice whe desigig a pixel-based autofocusig algorithm. This paper is orgaized as follows. I Sectio 2, we preset differet sharpess fuctios reported i literature as well as ew oes we itroduced. Sectio 3 provides a discussio o performace ad practical issues i implemetig the sharpess fuctios. Various search algorithms icludig ew oes are preseted i Sectio 4. I Sectio 5, we summarize our work ad provide recommedatios. 2. Sharpess Fuctios The sharpess fuctio computes the sharpess or degree of focus o a image or a regio (area) of a image. At differet les positios, the sharpess of a image chages. Autofocusig meas automatically movig the les positio such that the sharpess is maximized, i.e, image is i best focus. I literature, there are eight differet sharpess fuctios. The first two, the amplitude method ad the variace method, are quite similar. They are: 0-7803-6475-9/0/$0.00 200 IEEE 279

grey level amplitude, I( x, I, ad grey level variace, ( I( x, I ) 2 The ext two, the Teegrad ad the Laplacia are based o stadard edge detectio masks. Each of them are used with a threshold settig of zero, as suggested by Krotkov [7]. I the Teegrad method [9, 0], the Sobel horizotal ad vertical operators 0 2 =, ad i 2 0 2, x i 4 y = 0 0 0 4 0 2 are used to fid the stregth of the horizotal ad vertical gradiets. The image sharpess is the defied as ( S, for S > T ), 2 2 where S = i x + i y ad T is the threshold. The Laplacia [7] is almost idetical, except that the operator 4 L = 4 20 4 6 4 is used ad the image sharpess is ( L, for L > T ). The Fast Fourier Trasform (FFT) method [3], as its ame implies, evaluates image sharpess from the Fast Fourier Trasform of the image. I this implemetatio, the image s grey levels are placed row by row ito a D array [2], ad its FFT evaluated. Sice the FFT is evaluated usig the successive doublig algorithm [8], the array is first zeropadded before performig the FFT. From the real ad imagiary parts of each elemet of the ew array, the quatity is evaluated as the sharpess fuctio. real 2 + imagiary 2 ta imagiary real ( ) ( ) / The Sum-Modulus-Differece method [4], or SMD, simply sums up the differeces betwee adjacet pixels ad is defied as I ( x, I( x, y ) + I ( x, I( x +,. Two histogram methods, the histogram etropy [4, 9] ad the histogram of local variatios [7], work by evaluatig a quatity from the image s itesity histogram. If the itesity histogram is, h(i), where h(i) is the frequecy of pixels of itesity i, the the histogram etropy is defied as E = -Σ {h(i) l (h(i)), for h(i) 0}. I the histogram of local variatios, the itesity histogram is evaluated with pixel itesities compressed logarithmically ad the gradiet of the lie of best fit through the poits, m, is evaluated. The quatity, m, is at a miimum for the sharpest image. Sice there are 256 gray levels, i = 0 to 255, Σ{l(i+)} ad Σ {l(i+)} 2 is kow ad m may be evaluated as 256 log( i + ) h ( i ) 67.26 h ( i ). 60354. 00% 90% HLV Sharpess Fuctio (scaled from 0 to 00%) 80% 70% 60% 50% 40% 30% 20% SMD Laplacia Teegrad Etropy FFT Variac Amplitude Image Used Image Used Amplitude Variace Teegrad Laplacia FFT SMD Etropy HLV 0% 0% 0 25 50 75 00 25 50 75 200 225 250 Focusig Les Step Number (0=furthest, 255=earest) Figure : Plot of Image Sharpess (as determied by differet sharpess fuctios) vs. Les Positio 2792

Varyig the positio of the focusig les will chage the image, ad hece the image sharpess. Thus, a graph of the magitude of the image sharpess agaist the camera s les positio may be obtaied. The JAI camera was used, where 256 differet les positios could be set. For the purpose of repeatability, the image at each of the 256 les positios was saved to the hard disk before processig. To fit the eight differet sharpess fuctios to the same rage o the vertical axis as well as for compariso purposes with results by Krotkov [7], each was first scaled to the rage 0 to 00% before plottig them. The graphs are the aalyzed to determie whether the global maximum of each sharpess fuctio correspods to the focused image, as well as whether the global maximum for each sharpess fuctio agrees with the rest. Observatio is made as to whether or ot these graphs chage, if a differet color chael is used with the sharpess fuctio. Figure shows the image sharpess plots for differet sharpess fuctios usig the red chael of the frame grabber. The shape of the sharpess fuctio graph is a importat idicator to determie the ease of which the global maximum may be foud as well as to check the accuracy of the positio of best focus. The shapes of the Teegrad ad Laplacia ad SMD are particularly good for this image. The variace method also works acceptably well, its mai problem beig its low sigal to oise ratio. The image of best focus was determied as frame #72 (les positio 72) by the amplitude ad variace method. The Teegrad determied frame #73 as the sharpest image while for the Laplacia ad summodulus-differece fuctios, the frame was #69. The histogram etropy method obtaied frame #64 while the histogram of local variatios icorrectly obtaied frame #45. The depth of field as determied by the size of the flat regio of each of the sharpess fuctios was from frame #64 to frame #83. Visual ispectio reveals virtually o differece betwee the frames i this rage. All the sharpess fuctios, except the HLV were able to fid a image ear the poit of best focus without too much difficulty. The results differ from the results obtaied by Krotkov, whereby the Laplacia method failed o a high cotrast cross as well as o text. I additio, the etropy method could ot accurately fid a focus o text. From these results, as well as those by Krotkov, the best fuctios appear to be the variace ad the Teegrad, followed by the SMD. For this reaso, oly the first two fuctios, the variace ad the Teegrad were chose for obtaiig depth maps later. Teegrad Sharpess Fuctio (scaled from 0 to 00%) was observed betwee the differet plots for the three sharpess fuctios tested, the variace, Teegrad ad histogram etropy methods. Figure 2 shows the plot for the Teegrad fuctio usig the 3 differet color chaels. I the tests, the global maximum did ot differ much betwee each colour chael. 3.0 Performace Issues The speed of focusig depeds partly o the speed of the camera s focusig motor ad partly o the speed of the focusig algorithm. I terms of speed or performace, greyscale focusig may be doe at higher frame rates tha colour focusig, icludig colour focusig based o oly oe chael, because the hardware grabbig for a colour image takes loger tha a greyscale grab. 00% 90% 80% 70% 60% 50% 40% 30% 20% 0% blue blue gree gree 0% 0 50 00 50 200 250 Focusig Les Step Number (0=furthest to 255=earest) Figure 2: Differece i Usig Differet Color Chaels for Focusig for Teegrad Sharpess Fuctio If the greyscale chael is used for focusig, the sharpess fuctio may ot be able to detect a edge betwee, say a red patch ad a gree patch of the same itesity. Similarly, if oly the gree chael is used, as suggested above, a image cosistig predomiatly of red, blue or mageta (a mixture of red ad blue), but little gree may have isufficiet cotrast for focusig. Ideally, all three chaels of red, gree ad blue should be used ad combied i some way (for example, usig a weighted average). However, the tripled icrease i processig time is a cosiderable price to pay for this extra accuracy. Nevertheless, there may sometimes be applicatios whereby accuracy is much more importat tha processig time ad this method may fid its use there. I most other situatios, greyscale focusig should be adequate. red blue The same procedure was repeated to obtai plots for the gree ad blue chaels. No sigificat differece i patter 2793

Teegrad Sharpess Fuctio Of the differet sharpess fuctios, the fastest methods are the variace methods ad the histogram methods, averagig about 20ms to process a 768 x 576 focusig widow. O the slower ed are the Fast Fourier Trasform, the SMD ad the Teegrad, clockig 2500, 65 ad 68 millisecods respectively. The timigs are for a Petium II-300 machie. Although the Fast Fourier Trasform method provides a good way of obtaiig the degree of defocus, it is geerally too slow for use i focusig, eve with the successive doublig algorithm. A -D trasform that oly icludes a row or colum of pixels may accelerate this, but this will oly allow focusig o a D regio of the scee, rather tha a twodimesioal regio. I additio, it is useful to have a larger focusig widow as it is easier to esure that the object of iterest falls etirely withi the widow. Large widow sizes allow focusig uder small movemet of the object i questio as well as camera vibratio. The mai problem with the Fast Fourier Trasform techique is its slow speed of about 2.5 secods or 0.4 Hz usig a 768 x 576 focusig widow. The image grabbed i ay sigle frame will differ slightly from a subsequet grab due to oise ad small chages i scee illumiatio. To test the degree to which this affects each sharpess fuctio, aother 256 images of the same scee were grabbed cosecutively ad saved to the hard disk. Next, the sharpess of each image was determied usig each sharpess fuctio. The results show that for the same scee take at differet times, there is a sigificat variatio i the sharpess fuctio. Figure 3 shows the variatio of the Teegrad sharpess fuctio for the same scee take at differet iteratios. It has ot bee tested whether the amout of oise i each fuctio is idepedet of the scee captured or the scee illumiatio. 550 500 450 400 Red Gree 350 0 50 00 50 200 250 iteratio umber Blue Figure 3: Variatio of Teegrad Fuctio with Time. The values of the sharpess fuctios are ot costat with time, eve without movig the camera s les. This is because the pixels i the image fluctuate with time. Observatio of a sigle pixel ear the ceter of the image revealed a variatio i itesity of about ± 7 to ± 8 for each color chael. To test whether this effect was etirely due to scee illumiatio, a les cap was used to cover the camera les. However, this did ot elimiate the oise totally the variatio i itesity was reduced to about ± 3 to ± 4. The light source is i this case was the room s fluorescet lightig, powered by a 50Hz AC supply. Roughly half the oise may thus be attributed to scee illumiatio ad the other half due to oise i the aalog PAL sigal. 4. Searchig Algorithms The searchig algorithms suggested by Krotkov ad the Matrox Imagig Library were evaluated. This sectio describes the global search method ad the Fiboacci search method ad the goes o to itroduce two more search methods the search by percetage drop ad oe still image techique. I additio, two search refiemet methods are itroduced, the search by cetre of area ad the search by pulsig. The problems ecoutered with the developmet of these searchig methods led to some additioal safety checks to make the algorithm more robust. I the global search method, each les positio is scaed ad the sharpess of its image calculated. After all les positios have bee scaed, the searchig algorithm attempts to move the les back to the positio where the best image was obtaied. Krotkov recommeds the Fiboacci search techique [] as the optimal search strategy [6, 7]. This strategy is based o cotiuously arrowig the search regio by subdividig it accordig to the Fiboacci sequece. The required umber of iteratios for this search is the least iteger N, such that FN ³ the iitial search iterval. For 256 differet les positios, the first Fiboacci umber just exceedig 256 is F3 = 377, so this search will require at most 3 steps to fid the focused image. However, this search techique ca oly be implemeted o a camera for which the focusig les may be cotrolled by specifyig its positio. Moreover, i the case of the camera system used, the les motor moves too slowly for the Fiboacci search to work well. The Matrox Imagig Library i versio 6.0 provides a smart search techique that repeatedly halves the search regio ito smaller ad smaller portios. I additio, a global search techique is also supported. The mai advatage of the global search method is that it geerally guaratees that a local maximum will ot be mistake for a global maximum. The problem with this fuctio-based method is that the sharpess fuctio is affected by oise i the image. Thus, eve if the les were to retur to the exact positio correspodig to maximum image sharpess, the sharpess fuctio would ot retur the same value. Thus, some allowable degree of error must be allowed i the sharpess fuctio, meaig that the les will be close to, but ot at the focus positio.. 2794

We itroduce the Searchig by Percetage Drop as a modificatio of the global search, where ot all les positios are scaed. Rather, the les positios are scaed util a percetage drop by a predetermied amout is detected. To calculate the percetage drop i the sharpess fuctio, the formula used was ot (fmax-f)/fmax, (where f is the value of the sharpess fuctio), sice some sharpess fuctios have rages which do ot start at zero, (e.g. 7.5 to 8.5 or 400 to 700). The formula was modified to (f-fmi)/(fmax fmi) i order to accout for this. However, it should be oted that edge-detectio based fuctios, amely the Teegrad ad the Laplacia, do ot suffer from this problem sice fmax >> fmi. I the search by percetage drop, several parameters must be passed to the fuctio the oise amplitude i the image, the iitial directio of search, the magitude of the percetage drop to look for, the miimum umber of steps before startig to search for the drop, as well as the criteria for determiig whether the les has reached its miimum or maximum positio. This techique aims to improve upo the global search techique by reducig the distace for which the les motor must move. The speed of this techique is determied i part by the allowed drop i the sharpess fuctio. Settig too low a allowed drop might cause the searchig algorithm to be caught i a local maximum. Too high a drop would result i a large overshoot. I additio, if the sharpess fuctio does ot decrease by a large eough amout after hittig the maximum, the techique will fail. I additio, there is some difficulty i determiig whe the les has overshot its limit. This algorithm is also subject to the accuracy of the several parameters it is supplied with the oise amplitude, the magitude of the percetage drop, ad the coditio to determie whe the les has reached its maximum positio. We itroduce the Searchig by Cetre of Area method as a way to further refie a search. Whe the poit of best focus is ear to beig foud, a baselie is specified ad cetre of area method uses the regio of the sharpess fuctio above this baselie for its calculatio. The locatio of the cetre of area of this regio is i geeral, ot the maximum poit of the sharpess fuctio. However, due to the shape of most sharpess fuctios, the curve is almost flat at the top possibly due to the depth of field. For this reaso, the cetre of area is a good estimate for a locatio that will fall withi the depth of field of the camera. Aother possible method is to fid the average of the two limits of the depth of field. However, this estimate is ot as good as the cetre of area method i a typical sharpess fuctio as show i Figure 4. Image Sharpess Estimated Depth of Field Limit # Estimate obtaied from Dept of Field Limits Poit of Best Focus Estimated Depth of Field Limit #2 Estimate obtaied from Cetre of Area Focus Motor Positio Figure 4: Cetre of Area Estimate for Poit of Best Focus Some iformatio that must be passed to this fuctio iclude the baselie value, the oise amplitude of the sharpess fuctio ad the iitial directio of search. It should be oted also that if the oise is reduced by smoothig the curve i Figure 5- above, the poit of best focus as measured by the smoothed curve would almost coicide with the estimate obtaied by the cetre of area. I geeral, the accuracy of this method will deped o how close the value of the base is to the maximum of the fuctio. This method works best after aother search method has already foud the approximate locatio of the maximum poit. This method works as a good alterative to fidig the miimum ad maximum positios of the depth of field ad takig the average. This is because fidig the miimum ad maximum positios is quite differet as there may be several poits where the sharpess fuctio crosses the baselie. Noise may further decrease the accuracy of determiig the miimum ad maximum positios. Oe problem with this method is that if the sharpess fuctio ever drops below the baselie specified, the search method will fail. This problem is oly likely to occur if the focus positio is ot ear eough. I additio, the assumptio that the cetre of area is ear to the fuctio maximum is oly true whe the focusig positio is ear. For these two reasos, the search by the cetre of area is recommeded oly as a search refiemet method. 5. Coclusios Based o the good performace of the variace ad Teegrad methods i determiig image sharpess, ot oly i these tests, but also i the results by Krotkov, these two methods have bee determied to work well i the crae scee aalysed. The implemetatio of these two methods i the differet search algorithms was successful, while i the crae scee, these two methods failed where the image was uiform. 2795

Of the search algorithms tested, the global search ad the search for the percetage drop were foud to be the fastest iitial search methods. The other two methods, the cetre of area method ad the pulsig method were foud to be more suited for refiig a search whe the focusig poit is ear to beig foud. As the graphs comparig the effect of colour o image sharpess show, the wavelegth of light apparetly does ot play a sigificat role i affectig the positio of best focus for the crae scee tested. Focusig based o oly oe colour chael, red, gree, blue, or grey is likely to be sufficiet for most situatios. The best performace gai comes from usig greyscale values for focusig as well as smaller focusig widows. Despite the differeces i speed of the various sharpess fuctios, this effect is small compared to image grab time for all the sharpess fuctios except the Fast Fourier Trasform. I implemetig a camera system with pixel-based autofocusig, the followig may be take ito cosideratio. Firstly, oe must ask whether focusig is required. If the depth of field is large, a fixed focus settig may be sufficiet. I additio, the built-i autofocus of the camera is ofte adequate. I terms of sharpess fuctios, the variace method ad the Teegrad have bee foud to be adequately suited for the task, based o the results of this project ad the results obtaied by Krotkov. The combiatio of the three searchig algorithms the search for percetage drop method as the iitial search algorithm, with the search for the cetre of area, followed by the pulse search method for further refiemet, appears to be sufficiet to hadle most situatios. However, some tweakig of the criteria to determie whe the camera les has reached its limits is ecessary to improve the robustess of the search for percetage drop method so that it ca hadle the situatio where the iitial les positio is ukow. For the purposes of focusig, the camera with les positio feedback is highly recommeded, due to its combiatio of speed, repeatability, as well as its ability to give positio feedback. I order to further reduce the oise i the sharpess fuctio, a camera followig the IEEE394 digital stadard may be used as more become available. Ackowledgmets The support of Sigapore s Natioal Sciece ad Techology Board, Jurog Shipyard ad Sembawag Shipyard is gratefully ackowledged. Autofocusig is used i our pipe measuremet project for these two shipyards. Refereces. Beveridge, G. S., Schechter, R. S. Optimizatio: Theory ad Practice, McGraw-Hill, New York. 970. 2. Davies, E.R. Machie Visio: Theory, Algorithms, Practicalities. Academic Press Limited. 990. 3. Hor, B. K. P., "Focusig," Techical Report AIM-60, Massachusetts Istitute of Techology, May, 968. 4. Jarvis, R. A., "Focus Optimisatio Criteria for Computer Image Processig," Microscope 24(2), 976, pp. 63-80. 5. Johso, S.M. Optimal Search for a Maximum is Fiboaccia, Techical Report P-856, RAND, Sata Moica, Califoria, 956. 6. Kiefer, J., Sequetial Miimax Search for a Maximum, Proc. Am. Math. Soc.(4), pp. 502-506. 953. 7. Krotkov, Eric Paul. Active Computer Visio by Cooperative Focus ad Stereo. New York: Spriger- Verlag. 989. 8. Press, W. H., Teukolsky, S. A., Vetterlig, W. T., Flaery, B. P. Numerical Recipes i C: The Art of Scietific Computig. Cambridge Uiversity Press. 2d Editio. 992. pp. 504-508. 9. Schlag, J. F., A. C. Saderso, C. P. Neuma, ad F. C. Wimberly, Implemetatio of Automatic Focusig Algorithms for a Computer Visio System with Camera Cotrol, Techical Report CMU-RI-TR-83-4, Caregie Mello Uiversity, August, 983. 0. Teebaum, J. M., Accomodatio i Computer Visio, Ph.D. Dissertatio, Staford Uiversity. November, 970. Cotiuous focusig is difficult to achieve usig the curretly available hardware. The sharpess fuctio is oly a idicatio of the degree of defocus, but it does ot provide iformatio as to where to move the camera les. The hardware focusig method by Petax shows that at least two images are required to kow the directio i which the les should be moved. 2796