3D SURFACE RECONSTRUCTION AND ANALYSIS IN AUTOMATED APPLE STEM END/CALYX IDENTIFICATION

3D SURFACE RECONSTRUCTION AND ANALYSIS IN AUTOMATED APPLE STEM END/CALYX IDENTIFICATION L. Jang, B. Zhu, X. Cheng, Y. Luo, Y. Tao ABSTRACT. Machne vson methods are wdely used n apple defect detecton and qualty gradng applcatons. Currently, D near nfrared (NIR) magng technology s used to detect apple defects based on the dfference n mage ntensty of defects from normal apple tssue. However, t s dffcult to accurately dfferentate an apple's stem end/calyx from a true defect due to ther smlar D NIR mages, whch presents a major techncal challenge to the successful applcaton of ths machne vson technology. In ths research, we used a novel two step 3D data analyss strategy to dfferentate apple stem ends/calyxes from true defects accordng to ther dfferent 3D shape nformaton. In the frst step, a D NIR magng was extended to a 3D reconstructon usng a shape from shadng (SFS) approach. After successfully obtanng 3D nformaton, a quadratc facet model was ntroduced to conduct the 3D concave shape fttng such that the dentfcaton of apple stem ends and calyxes could be acheved based on ther dfferent 3D structures. Sgnfcant mprovement n terms of the detecton rate could be obtaned based on 3D shape fttng n comparson to the tradtonal D ntensty fttng approach. Samples of the reconstructed 3D apple surface maps as well as the dentfed stem ends/calyxes were shown n the results, and an overall 90.15% detecton rate was acheved, compared to the 58.6% detecton rate of the tradtonal D ntensty fttng approach. Keywords. 3D reconstructon, Apples, Automated detecton, Caly Facet model, Near nfrared, Shape from shadng, Stem end. Machne vson technology plays an mportant role n the apple ndustry by transformng the tradtonal apple by apple vsual nspecton to automated on lne sortng and gradng. Researchers have nvestgated the propertes of dfferent lght spectra from short to long wavelengths, and employed those lght spectra nto machne vson based apple gradng and sortng systems. Some of the systems (Good Frut Growers, 1993) have been successfully used n the ndustry. Shahn et al. (00) used an X ray lne scanner to acqure X ray mages of apples. Then spatal and transform mage features were extracted and fed nto separate artfcal neural network (ANN) classfers n order to dstngush between dfferent bruse types on the apple surface. Yang and Marchant (1996) employed a charge coupled devce (CCD) monochromatc vdeo camera to capture apple mages n a lghtng chamber, followed by a floodng algorthm to coarsely segment out apple defects such as bruses, nsect btes, and scabs. Subsequently, an actve contour model was appled to refne the segmentaton n order to mprove the localzaton and sze accuracy of the detected blemshes. Leemans et al. (1999) Submtted for revew n February 008 as manuscrpt number IET 7386; approved for publcaton by the Informaton & Electrcal Technologes Dvson of ASABE n August 009. The authors are Lu Jang, ASABE Member Engneer, Doctoral Student, Bn Zhu, Former Doctoral Student, Xueme Cheng, Former Doctoral Student, Yaguang Luo, Scentst, Envronmental Mcrobal and Food Safety Lab, USDA ARS, Beltsvlle, Maryland; and Yang Tao, Professor, Bo magng and Machne Vson Laboratory, Fschell Department of Boengneerng, Unversty of Maryland, College Park, Maryland. Correspondng author: Yang Tao, Unversty of Maryland, 147 Agrculture and Anmal Scence Bldg. College Park, MD 074; phone: 301 405 1189; fax: 301 314 903; e mal: ytao@umd.edu. chose a three color CCD camera to acqure color mages of b color apples. A method to segment defects, based on a Bayesan classfcaton process, was then used. Among all the studed spectra, Brown et al. (1974) showed that n the near nfrared (NIR) mage range between 700 and 000 nm, there was less reflectance n brused areas than n unbrused areas on apples. Snce then, as an effectve, whle low cost, magng technology, NIR based approaches for apple nspecton have been ntensvely studed (Wen and Tao, 1998a; Tao and Wen, 1999; Wen and Tao, 1999; L et al., 00; Zhu et al., 007a; Zhu et al., 007c). In most machne vson based automated apple gradng and sortng systems, t s mportant to dentfy apple stemends and calyxes n apple mages because these mages often exhbt patterns and ntensty values that are smlar to defects and result n false alarms durng defect sortng. In addton, stem end/calyx dentfcaton s necessary for estmatng the apple frmness because the locaton of the stem end and calyx must be known f an effcent frmness measurement devce s to be perfected (Throop et al., 001). To solve ths problem, Wen and Tao (000) bult a dual camera magng system that ncorporated an NIR camera (700 to 1000 nm) and a mddle nfrared (MIR) camera (3.4 to 5 m) to dentfy apple stem ends and calyxes. They dscovered that unlke a tradtonal NIR camera that s senstve to stem ends, calyxes, and defects, the MIR camera was only senstve to apple stem ends and calyxes. Based on ths fact, the true defects were easly extracted by comparng the NIR and MIR mages. Although a very hgh detecton rate could be obtaned based on the aforementoned NIR/MIR system, the cost of an MIR magng devce was too hgh to be accepted by the ndustry. Unay and Gosseln (004) developed a two cascadedclassfer approach to localze stem ends and calyxes of Jonagold apples. Frst, an ANN was used to extract canddate Transactons of the ASABE Vol. 5(5): 1775-1784 009 Amercan Socety of Agrcultural and Bologcal Engneers ISSN 0001-351 1775

objects. A nearest neghbor classfer was then appled to dscrmnate stem ends and calyxes from other canddates. Penman (001) utlzed blue lnear lght sources and a standard color vdeo camera to detect apple reflecton patterns, whch were formed by lght strpes. Because the reflecton patterns were shape and orentaton dependent, t was possble to dentfy the locaton of stem end and calyx regons. Throop et al. (001) tested two conveyers for automatc apple orentaton. B rollers were used n the conveyer system to prevent the stem end/calyx from showng n the camera's feld of vew n a mechancal way. Bennedsen et al. (005) set up an expermental machne vson system to locate apple surface defects whle elmnatng other non defect dark areas. The basc dea of ther method was to rotate apples n front of the camera so that multple mages were acqured. Dark areas n these mages, whch kept the same poston relatve to the apple durng the rotaton, were consdered defects, whle other dark areas, whose postons kept changng, were classfed as non defects, such as stem ends and calyxes. Because of the smlarty n the mages of apple stemends/calyxes and defects, t s generally dffcult to dstngush them based on ther D nformaton such as shape and mage ntensty. However, apple stem ends and calyxes have specal 3D characterstcs, ncludng bowl shaped concaves. Consderable effort has been made n the area of apple 3D property analyss. Zon et al. (1995) developed a fast computerzed method to detect bruses based on magnetc resonance magng (MRI) mages of apples. Ths approach has the potental to be expanded to 3D magng and mage analyss f the computaton tme of the 3D reconstructon algorthm can be reduced such that on lne processng requrements are met. Yang (1996) used structured lghtng to detect stem ends and calyxes. A set of evenly spaced parallel lght strpes were projected onto the apple surfaces smultaneously. Generally, the strpes on convex apple surfaces had a parallel and parabolc pattern. However, when the stem ends/calyxes came nto vew, ths pattern was dsturbed, and sharp change/broken strpes were observed around concave areas. Based on such dsturbances, localzaton of stem ends/calyxes could be acheved. The lmtaton of such an approach was that when the stem end or calyx was orented n the same drecton as the strpe lght source, the deformaton of strpes was not obtanable. To compensate for the gradent reflectance of the curved surface of an apple, Wen and Tao (1998b) ntroduced a brghtness nvarant mage segmentaton method for on lne frut defect detecton; Tao (1996) developed a sphercal transform algorthm that converted a 3D mage to a D mage by means of compensatng the ntensty gradent on curved objects, such as apples, so that the ntensty dstrbuton became almost unform after the transform. A preservaton transform was also ntroduced n order to extract defects wth the ntensty below background level. Jng and Tao (1999) desgned a laser range magng system for realtme hgh resoluton 3 D shape reconstructon of mages of poultry meat. A sngle laser lne was projected onto samples, and two cameras were synchronzed by a conveyer belt and used to contnuously grab the laser profle mages. Based on the trangular relatonshp among the laser lne, camera vew angle, and sample thckness, the 3D shapes of samples were precsely rebult. Snce ths was not a sample dependent approach, t had great potental to be appled to apple 3D shape recovery. The objectve of ths research was to develop an apple 3D shape recovery/analyss based scheme for the reconstructon of 3D apple surfaces and the effcent dentfcaton of stemends and calyxes. In ths study, a novel two step 3D data analyss strategy was ntroduced to dentfy apple stem ends and calyxes. In the frst step, the Lambertan model was employed to evaluate the reflectance map of the apple surface, and then a shape from shadng method was appled to rebuld the 3D apple surface based on a smplfed human percepton model (Zhu et al., 005; Pentland, 1989). After the 3D reconstructon, a quadratc facet model was used to explan, and hence detect, the 3D concave shape of the apple stem end/calyx. METHODS IMAGE ACQUISITION AND MATERIALS The machne vson system (fg. 1) for apple nspecton conssted of a computer controlled mage acquston module and an NIR sensng system, whch was a Htach KP MI CCD monochromatc camera wth a C mount lens and 16 mm focal length and a Corron 700 nm nterference longpass flter. The wavelength range of ths system was from 700 nm to 1000 nm. Image resoluton was 1.09 mm pxel -1, the shutter speed was 1/50 s, and the onlne magng system acqured the mages at a rate of 30 frames s -1. A lghtng chamber made by Agr Tech, Inc., was used to provde unform llumnaton for the nfrared sensor. The 10 (W) 100 (L) 5 (H) cm chamber was made of lattcepatterned sheet metal, and the V shaped nteror surface of the chamber was panted flat whte to provde dffuse lght reflecton and elmnate shadows. Lghtng was provded by ten warm whte fluorescent lamps (GE SPX30 fluorescent lnear lamp wth 3 W and 110 VAC power suppl arranged unformly around a V shaped surface rght above the conveyor and used to provde unform llumnaton for the nfrared sensor (Wen and Tao, 000; Cheng et al., 003). A conveyor comprsed of rollers at a speed of 10 cm s -1 was used to keep the apples separate and to rotate each apple freely n order to make every apple sde avalable to the NIR camera durng multple exposures. A total of 03 NIR Golden Delcous apple mages were acqured, of whch 63 were samples wthout stem ends/ calyxes facng the camera, whle 140 had stem ends/calyxes showng n the mage. The sample apples were placed on the conveyer manually at random orentatons. PREPROCESSING An orgnal NIR mage of apples s shown n fgure a. The ntensty of the background vares and s relatvely darker than the apples. In order to perform the 3D reconstructon, the non unform background has to be removed. In addton, each apple s a regon of nterest (ROI) and needs to be extracted ndvdually. In ths study, based on the total of 03 NIR mages of Golden Delcous apples, a sngle threshold T 1 was used to coarsely segment apples from the background. Because some dark areas wthn apples could also be removed durng the thresholdng, a two step morphologcal operaton was then employed to refne the segmentaton (Zhu et al., 005). Frst, a hole fll operaton was performed, whch flled n the holes n the mage. A hole was defned as an area of dark pxels surrounded by lght pxels. In ths case, t referred to those removed dark areas wthn apples, such as defects, stem ends, and calyxes, that needed to be preserved 1776 TRANSACTIONS OF THE ASABE

(a) was obtaned by the aforementoned two step segmentaton. Then coordnates of the crcumscrbed rectangle of each apple could be easly determned usng the boundary nformaton. The segmented mages as well as ndvdual apple mages can be seen n fgures b and c, respectvely. (b) Fgure 1. NIR machne vson system for automatc apple sortng and gradng: (a) photograph of the system and (b) schematc representaton. for further analyss. Second, an area open operaton was performed, based on pxel counts, that removed the small foreground objects wth T pxels or less. T was a pre determned threshold, whch was set to 400 n ths research. In other words, ths operaton dscarded those brghter pxels that dd not belong to the apples but to the background. The dscarded pxels came from the false segmentaton due to the ntensty varaton of the background. A small set of ten mages was used for threshold T 1 and T tranng. Then the selected thresholds were appled through testng data set. Because the lghtng condton n the magng chamber was well controlled, mage contrast was consstently hgh from mage to mage. Therefore, a small tranng data set was suffcent to determne thresholds T 1 and T. Each apple n a sngle mage also needed to be extracted ndvdually. To acheve ths goal, the boundary of each apple ESTIMATION OF ILLUMINANT DIRECTION In ths study, the drecton of the lght source was estmated accordng to the Lambertan reflectance model and the shadng nformaton,.e., D mage ntensty (Pentland, 198; Zhu et al., 007b). Gven any partcular 8 neghborhood drecton n the mage plane, the followng relatonshp can be obtaned: di 1 dx 1 dy1 di = dx dy X M M M Y (1) di dxl dy L L where d I s the average mage ntensty change along the th drecton (dx, dy ), (X, Y) s the estmaton of the x and y components of the tlt angle (the angle that the mage plane component of the lght source vector makes wth the x axs), and L s the total number of drectons consdered. In ths research, a total of eght drectons (0, 45, 90, 135, 180, 5, 70, 315, and 360 ) were consdered. More or fewer angles could be used; eght angles were chosen by balancng the effectveness and the computaton tme. When D s defned as the drecton matrx n equaton 1, the followng equaton can be obtaned: di 1 dx 1 dy1 X T 1 T = ( D D) D di, D = dx dy () Y M M M di L dxl dyl Then, the tlt angle can be determned by: Y τ = arctan (3) X and the slant angle (the angle between the llumnant vector and the z axs) s calculated as: (a) (b) (c) Fgure. (a) Orgnal NIR mage of apples, (b) NIR mage after two step segmentaton, and (c) extracted ndvdual apple mages. Vol. 5(5): 1775-1784 1777

arccos 1 ( X σ = 0 Fgure 3. Illustraton of Lambertan model. + Y ) / K f X + Y otherwse K (4) The above facts mpled that an approach that could smulate the human eye's ablty to locate 3D shapes mght be feasble for apple stem end/calyx dentfcaton. Because Pentland's method was derved from human eye percepton propertes and was close to the way human eyes recover 3D nformaton from a D scene, t was chosen as the most feasble approach. In order to ntroduce the SFS algorthm equaton, t s useful to rewrte the lght source and surface normal vectors n the followng equaton: and S ( s, s, s ) = (cos τsn σ,sn τsn σ, cos σ) (5) = x y z ( P, Q,1) N = ( nx, ny, nz ) = (6) P + Q + 1 where E{ di } ( E{ di} ) K = and the E{ } operator s the expectaton statstcs. The relatonshp among tlt angle, slant angle, surface normal N, and lght source drecton S s schematcally llustrated n fgure 3. Notce that the lght settngs of our automatc apple magng system was always fxed and was not changed durng the onlne nspecton. Therefore, the llumnant drecton was calculated only once and could be done offlne before runnng the system. In ths study, ten unformly arranged warmwhte fluorescent lamps were used to provde a unform lghtng condton, whch meant that even the hardware confguraton was known. The calculaton of the lght source was stll necessary n order to determne the equvalent pseudo lght source drecton, and hence provde the tlt and slant angle for the shape from shadng (SFS) approach. Gven the Lambertan model as well as the lght source nformaton, the SFS method could be used to retreve the 3D shape of apples based on D data. SHAPE FROM SHADING MODEL FOR 3D APPLE SURFACE RECONSTRUCTION Shape from shadng (SFS) technques were ntroduced n the early 1970s (Horn, 1970). They are stll wdely studed by researchers (Prados et al., 00; Kmmel and Sethan, 001; Prados and Faugeras, 003; Crouzl et al., 003; Tankus et al., 004). The basc dea of ths approach s to derve a 3D scene descrpton from D nformaton, such as a D mage ntensty map. Accordng to Zhang et al. (1999), SFS algorthms can be categorzed nto four approaches: mnmzaton, propagaton, local, and lnear. Generally, mnmzaton approaches are more robust, whle the other approaches are faster. In ths research, Pentland's SFS method (Pentland, 1989), whch belongs to the lnear category, was selected for the 3D apple surface recovery. Pentland's method was chosen for the followng reasons: It gave relatvely low reconstructon errors under short computaton tme, whch s crtcal for on lne ndustral applcatons. It was dscovered that n most of our experments, gven the NIR mages of apples, the human eye could dentfy stem ends and calyxes accordng to ther 3D nformaton. (The stem ends and calyxes often had contnuously changng ntensty due to ther concave shape, whle the apple defects and other surfaces dd not). z( z( P =, Q = (7) x y Based on the Lambertan model, and Taylor seres expanson up to the frst order, the followng equaton can be obtaned: I ( = cos σ + P cos τsn σ + Q sn τsn σ (8) By takng the Fourer transform of both sdes of equaton 8, takng off the DC component, and rearrangng the equaton, t s easy to get: 1 FI Fz ( f1, f) = (9) π f1 cos τsn σ + f sn τsn σ where F z (f 1, f ) s the D Fourer transform of the apple depth map Z(, F I s the D Fourer transform of the apple mage I(, and f 1 and f are correspondng coordnates n the Fourer doman. Hence: Z x, y ) = IFT { F ( f 1, f )} (10) ( z Snce both D fast Fourer and nverse Fourer transforms are avalable, the soluton can be calculated very quckly. QUADRATIC FACET MODEL FOR STEM END AND CALYX CONVEX 3D SHAPE FITTING The dea behnd the facet model s to vew the spatal doman of an mage as the combnaton of connected surface peces, so called facets, each of whch satsfes certan shape constrants. A sloped/degree one facet model was employed to ft the test mages through least square estmaton (Haralck and Watson, 1981). The fttng results were acceptable. However, some detals were lost n the ftted mages due to only usng the frst order polynomal functon n ths study. Other 3D shape fttng approaches, ncludng hgh order facet models, have also been appled by many researchers n the area of mage processng, such as edge detecton (Haralck, 1983, 1984; J and Haralck, 00), mage segmentaton (Besl and Jan, 1988; Lukács et al., 1998), object recognton (Hebert et al., 1995; Blane et al., 000), and mage regstraton (Scott et al., 1995; Jang et al., 199; Wyngaerd and Gool, 00). In ths research, the dea of a facet model was extended to ft the 3D depth data nstead of tradtonal mage ntenstes. By dong so, much better results could be obtaned n terms of apple stem end/calyx dentfcaton. A quanttatve comparson between 3D depth fttng and mage ntensty fttng s gven n the Results and Dscusson secton. 1778 TRANSACTIONS OF THE ASABE

Because of the apple's convex shape (concave at the stemend/calyx) and ts smooth surface, t s reasonable to assume that the 3D depth value of a small neghborhood on the apple surface can be approxmated by a bvarate quadratc functon g, and the canoncal form of g can be gven by: Z( g( = k + k x + k y + k x 1 3 4 + k xy + k 5 6 y (11) The above equaton can be rewrtten based on a set of dscrete orthogonal polynomal bass: 6 g ( = Kh ( (1) = 1 where h ( = {1, y, x -, xy, y - } s a set of orthogonal polynomals, and x = {-W,..., -1, 0, 1,..., W}, y = {-W,..., -1, 0, 1,..., W} wthn the small neghborhood, where W + 1 refers to the wndow sze of the neghborhood, and W was set to n ths study. By comparng equaton 11 wth equaton 1, t s obvous that: k k 1 = K1 K 4 K 6 = K, =,3,..,6 (13) The fttng coeffcents K can be obtaned by projectng the apple 3D surface map onto the orthogonal polynomal bass: where K = y h ( Z( = Z w h ( y w = y h y h ( ( (14) (15) where s the convoluton operator. The fttng coeffcent K s computed by convolvng the 3D surface map wth the correspondng weght kernel w (J and Haralck, 00), whch makes the computatons much easer than calculatng the k values from equaton 11 drectly. Among computed fttng coeffcents, only K 4 and K 6 are needed to descrbe the shape of a quadrc surface of a gven type, whle the other coeffcents are used to control the orentaton and translaton of the surface (Besl and Jan, 1985). Some typcal quadrc shapes affected by coeffcents K 4 and K 6 are llustrated n fgure 4. In ths research, the orentaton of the facet was neglgble compared to ts shape; meanwhle, the translaton of the facet has already been taken care of durng the convoluton. Therefore, dentfcaton of the concave shape on the convex apple surface could be acheved by smply checkng coeffcents K 4 and K 6 under a pre determned threshold. K 4 and K 6 were chosen accordng to how they affect the concavty of a 3D surface. Both parameters were set to zero (a very general threshold) and kept fxed durng entre experment for 03 sample mages. Ths showed that the detecton rate of stemend/calyx dentfcaton was not very senstve to changes n K 4 > 0 and K 6 > 0 K 4 < 0 and K 6 < 0 K 4 < 0 and K 6 > 0 Fgure 4. Surface shape affected by K 4 and K 6. these coeffcents. Ths was expected, snce the dfference between 3D concave and convex shapes s qute sgnfcant. Therefore, only qualtatve analyss of the surface shape was necessary for apple stem end/calyx dentfcaton. In other words, t was only necessary to know whether the ftted shape was concave or not, and not exactly how concave the shape was. In addton, ths was a smplfed verson of 3D surface fttng, and t was easer to perform than other 3D fttng approaches. (Smplcty s always preferred by the ndustry, snce t wll save tme and hence ncrease throughput.) Smlarly, the threshold was not very vulnerable to changes n lghtng settngs as well as apple varetes, snce the dfference between an apple 3D surface and ts concave stem end/ calyx would always reman the same. However, f a lghtng change caused naccuracy of apple 3D reconstructon, then the detecton accuracy could be affected. Vol. 5(5): 1775-1784 1779

The procedure for usng the quadratc facet model for apple stem end/calyx convex 3D shape fttng s summarzed as follows: 1. Estmate the azmuth angle and the slant angle accordng to equatons 3 and 4.. Compute the 3D depth map of the orgnal NIR apple mage I accordng to equatons 9 and 10. 3. Decompose the 3D depth map of the NIR apple mage nto quadratc format accordng to equatons 11 through 14 and obtan the quadratc coeffcents K. 4. Threshold K to dstngush concave shapes on the convex apple surface n order to extract the apple stem end and calyx. Because the proposed approach dfferentated apple stemends/calyxes from defects based on ther dfferent 3D shapes, t gave a better performance than the classfcaton methods that only used D nformaton such as mage ntensty and shape. RESULTS AND DISCUSSION 3D SURFACE RECONSTRUCTION Reconstructed 3D surface maps of Golden Delcous apples usng equaton 10 are shown n fgure 5, whch ncludes fve groups (two mages per group) based on dfferent apple/mage condtons. As shown n fgure 5, 3D apple surfaces were successfully recovered from D NIR mages (shown at the top left corner of each mage). Gven nsuffcent data, such as half or part of the apple mage, the 3D map of half apples n (a) and (g) could stll be restored wthout any vsble dstorton. Dfferent 3D shape propertes among normal apple surfaces, stem ends/ calyxes, and defects can be observed from (c) to (h). Generally, the normal apple surface was a convex 3D shape, whle the defects exhbted small ndentatons, whch were much shallower and flatter n ther 3D depth than the concave shape of the stemends and calyxes. The deep concave shape of the stem ends/ calyxes made t possble for the facet model to detect ther correct postons on the apple surface. Corrupted mage data were also tested to show the robustness of the algorthm. Relatvely good results were acheved, as shown n () and (j). Once the depth value for apple stem ends/calyxes was obtaned, localzaton of the stem end/calyx could be acheved accordng to the quadratc facet model. The dentfcaton results are gven n the next secton. 3D QUADRATIC FACET FITTING Fgure 6 shows how the stem ends and calyxes can be detected by the facet fttng approach usng 3D depth data. Fg- (a) Normal half apple (b) Normal whole apple (c) Apple wth sngle defect (d) Apple wth multple defects Fgure 5. (contnued on next page) 1780 TRANSACTIONS OF THE ASABE

(e) Apple wth calyx (f) Apple wth calyx (g) Apple wth stem/calyx and defects (h) Apple wth stem/calyx and defects () Corrupted mage data (normal apple) (j) Corrupted mage data (apple wth calyx) Fgure 5. Reconstructed 3D surface maps of fve groups of Golden Delcous apples (two mages per group) based on dfferent apple/mage condtons. Note: The z axs ndcates relatve depth. Durng the reconstructon of the apple 3D surface, DC components and hgh order Taylor expanson were removed. Therefore, the unt of the z axs s actually undefned. However, ths wll not affect the detecton results snce only relatve depth of the apple 3D surface was consdered. Vol. 5(5): 1775-1784 1781

(a) (b) (c) (d) (e) Fgure 6. Quadratc facet fttng usng apple 3D depth map. ure 6a shows that several small defects on the apple, but no stem ends/calyxes, are facng the camera. Fgure 6b shows a /3 apple mage wth a calyx at the center of the apple. The calyx was located correctly by the 3D fttng model. Fgure 6c shows an apple wth many defects, and ts stem end s facng down. The fttng approach successfully detected the locaton of that stem end, whle leavng all defects unlabeled. Ths can also be seen n fgure 6d, although the defect on the bottom of the apple s large and dark. The fttng model correctly dstngushed the stem end from that defect accordng to ther dfferent 3D propertes. Fgure 6e shows a good apple sample. Unlke the tradtonal facet fttng approaches, whch usually use mage ntensty value as the fttng nput, ths study used the recovered 3D depth as the fttng data. In other words, the mage ntensty I( was substtuted by the 3D depth Z( n equaton 14. By dong so, much better results could be acheved. A comparson between these two methods s shown n fgure 7, whch shows the orgnal and processed NIR mages. The frst row shows the orgnal mages, the second row shows the stem end/calyx dentfcaton results usng ntensty data as the nput, and the thrd row represents the dentfcaton results usng 3D depth as the nput. Fgure 7a s a good apple sample, but one false alarm was generated by the tradtonal fttng method. One calyx n fgure 7b was msdentfed by the tradtonal method, and a false alarm can be found at the same tme. In addton, many apple defects were msclassfed as stem end/calyx by the tradtonal method, whch can be seen n fgures 7c and 7d. For all fve examples n fgure 7, the method presented n ths study gave the correct dentfcaton results. A total of 03 NIR Golden Delcous apple mages were tested. The detaled composton and statstcs of the data are shown n table 1. Gven three dfferent test crtera, a consstent detecton rate was acheved by the 3D depth fttng method. All detecton rates were greater than or equal to 90%. A comparson between the tradtonal fttng method and 3D fttng method s also demonstrated n fgure 8. Both type I and type II errors (Ott and Longnecker, 001) were consdered n the study to evaluate the performance of the proposed (a) (b) (c) (d) (e) Fgure 7. Comparson on fttng results usng orgnal mage ntensty (second row) and 3D depth (thrd row). 178 TRANSACTIONS OF THE ASABE

Table 1. Data composton and detecton rate of 03 test samples. No. of Samples Detected Detecton Rate (%) No. of Category Samples Samples wth stem end or 140 16 90.00 calyx facng the camera Samples wthout stem end 63 57 90.48 or calyx facng the camera Total samples 03 183 90.15 CONCLUSIONS In ths study, apple 3D surface reconstructon was acheved based on an SFS algorthm. Unlke structured lght range magng, whch uses only partal nformaton of the apple surface, ths approach took advantage of the complete mage nformaton. Every pxel value contrbuted to the reconstructed 3D map, whch meant that a more detaled 3D descrpton could be obtaned. In addton, because the camera used n ths study operated n nterlaced scannng mode, a zgzag effect around the apple boundary was nevtably generated due to the hgh conveyer speed (see fg. 1). However, ths method showed ts robustness and stll worked well regardless of ths effect. The results can be mproved f a hghresoluton, hgh speed progressve scan camera s used n the future. There was no addtonal lght source requred n the system; normal vsble whte lght plus an NIR flter was suffcent. There was also no need to mage a whole apple; just half of an apple n the camera's feld of vew could be recovered wthout any dstorton. Gven successfully recovered 3D depth data, a quadratc facet model was followed to locate the apple stem ends/calyxes based on ther 3D propertes. A total of 03 Golden Delcous apple mages were tested, and an average 90.15% detecton rate was acheved. A comparson between a tradtonal facet fttng method and the 3D depth fttng method showed that the latter approach performed much better than the tradtonal one. The method used n ths study obtaned a 9.85% total error rate compared to 41.38% when usng the tradtonal method. Ths also meant that the 3D surface model was effectve for apple stem end/calyx dentfcaton. In ths research, both the expermental results and the comparson wth tradtonal D surface fttng method showed the effectveness of the proposed 3D based approach. However, a quanttatve analyss of the accuracy for SFS 3D recovery s stll necessary n future study. Furthermore, more test samples wll be needed to evaluate the robustness of the proposed method. Some other surface reflecton models as well as 3D fttng models mght also be consdered to further explot the applcaton of apple stem end/calyx dentfcaton. Fgure 8. Comparson between 3D depth and mage ntensty fttng. approach. Type I error was calculated as the number of ncorrectly classfed samples (.e., defected apple mages) dvded by the total number of samples, whle type II error was computed as the number of false classfed samples over the total number of samples. Although the type I error of the frst method was slghtly lower than that of the second approach, a much lower type II error was obtaned by the second method. As a result, the overall error rate was reduced sgnfcantly (from 41.38% to 9.85%) by usng the 3D depth fttng method. In addton, the computaton tme of the proposed algorthm was tested accordng to all 03 apple mages, and an average of 0.04 ms per mage was acheved usng a moderate PC confguraton (Dell Dmenson 4600C), whch ncluded a Pentum 4.4 GHz CPU and 56 MB of RAM. Matlab Verson 7.0.0 was used as the programmng software. ACKNOWLEDGEMENTS The authors are grateful to Dr. Qnfen Zheng at the Unversty of Maryland for hs valuable suggestons. Thanks are also due to Ms. Abby Vogel for her crtcal readng of the manuscrpt. REFERENCES Bennedsen, B. S., D. L. Peterson, and A. Tabb. 005. Identfyng defects n mages of rotatng apples. Comput. Electron. Agrc. 48(): 9 10. Besl, P. J., and R. C. Jan. 1985. Three dmensonal object recognton. ACM Computng Surveys (CSUR) 17(1): 75 145. Besl, P. J., and R. C. Jan. 1988. Segmentaton through varable order surface fttng. IEEE Trans. PAMI 10(): 167 19. Blane, M. M., Z. Le, H. Cv, and D. B. Cooper. 000. The 3L algorthm for fttng mplct polynomal curves and surfaces to data. IEEE Trans. PAMI (3): 98 313. Brown, G. K., L. J. Segerlnd, and R. Summt. 1974. Near nfrared reflectance of brused apples. Trans. ASAE 17(1): 17 19. Cheng, X., Y. Tao, Y. R. Chen, and Y. Luo. 003. NIR/MIR dual sensor machne vson system for on lne apple stem end/calyx recognton. Trans. ASAE 46(): 551 558. Crouzl, A., X. Descombes, and J. D. Durou. 003. A multresoluton approach for shape from shadng couplng determnstc and stochastc optmzaton. IEEE Trans. PAMI 5(11): 1416 141. Good Frut Growers. 1993. MERLIN electronc color sorter. Good Frut Growers (March 15). Haralck, R. M. 1983. Rdges and valleys on dgtal mages. Comput. Vson Graph. Image Proc. : 8 38. Haralck, R. M. 1984. Dgtal step edges from zero crossng of second drectonal dervatves IEEE Trans. Pattern Analyss and Machne Intellgence 6(1): 58 68. Haralck, R. M., and L. Watson. 1981. A facet model for mage data. Comput. Graphcs Image Proc. 15: 113 19. Hebert, M., K. Ikeuch, and H. Delngette. 1995. A sphercal representaton for recognton of free form surfaces. IEEE Trans. PAMI 17(7): 681 690. Vol. 5(5): 1775-1784 1783

Horn, B. K. P. 1970. Shape from shadng: A method for obtanng the shape of a smooth opaque object from one vew. PhD dss. Cambrdge, Mass.: Massachusetts Insttute of Technology. J, Q., and R. M. Haralck. 00. Effcent facet edge detecton and quanttatve performance evaluaton. Pattern Recog. 35(3): 689 700. Jang, H., R. A. Robb, H. Tanter, and S. Kerre. 199. New approach to 3 D regstraton of multmodalty medcal mages by surface matchng. Proc. SPIE 1808: 196 13. Jng, H., and Y. Tao. 1999. Real tme hgh resoluton 3 D laser range magng of poultry products. ASAE Paper No. 99314. St. Joseph, Mch.: ASAE. Kmmel, R., and J. A. Sethan. 001. Optmal algorthm for Shape from Shadng and path plannng. J. Math. Imagng and Vson 14(3): 37 44. Leemans, V., H. Magen, and M. F. Destan. 1999. Defect segmentaton on `Jonagold' apples usng colour vson and a Bayesan classfcaton method. Comput. Electron. Agrc. 3(1): 43 53. L, Q., M. Wang, and W. Gu. 00. Computer vson based system for apple surface defect detecton. Comput. Electron. Agrc. 36( 3): 15 3. Lukács, G., R. Martn, and D. Marshall. 1998. Fathful least squares fttng of spheres, cylnders, cones, and tor for relable segmentaton. Lecture Notes n Comput. Sc. 1406: 671. Ott, L., and M. Longnecker. 001. An Introducton to Statstcal Methods and Data Analyss. 5th ed. Pacfc Grove, Cal.: Wadsworth Group. Penman, D. W. 001. Determnaton of stem and calyx locaton on apples usng automatc vsual nspecton. Comput. Electron. Agrc. 33(1): 7 18. Pentland, A. P. 198. Fndng the llumnant drecton. J. Opt. Soc. Amerca A 7(4): 448 455. Pentland, A. 1989. Shape nformaton from shadng: A theory about human percepton. Spatal Vson 4(/3): 165 18. Prados, E., and O. Faugeras. 003. Perspectve shape from shadng and vscosty solutons. Proc. 9th IEEE Intl. Conf. on Comput. Vson : 86 831. Pscataway, N.J.: IEEE. Prados, E., O. Faugeras, and E. Rouy. 00. Shape from shadng and vscosty solutons. Proc. 7th European Conf. on Comput. Vson : 790 804. Berln, Germany: Sprnger. Scott, A. M., H. Macapnlac, J. Zhang, F. Daghghan, N. Montemayor, H. Kalagan, G. Sgouros, M. C. Graham, K. Kolbert, S. D. J. Yeh, E. La, S. J. Goldsmth, and S. M. Larson. 1995. Image regstraton of SPECT and CT mages usng an external fducary band and three dmensonal surface fttng n metastatc thyrod cancer. J. Nuclear Med. 36(1): 100 103. Shahn, M. A., E. W. Tollner, R. W. McClendon, and H. R. Arabna. 00. Apple classfcaton based on surface bruses usng mage processng and neural networks. Trans. ASAE 45(5): 1619 167. Tankus, A., N. Sochen, and Y. Yeshurun. 004. Perspectve shape from shadng by fast marchng. Proc. IEEE Conference on Computer Vson and Pattern Recognton (CVPR '04) 1: 43 49. Pscataway, N.J.: IEEE. Tao, Y. 1996. Sphercal transform of frut mages for on lne defect extracton of mass objects. Opt. Eng. 35(): 344 350. Tao, Y., and Z. Wen. 1999. An adaptve sphercal mage transform for hgh speed frut defect detecton. Trans. ASAE 4(1): 41 46. Throop, J. A., D. J. Aneshansley, B. L. Upchurch, and B. Anger 001. Apple orentaton on two conveyors: Performance and predctablty based on frut shape characterstcs. Trans. ASAE 44(1): 99 109. Unay, D., and B. Gosseln. 004. An approach for recognzng stem end/calyx regons n apple qualty sortng. In Proc. Advanced Concepts for Intellgent Vson Systems Conf. (ACIVS 004). Brussels, Belgum. Wen, Z., and Y. Tao. 1998a. Fuzzy based determnaton of model and parameters of dual wavelength vson system for on lne apple sortng. Opt. Eng. 37(1): 93 99. Wen, Z., and Y. Tao. 1998b. Brghtness nvarant mage segmentaton for on lne frut defect detecton. Opt. Eng. 37(11): 948 95. Wen, Z., and Y. Tao. 1999. Buldng a rule based machne vson system for defect nspecton on apple sortng and packng lnes. Expert Sys. Appl. 16(3): 307 313. Wen, Z., and Y. Tao. 000. Dual camera NIR/MIR magng for stem end/calyx dentfcaton n apple defect sortng. Trans. ASAE 43(): 449 45. Wyngaerd, J. V., and L. V. Gool. 00. Coarse regstraton of surface patches wth local symmetres. Lecture Notes n Comput. Sc. 351: 57. Yang, Q. 1996. Apple stem and calyx dentfcaton wth machne vson. J. Agrc. Eng Res. 63(3): 9 36. Yang, Q., and J. A. Marchant. 1996. Accurate blemsh detecton wth actve contour models. Comput. Electron. Agrc. 14(1): 77 89. Zhang, R., P. S. Tsa, J. E. Cryer, and M. Shah 1999. Shape from shadng: A survey. IEEE Trans. Pattern Analyss and Machne Intellgence 1(8): 690 706. Zhu, B., L. Jang, X. Cheng, and Y. Tao. 005. 3D surface reconstructon of apples from D NIR mages. Proc. SPIE 6000: 4 51. Zhu, B., L. Jang, Y. Luo, and Y. Tao. 007a. Gabor feature based apple qualty nspecton usng kernel prncpal component analyss. J. Food Eng. 81(4): 741 749. Zhu, B., L. Jang, and Y. Tao. 007b. 3D shape enhanced transform for automatc apple stem end/calyx dentfcaton. Opt. Eng. 46(1): 1701.1 9. Zhu, B., L. Jang, and Y. Tao. 007c. 3D Surface reconstructon and analyss of apple near nfrared data for the applcaton of apple stem end/calyx dentfcaton. ASABE Paper No. 073074. St. Joseph, Mch.: ASABE. Zon, B., P. Chen, and M. J. McCarthy. 1995. Detecton of bruses n magnetc resonance mages of apples. Comput. Electron. Agrc. 13(4): 89 99. 1784 TRANSACTIONS OF THE ASABE