3D object recognition using spin-images for a humanoid stereoscopic vision system.

Transcription

1 D object ecognition using spin-images fo a humanoid steeoscopic vision system. Olivie Stasse, Sylvain Dupitie and Kazuhito Yokoi AIST/IS-CRS/STIC Joint Japanese-Fench Robotics Laboatoy (JRL Intelligent Systems Reseach Institute (IS, ational Institute of Advanced Industial Science and Technology (AIST AIST Cental, Umezono --, Tsukuba, Ibaaki, Japan {olivie.stasse,kazuhito.yokoi}@aist.go.jp Abstact This pape pesents a D object ecognition method based on spin-images fo a humanoid obot having a steeoscopic vision system. Spin-images have been poposed to seach CAD models database, and use D ange infomations. In this context, the use of a vision system is taken into account though a multi-esolution appoach. A method fo quickly computing multi-esolution and intepolating spin-images is poposed. The esults on simulation and on eal data ae given, and show the effectiveness of this method. Index Tems Spin-images, multi-esolution, D ecognition, humanoid obot. is computed, section IV details the seach pocess, finally section V pesents the simulation and the expeiments ealized with the pesented algoithm. P β α β α I. ITRODUCTIO D Mesh Discete spin image Efficient eal-time tacking exists fo collections of D views [] []. Howeve in a humanoid context, D geometical infomation is impotant because the high edundancy of such obot allows seveal kinds of D postues. Moeove if the infomation is pecise enough, it can also be used fo gasping behaviou. Recent woks on D object model building make possible a desciption based on geometical featues. Towads the design of a seach engine fo databases of CAD models, seveal D desciptos have been poposed to build signatues of D objects [], [4], [5]. The ecognition pocess poposed hee is based on spin-images poposed initially by []. The main diffeence in the conventional wok and this one lies on the tageted application and a seach scheme based on multi-esolution spin images. Moeove the computation of the multi-esolution scheme is efined and allows a fast implementation. The tageted application us a Teasue hunting behaviou on a HRP- humanoid obot [6]. This behaviou consists in two majos steps: fist building an intenal epesentation of an object unknown to the obot, second finding this object in an unknown envionment. This behaviou is useful fo a obot used in an industial envionment, o as an aid fo eldely peson. It may incementally build its knowledge of its suounding envionment and the object it has to manipulate without any a-pioi models. The time constaint is cucial, as a easonable limit has to be set on the time an end use can wait the obot to achieve its mission. Finally the method to cope with the widest set of objects should ely on a limited set of assumptions. The eminde of this pape is as follow: in section II the computation of spin images ae intoduced, section III details how the multi-esolution signatue of objects Fig.. A. Desciption Example of spin image computation. II. SPI IMAGES A spin-image can be seen as an image epesenting the distibution of the object s density view fom a paticula point []. Moe pecisely, it is assume that all the D data ae given as a mesh Mesh = V,E whee V ae the vetices and E the edges. Let s conside a vetex P V. The spin image axis ae the nomal to the point P, and a pependicula vecto to this nomal. The fome one is called β, and the latte one α. The suppot egion of a spinimage is a cylinde cented on P, and aligned aound its nomal. Fom this, each point of the model is assigned to a ing with a height along β, and a adius along α. An example of spin-images fo a dinosau model is given in Fig.. They ae two paametes of impotance while using the spin-images: the size of the ings (δα,δβ, and the boundaies of the spin-image (α max,β max. The size of the ings is simila to a esolution paamete. The limitation (α max,β max allows to impose constaints between the points chosen fo computing the spin-image P and othe points of the model P. This is paticulaly meaningful to take into account occlusion poblem. In ou implementation, two points should have less than 90 degees between thei nomals. A geate value would implies that P is occluded by some othe points while P is facing the camea. B. omal computation When computing spin-images, the nomal computation should be as less sensitive as possible to noise. This is

2 specially impotant fo vision based infomations whee the noise might be significant. Following the tests done in [7], 8 methods have been tested: gavity cente of the polynoms fomed by neighbous of each point; inetia matix; nomal aveage of each face; nomal aveage of faces fomed by neighbou points only; nomal aveage weighted by angle; nomal aveage weighted by sine and edge length ecipocal; nomal aveage weighted by aeas of adjacent tiangles; nomal aveage weighted by edge length ecipocals; nomal aveage weighted by squae oot of edge length ecipocals. Using the Stanfod Bunny model, and adding a Gaussian noise of 0 pecent fom the aveage adjacent edge, the most stable method found was the gavity cente of the polynoms fomed by the neighbous of each point. Diect image filling δα Bilinea image filling III. MULTI-RESOLUTIO One of the most impotant featue needed in ou case, is the possibility to peceive the object at diffeent distances, and thus at diffeent esolutions. This implies to build a multi-esolution signatue of the object, and to be able to compute the esolution at which the object has been peceived. In the following, the finest spin-image SI max has the highest esolution which coespond to ( δα δβ max, max, while the spin-image SI k has a esolution ( δα, δβ = (δα k k k,δβ k. A. Computing esolution of an object Image Left d Image Right Inteval analysis model ( ab ab δβ M (α,β M (α,β Gaussian model b Optical cente Optical cente ( a( b a a( b ( α, (, i β α j i β j Fig.. Two ways to fill a spin-image: (a diect way (b bilinea intepolation. C. Spin-image filling Regading the spin-image filling, Johnson popose two ways: eithe using a diect accumulation, o a bilinea intepolation. Those two methods ae depicted in Fig.. M is the pojection of a point P V. The fist solution elates M = (α,β in suface (α i,β j -(α i+,β j - (α i+,β j+ -(α i,β j+ to the point (α i,β j egadless its position in the suface. This makes the spin-image sensitive to noise. Indeed if M is close to a bounday, it will involves impotant discete modification. To solve this poblem, a bilinea intepolation allows to smooth the effect of noise by shaing the density infomation among the 4 points connected to the suface. This is achieved by computing the distance of M to those 4 points, using two paametes (a,b as depicted in Fig.. If the points ae pocessed iteatively in the following {0,,...,k,k +,... V }, then densities ae updated as follows: W i, j (k + = W i, j (k + ( a( b W i+, j (k + = W i, j (k + a( b W i, j+ (k + = W i, j (k + ( ab W i+, j+ (k + = W i, j (k + ab whee a = (α α i /δα and b = (β β j /δβ. It is staightfowad to check that fo a point M the sum of each contibution is one. In the emainde of this pape, fo sake of claity the iteation numbe is implicit. Fig.. Model induces by the suface natue of the pixels. The esolution of the peceived object depends upon the steeoscopic system capabilities, the distance between the obot and the object, and the possible sub-sampling scheme duing image pocessing. This eo may also be induced by the segmentation used to match two points in the ight and the left images, in ou case a coelation. If the pixel is consideed as a suface on the image plane, the steeoscopic vision system may be seen as a senso which peceive D volumes. Those volumes ae the intesection of the cones epesenting the sufaces on the image planes. A D epesentation is given in Fig.. They can be intepeted also as the location eo of a D point. [8] and [9] poposed an ellipsoid based appoximation of this volume, while [0] poposed a waanted bounding box using inteval analysis. Both technics show the non-lineaity of the uncetainty elated to the econstuction of a D point. Howeve fom those pevious wok, it is clea that the eo estimation, and hee the esolution, may be diffeent fo diffeent pats of the object. While computing the signatue, the esolution of the model is given by the aveage edge s length L model = E e E E of its coesponding data. The numbe of multiple esolution m pictues can be deduced fom the following elationship: B model = L model m whee B model = min{x max,y max,z max } and {X max,y max,z max } is the bounding box englobing the model. Thus in ode to extact a global esolution fom the scene, the aveage edge s length L scene is also used. The esolution is chosen in the signatue such as: min{ L scene < L model } ( B. Multi-esolution signatue The dyadic scheme consists in dividing by each dimension of the spin image between two esolutions. Using

3 the diect filling way, it is possible to compute, fom the esolution to +, the density of a point M = (i, j in SI by: W(i, j = ( j, j +W(i+, j +W(i, j+ +W(i+, j+ Using the bilinea intepolated image, the elationship between and + is not so obvious. In Fig. 4, the points fom esolution and + ae depicted. Ou goal is to find a elationship between the density W(i, j and the densities + (i+k, j+l fo k {,,0,,} and l {,,0,,}. The main question is how to shae the infomation caied by the points which will disappea. In Fig. 4 let s conside. As this point is not pesent in esolution +, its contibution has to be edistibuted to the fou adjacent points emaining at esolution. Howeve as the density of a point M depends upon its distance, if M was in Q (i, j 0, = Q + (i, j, then its contibution has aleady been patially taken into account by + (i, j, but not by + (i, j, + (i, j, and + (i, j. Fo this thee points, an offset of ( δα, δβ has to be intoduce while pocessing (i, j. We note Q (i, j the suface descibed by the points (i, j, (i+, j, (i+, j+, (i, j+. This suface can be cut in fou quadants Q (i, j l l {0,,,} as depicted in Fig. 4. Fo convenience, and following those notations, those quadants may also be divided by fou and will be noted Q (i, j l,k k {0,,, }. One can notice that the same quadant may have seveal notations depending of the efeence point used. Fo instance Q (i, j = Q (i+, j+ 0, o Q (i, j 0, = Q+ (i, j. The notation used fo the vaiables (a,b is now extended as they change accoding to the esolution. a(m,(i, j is the distance along α fom (i, j to M. b(m, (i, j is the same along β. The elationship between those vaiables fom one esolution to the next one is summaised in Tab. I. TABLE I COEFFICIETS FOR COMPUTIG THE MULTI-RESOLUTIO BILIEAR Aeas Distances ITERPOLATIO Q (i, j 0 a(m,(i, j = a(m,+ (i, j b(m, (i, j = b(m,+ (i, j Q (i, j a(m,(i, j = a(m,+ (i+, j + δα + b(m,(i, j = b(m,+ (i+, j Q (i, j a(m,(i, j = a(m,+ (i+, j+ + δα + b(m,(i, j = b(m,+ (i+, j+ + δβ + Q (i, j a(m,(i, j = a(m,+ (i, j+ b(m,(i, j = b(m,+ (i, j+ + δβ + Lemma: Let s note W(i, j (Q the contibution of the quadant Q fo the density at point (i, j of a spin image having a esolution filled by bilinea intepolation. If m { + (i+k, j+l } fo k {0,,} and l {0,,}, and Q Q 0 (i,j Y size at esolution + (i,j 0 (i,j + (i, j Q 0, Q 0, Q Q 0,0 0, X size at esolution + Point at esolution + 6 (i,j+ + (i, j (i+, j+ X size at esolution Point at esolution (i+, j (i+,j+ Q Q (i,j (i+,j Y size at esolution Fig. 4. Computing bilinea intepolated spin-images fom one esolution to the othe. m = k + l, then we have: (i, j (Q (i, j = (i+, j (Q (i+, j = (i, j+ (Q (i, j+ = (i+, j+ (Q (i, j a m a m (i,j ( a m ( b m W + δα δβ m ( b m W + δα δβ m,n ( a m b m W + δα δβ m,n,n b m 0 = W + δα δβ m 0,n ( with a m = a(m m,(i, j, b m = b( m,(i, + j, and W m = +. Finally + (i+k, j+l W(i, j = W(i, j (Q (i, j n ( n=0 Poof: We give hee a patial poof to illustate the geneal concept. Lets conside the point M Q (i, j, = Q + (i+, j+ = Q + at esolution +. The points, 5 and 7 of the spin images mesh ae consideed. The contibution povided by M to each of those points is computed as follows: + (Q + = + 5 (Q + = + 7 (Q + = + 8 (Q + = a(m, δα + ( b(m, δβ + ( a(m, δα + b(m, δβ + ( a(m, δα + ( b(m, δβ + a(m, δα + b(m, δβ +

4 ow the same point M Q + at esolution can be computed though bilinea intepolation filling. This may be witten fo (i, j : (i, j (Q+ = ( b(m, (i, j δβ ( a(m, (i, j δα Fom Tab. I, and having δα + = δα Eq. 4 can be ewitten: (i, j (Q+ = (i, j (Q (i, j, = = ( a(m, + δα + δα + ( b(m, + δα + δβ + = ( a(m, δα + ( b(m, = δβ + 4 W + 4 (Q + Using the same aguments, we can find: Thus (i, j (Q (i, j,0 = + 0,0 + +,0 + +, ,0 (i, j (Q (i, j, = +, + 4 +, (i, j (Q (i, j, = +, + 4 +, W(i, j (Q (i, j = n=0 (i, j (Q (i, j,n = + 0,0 + +,0 + +, ,0 + +, + 4 +, + +, + 4 +, = ( a m ( b m W + δα δβ m,n (7 The same aguments holds fo the othe points and poof the lemma. The multi-esolution computation of the spin images is done fist by computing the most pecise spin-image though examination of evey points. Fo each point of the spin image, fou densities coesponding to each quadant ae stoed. Fo lowe esolution images, the density is computed using the position of the point egading the quadant consideed and Eq.. (4 (5 (6 It should be stess hee that in ou cuent implementation, only the spin-images ae submit to a multi-esolution scheme. In this fist step, no sub-sampling of the mesh has been applied. Thus if the size of the spin-images decease in this pocess, the numbe of points does not. Simulato scene IV. SEARCH PROCESS Simulato scene being analysed Fig. 5. A D mesh extacted fom the Stanfod Bunny flying in the OpenHRP simulato. The scene is cut accoding to the bounding box model. The seach pocess descibed hee is based on a D mesh. This can be eithe a single view of the envionment o an incementally build epesentation. In ou cuent implementation, it is a single view povided by the steeoscopic system. In the following, it is called the scene. The scene is divided in sub-blocks. The sub-block size is given by the bounding box of the seached object as depicted in Fig. 5. On each of the sub-block the following algoithm is applied: Select the best esolution accoding to the aveage edge-length; Get the main igid tansfomation which poject the model into the scene; Check if if the model is in the scene using the peviously computed igid-tansfomation. This povides a main coelation coefficient, and the position plus oientation in the scene of the seen object. A. Selection of the best esolution Fom section III, the object esolution is the aveage edge s length in the scene. Then the esolution fo the model s spin-images is chosen accoding to Eq.. Two spin-images (p, q with the same esolution ae compaed using the following coelation function as poposed in []:. i=0 R = p i.q i i=0 p i. i=0 q i. i=0 p i ( i=0 p i.. i=0 q i ( i=0 q i R [ ; ] (8 with the numbe of non-empty points in spin-image of the scene. This coelation can be poven to be independent to the nomalisation of a spin-image. Thus duing the multiesolution phase the spin-images ae not nomalised. B. Rigid tansfomation evaluation The main igid tansfomation is obtained as follows: Some points ae andomly selected in the scene. Thei coesponding points in the model ae seached by compaing

5 Scene Model Best match Fig. 6. Matching points. Fig. 9. The Stanfod Bunny with a white noise based on the aveage edge length. thei spin-image to all the model s spin-images as depicted in Fig. 6. This gives a list L C of matching points soted by thei coelation coefficients. To emove false matching, the last 0 % elements of L C ae discaded. Fom this, a list of igid tansfomation L is extacted by consideing sets of 4 points in the list L C as depicted in Fig. 7. Matching list couples tab. Fig. 0. The Stanfod Bunny with self occlusion. Compute a igid tansfomation fo a set of 4 points. Fig. 7. Rigid tansfomation : Get a list of. Sets of 4 points used to compute igid tansfomation. Fo each igid tansfomation e L, a mak is computed by consideing all the couples of L C. If e is the eal igid tansfomation, then it should poject the maximum numbe of points fom the scene to the model. C. Final coelation coefficient On ode to veify the main igid tansfomation, points of the model ae chosen andomly and veified against the scene using the poposed main igid tansfomation. The main coelation coefficient is the aveage of the 80 % best coelation coefficients. This pocedue is applied to each of the sub-space. A. Simulation Stanfod Bunny Coelation coefficient Fig. 8. V. EXPERIMETS Stanfod Bunny Stanfod Bunny Coelation coefficient Spin images compaison examples. Dinosau The peviously descibed algoithm was tested on diffeent situations to check its efficiency. Fist, a Stanfod Bunny spin-image was tested against a spin-image of the dinosau epesented in Fig.. This intended to evaluate the coelation value against a vey diffeent spin-image. The etuned coelation was 0.9. ext, a 0 % white noise has been added to the Stanfod Bunny afte a igid tansfomation including two otations: 45 degees aound the X axis, and 90 degees aound the Y axis and no tanslation as depicted in Fig. 9. This noise was taken accoding to the aveage length of the connected edge fo a point. The etuned coelation was 0.9 and the otation evaluated to 4 degees aound X, 9 degees aound Y and aound Z. The thid case intends to simulate a single view of the complete D model, and the subsequent self-occlusion as shown in Fig. 0. The associated igid tansfomation has no otation and no tanslation. The esulting main coelation coefficient was 0.. Fom those simulation we can conclude that the seach seems to be otation invaiant, obust against noise but is sensitive to occlusion. Howeve the coelation coefficient is still highe when only patial infomations ae available, than with a complete diffeent object. This povides a good candidate fo the next view. B. OpenHRP[] simulato In this context, the HRP- humanoid obot is simulated inside a house envionment. The goal of this simulation was to ty to cope with diffeent objects pesent in the scene. In ode to discad any petubation fom the occlusion, and the multi-esolution, the model used fo the seach pocess

6 Simulato view used as efeence Simulato scene Fig.. Simulation using the OpenHRP simulato. ou tageted application. This implementation has not been optimised to take fully advantage of the newest Pentium capabilities. Moeove duing the ecognition pocess, as it has aleady been noted, if the size of the spin-images is deceasing in the multi-esolution scheme, it is not the case of the numbe of points. It has no impact on the efficiency of the ecognition as the coelation is not sensitive to this poblem. Howeve, this is clealy time consuming. Two kinds of impovement ae possible: using a compession scheme such as the Pincipal Component Analysis as poposed in [], o a Wavelet based appoach such as WaveMesh []. Chocochips: "hand made" model Chocochips: scene image Chocochips: econstucted data Fig.. Expeiment on a single view of the HRP humanoid obot was a view of the Stanfod Bunny fom the OpenHRP simulato. This model is theeafte seach inside a vitual house. The Stanfod Bunny is above a table, behind chais, and seveal objects ae pesents in the backgound, as depicted in Fig. and Fig. 5. Using the peviously descibed scheme, the model is found with a coelation coefficient close to In this context, we can conclude that the othe objects in the scene does not decease the efficiency of the seach. C. Real data The HRP- humanoid obot is equipped with a tinoptic vision system. In this paticula case, only two cameas ae used. Using a coelation method to match points between the left image and the ight image, clouds of D points ae computed using epipola geomety. The implementation is a modified vesion of the VVV system []. The object used fo this test is a box of cookies depicted in Fig..(b. Its model, hee hand-made, is epesented in Fig..(a. The econstucted mesh is displayed in Fig..(c. The ecognition pocess etuned a coelation coefficient equal to 0.4 which is simila to the esult obtained though simulation. D. Computation time To build the Stanfod Bunny model, it takes 6 minutes and 4 seconds fo 484 points. The ecognition pocess takes seconds fo a scene, using 00 spin images to compute the igid tansfomation. The ecognition pocess applied to 8 scenes takes minutes 9 seconds, using 50 spin images fo the igid tansfomation. Also ou multiesolution appoach deceased the initial esults obtained with this implementation, it is cuently not sufficient fo VI. COCLUSIO A visual seach pocess based on clouds of D points has been pesented in this pape. It elies on a multiesolution signatue using spin-images as desciptos. A fast iteative algoithm has been poposed to compute efficiently lowe esolution spin-images fom the finest one. A fist implementation and its applications to simulated and eal data have been pesented to validate the appoach. It shows the pocess obust against noise, otation invaiant, able to cope with size, and still able to povide infomation when an occlusion occus. Ou futue wok is too impove the efficiency of this implementation, and applied it in the context of a Teasue Hunting behaviou. REFERECES [] F. Juie and M. Dhome, Real time tacking of d objects : a obust appoach, Patten Recognition, vol. 5, no., pp. 7 8, 00. [] A. Ude and C. Atkeson, Pobabilistic detection and tacking at high fame ates using affine waping, in Poceedings of the Intenational Confeence on Patten Recognition, Quebec City, Canada, august 00. [] A. E. Johnson and M. Hebet, Using spin images fo efficient object ecognition in clutteed d scenes, 999. [4] M. Kazdhan, T. Funkhouse, and S. Rusinkiewicz, Rotation invaiant spheical hamonic epesentation of d shape desciptos, 00. [5] A. Fome, D. Hube, R. Kollui, T. Bülow, and J. Malik, Recognizing objects in ange data using egion and point desciptos, 004. [6] K.Kaneko, F.Kanehio, S.Kajita, H.Hiukawa, T.Kawasaki, M.Hiata, K.Akachi, and T.Isozumi, Humanoid obot hp-, in Poceedings of the 004 IEEE Intenational Confeence on Robotics & Automation, 004. [7] S. Jin, R. R. Lewis, and D. West, A compaison of algoithms fo vetex nomal computation, 00. [8] H. Hischmülle, P. R. Innocent, and J. M. Gaibaldi, Fast, unconstained camea motion estimation fom steeo without tacking and obust statistics, in 7th Intenational Confeence on Contol, Automation, Robotics and Vision, Singapoe, Decembe 00. [9]. Molton and M. Bady, Pactical stuctue and motion fom steeo, Intenational Jounal of Compute Vision, vol. 9, no., August 000. [0] B. Telle, O. Stasse, T. Ueshiba, K. Yokoi, and F. Tomita, d boundaies patial epesentation of objects using inteval analysis, in Intenational Confeence on Intelligent Robotics Systems and Systems (IROS, Sendai, Japan, 004. [] H. Hiukawa, F. Kanehio, and S. Kajita, Openhp: Open achitectue humanoid obotics platfom, in Poceedings of the Intenational Symposium of Robotics Reseach, 00. [] Y. Sumi, Y. Kawai, T.Yoshimi, and T. Tomita, d object ecognition in clutteed envionments by segment-based steeo vision, Intenational Jounal of Compute Vision, vol. 6, Januay 00. [] S. Valette and R. Post, Wavelet-based pogessive compession scheme fo tiangle meshes: Wavemesh, IEEE Tansaction on Visualization and Compute Gaphics, vol. 0, umbe, 004.