A Naural Feaure-Based 3D Objec Tracking Mehod for Wearable Augmened Realiy Takashi Okuma Columbia Universiy / AIST Email: okuma@cs.columbia.edu Takeshi Kuraa Universiy of Washingon / AIST Email: kuraa@ieee.org Kasuhiko Sakaue AIST Email: k.sakaue@ais.go.jp Absrac- In his paper, we describe a novel naural feaure based 3-D objec racking mehod. Our mehod deermines geomeric relaion beween known 3-D objecs and a camera, no using fiducial markers. Since our mehod only uses a camera o deermine his geomeric relaion, i is suiable for wearable augmened realiy (AR) sysems. Our mehod combines wo differen ypes of approaches for racking: a boom up approach (BUA) and a op down approach (TDA). We mainly use a BUA, because i acquires accurae resuls wih small calculaion cos. When BUA canno oupu an accurae resul, our mehod sars TDA o avoid misracking. An experimenal resul shows an accuracy and inegriy of our mehod. I. INTRODUCTION Tracking mehods are one of he mos imporan issues in he field of Augmened Realiy (AR). AR sysems overlay virual objecs ono he real world o help heir user do an aciviy in he real world. In many cases, AR sysems need o know accurae geomeric relaions beween real objecs and users viewing posiion o locae virual objecs ono suiable posiion of he real world. Tracking mehods deermine hese geomeric relaions. Various racking mehods have been developed in he field of AR [1][], and i is imporan o selec he appropriae racking mehod for he applicaion requiremens. For some applicaion sysems ha have cameras in heir configuraion, like a video see-hrough AR sysem or a wearable AR sysem, a vision-based racking mehod is appropriae. Vision-based racking mehods use cameras ha capure real scene as sensing devices. Some of hem don need o se any sensing devices in he environmen unlike ulra-sonic or magneic rackers do. Video see-hrough AR sysems have one or wo cameras, and capure real image sequences o be used as background for synheic AR images [3][4][5][6]. Therefore, video see-hrough AR sysems can use a vision-based racking mehod wihou changing heir sysem configuraion. We have been developing wearable AR sysems ha have video see-hrough configuraion and use a vision-based racking mehod [7][8]. We believe our design could give a pracical plaform for AR applicaions o consumer. In hese sysems, he camera posiion corresponds o he user s viewing posiion. In his case, he geomeric relaions ha are required by hese AR sysems are called exernal camera parameers. Vision-based racking mehods deermine hese geomeric relaions by maching a known model and image sequences. Figure 1 : Oupu sills of a 3-D online manual. However, i is difficul o mach capured image sequences and known models ha have general shape and color because of image noise, occlusions of he objec, and so on. Therefore, some vision-based racking mehods se fiducial markers on real objecs o help he maching process [3][4][5][6]. We call hese mehods fiducial-based racking mehods. For example, muliple color circle fiducials are used o deec known poins in he mehod described in ref. [3] and [4]. Anoher mehod defines marix codes o recognize objecs and use he four corners of a marix code as known poins [5]. The ARToolKi [6] uses black square regions wih black and whie paerns as fiducials. Fiducial-based racking mehods are also very pracical and appropriae for some AR applicaion; for example, video conferencing applicaions, inerior design applicaions, and so on []. However, an environmen wih oo many fiducials is unnaural and may limi applicaions. Some applicaions require racking mehods ha don use fiducials. A hree-dimensional online manual is a good example of such applicaions. Figure 1 shows he oupu examples of a prooype hree-dimensional online manual. Fiducial markers canno be se for annoaion on all objecs in his applicaion. Therefore, racking mehods ha don use fiducials have been aracing aenion. II. OBJECT TRACKING METHODS We call racking mehods ha don use fiducials naural feaure based racking mehods. Some research groups have been developing naural feaure-based racking mehods [7][8][9]. There are wo ypes of approaches o deermine
he camera parameers: Mehods based on he boom-up approach (BUA), Mehods based on he op-down approach (TDA). In he field of augmened realiy, BUA have mainly been used no only in naural feaure based mehods bu also in fiducial-based mehods. BUA-based mehods have he following wo dominan seps: 1. Reference poins racking sep: This sep oupus wo-dimensional image coordinaes of reference poins. We define reference poins as poins of which hree-dimensional posiions and emplaes for image maching are regisered in a daabase. Usually, BUA-based mehods use poins ha can be easily deeced such as corner poins or poins on edges as reference poins. In oher words, BUA-based mehods mach he capured image sequences and known models of reference poins in his sep.. Camera parameers calculaion sep: This sep calculaes camera parameers using he known hree-dimensional posiions and he obained image coordinaes of reference poins. When a leas hree reference poins are deeced or racked, his sep can calculae camera parameers. BUA-based mehods can calculae accurae camera parameers wih low calculaion cos. However, i is difficul o deec and rack he wo-dimensional posiion of reference poins. One imporan reason for his difficuly is a problem of reference poins changing he appearances. Appearances of local areas around reference poins change as he camera moves, so reference poins in a capured image and emplaes in he daabase do no mach. This causes mis-racking of he reference poins. Anoher imporan reason is occlusion of he reference poins. When reference poins are occluded by anoher real objec, emplaes of reference poins are probably mached o incorrec poins. This also causes mis-racking of reference poins. Mis-racking of he reference poins resuls in an inaccurae esimaion of he camera parameers. Therefore, BUA based racking mehods have o deal wih hese wo problems. On he oher hand, TDA-based mehods can robusly esimae camera parameers using conex and muliple hypoheses. One well-known TDA-based mehod is he ConDensaion [10] framework. The ConDensaion framework has muliple hypoheses represened wih a discree probabiliy densiy of parameers o be deermined. We can use he ConDensaion framework o esimae he camera parameers [8]. In his case, we represen he discree probabiliy densiy of he camera parameers wih a se of samples of each frame. A sample shows possible camera parameers. TDA-based mehods can rack an objec even if he objec is occluded or is in cluers. However, o rack he arge in real ime, we have effecively o limi he exen of sampling area and he number of samples. Inerial orienaion sensors can help vision-based racking mehods, because hey can give camera orienaion daa even if camera moves quickly. Quick camera moion causes an image moion blur ha makes vision-based racking difficul. Some racking mehods use inerial orienaion sensors wih vision-based racking. Inerial orienaion sensors can be easily added o wearable AR sysems and video see-hrough AR sysems because hese sensors do no require exernal sensing devices. However, oupu error of hese sensors accumulaes during racking process because hey don have any references. We have o rea his problem, which is called as drif, when we combine inerial orienaion sensors wih vision-based racking. III. OUR METHOD We propose a novel racking mehod based on hybrid framework of a BUA-based esimaion and a TDA-based esimaion. Our mehod also uses an inerial orienaion sensor. Figure shows a diagram of our mehod. The hick arrows indicae flow of he processes, and he hin arrows indicae flow of daa Firs, our mehod predics camera parameers using he camera posiion and velociy of he previous frame. To predic he parameers effecively, our mehod uses daa from an inerial orienaion sensor. Subsecion A describes he deail of his predicion sep. Then, our mehod esimaes he camera parameers using a BUA-based esimaion. The BUA-based esimaion calculaes a number of poenial camera parameers using he BUA and he prediced parameers, and hen i deermines he camera parameers using robus saisics. When he error in he esimaed parameers is sufficienly small, he mehod oupus he parameers. Subsecion B shows he deails of he BUA-based esimaion. When he error in he parameers esimaed by BUA-based esimaion is no sufficienly small, our mehod sars esimaing he camera parameers using a TDA-based esimaion. Our mehod uses he esimaed parameers by he BUA-based esimaion o creae an iniial discree probabiliy densiy. While he error in he oupu parameers from BUA-based esimaion is no sufficienly small, he TDA-based esimaion works. The deail of he TDA-based esimaion is described in Subsecion C. Our mehod compares he error of he TDA-based esimaion wih he error of he BUA-based esimaion. Then, our mehod oupus he parameers wih he smaller error. A daabase used in he proposed mehod has he following informaion: reference images: images ha capure objecs being racked, camera parameers of reference images: he camera parameers when each reference image is capured, posiion of reference poins: poins on he objec being racked ha are deeced as feaure poins on he reference images. Reference poins are defined as a local areas ha have big eigenvalues in he following marix [11]. M fp ( di ) ( di di dx dx)( dy) ( di di di dx)( dy) ( dy) = (1) The wo-dimensional image coordinaes on he reference image and he hree-dimensional posiions of reference poins are measured in advance. This informaion is used in
Inerial orienaion sensor daa Camera parameer predicion Inpu image Prediced parameers Creae new enry for daabase addiion BUA-based esimaion Esimaed parameers in he BUA Is error small enough? Yes No TDA-based esimaion Esimaed parameers in he TDA New enry for daabase Regiser a new enry Oupu he esimaed parameers in he BUA Decision of he oupu parameers Daabase Reference image Reference poins Oupu decided camera parameers Figure : Flow diagram of he proposed mehod. boh of he BUA-based esimaion and he TDA-based esimaion. To deal wih he problem of he feaure poins changing he appearances, he Auomaic Daabase Addiion (ADA) sep auomaically adds he appearance daa of he objec being racked o he daabase as appropriae while i is racking he objec. We describe he deail of he ADA sep in Subsecion D. A. Parameer predicion To rack an objec effecively, our mehod predics he camera parameers using he parameers of he previous frame, he velociy of he camera, and he inerial orienaion sensor daa. Because inerial orienaion sensors measure only he hree orienaion parameers, we assume hree ypes of movemen o predic he six camera parameers; hree posiion parameers and hree orienaion parameers. To eliminae effecs of drif and error accumulaion, he predicion sep uses he difference beween he daa of he sensor in he previous frame and ha in he curren frame. The hree ypes of movemens are hese: Movemens of he user when he/she moves he objec in he field of view. These movemens do no affec he roaion sensor daa. Movemens of he user when he/she is moving around he objec o observe i (objec-cenered roaion). The objec appears o be roaing around is cener. The roaion is in he direcion opposie o ha of he roaion of he sensor. Movemens of he user when he/she is looking around (user-cenered roaion). The objec appears o be roaing around he viewing posiion. The roaion angle and he direcion of roaion are he same as hose of he roaion sensor. Our mehod calculaes six ses of prediced camera parameers PCPn: PCP1: CP -1 PCP: CP -1 +VC -1 PCP3: T OCR (CP -1 ) PCP4: T OCR (CP -1 +VC -1) PCP5: T UCR (CP -1 ) PCP6: T UCR (CP -1 +VC -1) where CP -1 is he camera parameers in he previous frame, and VC -1 is he velociy of he camera which can be calculaed as CP -1 - CP -. T OCR (CP) and T UCR (CP) indicaes he ransformaion of he objec-cenered roaion and ha of he user-cenered roaion. B. Boom-up approach (BUA) based esimaion A BUA-based esimaion is a primary par of our racking mehod. As we menioned, a BUA-based approach has o deal wih mis-racking of reference poins. To reduce mis-racking by a problem of reference poins changing appearance, our BUA-based esimaion uses muliple reference images and a reference image roaion. In addiion, our BUA-based esimaion sep uses LMedS framework wih he mehod for solving P3P problems. The LMedS framework makes BUA-based esimaion possible o calculae he camera parameers from a reference poins racking resul ha includes mis-racking. The following subsecions describe he deails. 1) Reference poins racking sep using muliple reference images and reference image roaion.
In his sep, feaure poins ha correspond o reference poins are deeced in inpu image. Our mehod adops he Lucas-Kanade mehod, which racks feaure poins by ieraive calculaion using gradien informaion abou local areas[11]. Since he Lucas-Kanade mehod assumes only he ranslaions of feaure poins on an image plane, i canno deal wih he problem of he reference poins changing appearances. Therefore, our mehod prepares muliple reference images. To minimize he calculaion cos, only wo reference images are used in he racking for each se of he prediced parameers; PCP1 o PCP6. One reference image is he image ha is used in he previous frame. The oher one is seleced by comparing he prediced parameers wih regisered camera parameers of he reference images. (When hese wo reference images are he same, he only one image is used.) Our BUA-based esimaion uses he prediced parameers o calculae he iniial values of he ieraive calculaion in he Lucas-Kanade mehod. As a resul, he reference poins deecion is processed welve imes in maxim. We also use an image roaion o deal wih he problem of he reference poins changing he appearances. The image roaion can approximae he appearance of he objec when he objec roaes around he opic axis of he camera, even if he objec has a hree-dimensional shape. The roaion angles are calculaed as he difference beween he roaion parameers of reference image and ha of he prediced parameers. This image roaion sep does no need o be done wih he viewpoin of accuracy. However, i can preven oo much addiion of enries ino he daabase from he ADA sep described in subsecion D. I can conribue he oal efficiency of he sysem. ) Parameer esimaion using he mehod for solving P3P problems and he LMedS framework. Our BUA-based esimaion uses a mehod for solving P3P problems in he LMedS framework. Mehods for solving PnP problems can calculae accurae camera parameers using wo-dimensional image coordinaes and he hree-dimensional posiion of he n poins. However, as we menioned, he resuls of racking all he n poins are rarely accurae because of mis-racking of reference poins. We hus use he LMedS framework o esimae he camera parameers because he LMedS framework can acquire accurae camera parameers a very high possibiliy if more han a half of all reference poins are correcly racked. Afer he LMedS framework acquires he camera parameers, mis-racked reference poins can be eliminaed as he ouliers. Therefore, our mehod opimizes he parameers wih Levenberg-Marquad mehod using reference poins ha are racked correcly. The following shows he deail seps. Sep 1. This sep randomly selecs hree poins from n racked reference poins, and hen calculaes poenial camera parameers using a mehod for solving P3P problems [1]. Sep. This sep calculaes an error err LMedS defined by he following equaion. ( ) ( ) errlmeds = med x x y y i i + i i, () where ( x y ) are he image coordinaes of he racked i-h i i reference poin, ( x, i y i) are he image coordinaes ono which he hree-dimensional posiion of he reference poin is projeced wih he calculaed camera parameers, and med(f(i)) indicaes he median of he f(i) for all i. Sep 3. Seps 1 and are repeaed m imes. The smalles number of imes, m, is deermined by he following inequaliy: 3 p < 1 (1 r ) m (3) where p is he assumed probabiliy ha he camera parameers are calculaed wih correcly racked reference poins, and r is a rae a which reference poins are assumed o be racked correcly. Sep 4. To deec inliers, he hree-dimensional posiions of he reference poins are projeced ono image plane using he camera parameers wih he smalles err LMedS. Sep 5. This sep acquires he camera parameers ha have he smalles errbua using Levenberg-Marquad mehod. err is defined by he following equaion. BUA errbua = ( xi x i ) + ( yi y i ) (4) C. Top-down approach (TDA) based esimaion When he BUA-based esimaion canno obain camera parameers wih sufficienly small err BUA, our mehod uses he TDA-based esimaion. In his sudy, we design he TDA-based esimaion o process jus like he ConDensaion algorihm. Our TDA-based esimaion racks he arge objec by repeaing hree seps; Sampling, Observaion, and Decision sep. 1) The sampling sep. As we menioned before, TDA-based esimaion represens a discree probabiliy densiy of he camera parameers a each frame. The sampling sep generaes a new sample se for he discree probabiliy densiy. To creae effecive samples, he sampling sep uses he prediced parameers and he parameers esimaed by he BUA-based esimaion. The following paragraphs describe (1) () ( N ) he sampling seps ha creae sample se { s, s,, s } of ime, where s denoes a sample ha indicaes he camera parameers, and N denoes he number of samples. Firs, he sampling sep creaes a se comprising 1/7 of all (1) () ( N / 7) he samples { s, s,, s } using random sampling. The cener of disribuion used in he random sampling is se o he parameers esimaed by he BUA-based esimaion. The sampling sep selecs he mehod for creaing he res of he samples depending on wheher he BUA-based esimaion or he TDA-based esimaion was used o esimae he camera parameers of ime -1. Case 1) he oupu of ime -1 was esimaed by using he BUA-based esimaion: Each se comprising 1/7 of all he samples { } ( jn / 7 + 1) ( jn / 7 +, ),, (( j+ 1) N s / 7) s s, ( 1 j 6) is generaed by random sampling. The cener of disribuion used in he random sampling is se o PCPj. Case ) he oupu of ime -1 was esimaed by using he
TDA-based esimaion: The sampling sep creaes he res of samples using he following hree sub-seps; Firs sub-sep selecs he 6N/7 samples from a se of (1) () ( N s, s,, s ) according o samples wih ime -1 { 1 1 1 } ( i) π i heir weighs { }, ( 1,,..., ) 1 = N, which are calculaed in he previous observaion sep. The seleced samples (1) () (6 N / 7) are denoed by { s, s,, s }. (k ) s, ( k 1,..,6 N / 7) Second sub-sep ses = by he following equaion: (k) s (1 k N/ 7) (k) s + VC 1 ( N/ 7 + 1 k N/ 7) (k) ( k ) TOCR ( s ) ( N /7+ 1 k 3 N /7) s = (k) TOCR ( s + VC 1) (3 N /7+ 1 k 4 N /7) (k) TUCR ( s ) (4 N /7+ 1 k 5 N /7) (k) TUCR ( s + VC 1) (5 N /7+ 1 k 6 N /7) (h) The las sub-sep generaes samples s, from ( h N / 7) ( h = N / 7 + 1,..., N) using random walk. s ) The observaion sep. Our TDA-based esimaion sep calculaes an evaluaion value of each sample based on he image observaion. The observaion sep deecs naural feaure poins of inpu frame as local areas ha have big eigenvalues in he marix M fp of he equaion (1). The observaion sep uses he image coordinaes of he neares deeced poin ( x, y i i ) in place of he ( x, y ) i i coordinaes o calculae he error. () i errtda = med ( x x ) + ( y y i i i i). () i Then, he observaion sep calculaes a weigh π of each i-h sample by he following equaion: (i) () i err TDA / π = e. Finally, he observaion sep normalize N ( n) π = 1 () i π so ha n= 1. 3) The decision sep. The decision sep calculaes he weighed average of a sample se using π () i as a weigh of each sample, and oupus hese average camera parameers as he represenaion of he esimaed probabiliy densiy of he frame. D. Auomaic Daabase Addiion. To rack he objec from any viewpoin, he proposed mehod requires a daabase ha has he muliple reference images capured from differen viewpoins. To simplify he preparaion of a daabase, our racking mehod uses Auomaic Daabase Addiion sep. This sep auomaically adds new enry during he racking process. An enry of he daabase consiss of hree componens; 1) he frame image, ) calculaed parameers, and 3) image coordinae values and hree-dimensional posiions of reference poins. When he TDA-based esimaion sars, he auomaic daabase addiion sep adds hese hree componens of he previous frame as a new enry if he number of racked reference poins is larger han hreshold and he err BUA is smaller han hreshold. This sep can reduce he number of reference images in he iniial daabase and he necessiy of complicaed preparaion. IV. EXPERIMENT To evaluae he racking compeence of our mehod, we inpu real image sequences o our mehod. The iniial daabase had only one enry. The mehod was implemened on a PC (Inel Xeon, dual CPU, 1.7 GHz). Figure 3 shows he change in he min(err BUA, err TDA ) and he number of reference images in he daabase as well as some images ha he axes of he objec coordinaes are overlaid ono. These racking resuls show ha he proposed mehod can rack he real objec in real image sequences. These resuls also show he proposed mehod can rack even if he user s hand occludes he objec o be racked. The ime of processing each frame was, on average, 33 ms when only he BUA-based esimaion was used, and 5 ms when boh he BUA-based esimaion and he TDA-based esimaion were used. V. CONCLUSION In his paper, we described a naural feaure based 3-D objec racking mehod for video see-hrough and/or wearable augmened realiy sysems. This mehod is based on a combinaion of he BUA, TDA, and Auomaic Daabase Addiion. Basically, we use he BUA-based esimaion o rack objecs. The BUA-based esimaion can rack he objec even if miss-racking of he reference poins occurs because i is based on he LMedS framework. In he LMedS framework, mehod for solving P3P problems calculaes each poenial exernal camera parameers using he wo-dimensional coordinaes and hree-dimensional objec coordinaes of he racked reference poins. In he BUA-based esimaion, he reference poins racking sep use muliple reference images and image roaion o deal wih he problem of he reference poins changing he appearance. The TDA-based esimaion is effecive when more han half of he reference poins are occluded. The TDA-based esimaion racks he arge even when he BUA-based esimaion canno, bu i canno rack he arge for a long ime. In he proposed mehod, TDA-based esimaion works when he BUA-esimaion canno acquire accurae camera parameers. In he TDA-based esimaion, he number of samples is no enough and observaion is oo simple o rack objecs for long erm, because of he requiremen for real ime racking. In addiion, Auomaic Daabase Addiion sep exends he area of racking. The Auomaic Daabase Addiion sep adds new enry when he BUA-based racking fails o esimae he camera parameer. The proposed mehod can robusly rack objecs because hese wo approaches were effecively combined. An experimenal resul shows an accuracy and inegriy of our mehod.
pixel 5 4 3 1 The sysem found he arge The sysem seleced TDA oupu User's finger occluded he arge 50% or more par of he arge was ouside of he frame 0 1 10 19 8 37 46 55 64 73 8 91 100 109 118 17 136 145 154 163 17 181 190 199 08 17 6 35 44 53 6 71 80 89 98 307 316 35 # of Frame # of Ref. Image 30 0 10 0 1 7 13 19 5 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 11 17 133 139 145 151 157 163 169 175 181 187 193 199 05 11 17 3 9 35 41 47 53 59 65 71 77 83 89 95 301 307 313 319 35 331 337 # of Frames 10 h frame 8 h frame 16 h frame 144 h frame 185 h frame 30 h frame 83 rd frame 316 h frame Figure 3: error value and oupu sills of he experimen. Currenly, our mehod canno add new reference poins ino he daabase enry because i doesn esimae hree-dimensional coordinaes of new reference poins. In he fuure, we will combine he mehod for shape from moion wih our mehod o make he auomaic daabase addiion sep possible o add new reference poins ino a daabase. REFERENCES [1] Azuma, R.T., A survey of augmened realiy, Presence, vol.6, No.4, pp.355-385, 1997. [] Azuma, R., Baillo, Y., Behringer, R., Feiner, S., Julier, S., and MacInyre, B., Recen Advances in Augmened Realiy, IEEE Compuer Graphics and Applicaions, Vol. 1, No. 6, pp. 34-47, 001. [3] Neumann, U., and Cho, Y., A self-racking augmened realiy sysem, In Proc. VRST 96, pp. 109-115, 1996. [4] Neumann, U., You, S., Cho, Y., Lee, J. and Park, J., Augmened Realiy Tracking in Naural Environmens, Mixed Realiy Merging Real and Virual Worlds, Ohmsha & Springer-Verlag, pp. 101-130, 1999 [5] Rekimoo, J., Marix: A Realime Objec Idenificaion and Regisraion Mehod for Augmened Realiy, APCHI 98, 1998. [6] Kao, H., and Billinghurs, M., Marker Tracking and HMD Calibraion for a Video-based Augmened Realiy Conferencing Sysem, In Proc. he nd IEEE and ACM Inernaional Workshop on Augmened Realiy 99, pp.85-94, 1999. [7] Okuma, T., Kuraa, T., and Sakaue, K., Real-Time Camera Parameer Esimaion for 3-D Annoaion on a Wearable Vision Sysem, IEICE Trans. Inf. Sys., Vol.E84-E, No. 1, pp.1668-1675, 001. [8] Okuma, T., Kuraa, T., and Sakaue, K., VizWear-3D: A Wearable 3-D Annoaion Sysem Based on 3-D Objec Tracking using a Condensaion Algorihm, In Proc. IEEE Virual Realiy 00, pp.95-96, 00. [9] Simon, G. and Berger, M-O., Reconsrucing while regisering: A novel approach for markerless augmened realiy, In Proc. IEEE and ACM Inernaional Symposium on Mixed and Augmened Realiy, pp.85-94, 00. [10] Isard, M. A., Visual Moion Analysis by Probabilisic Propagaion of Condiional Densiy, Ph.D. hesis, Deparmen of Engineering Science, Universiy of Oxford, 1998. [11] Lucas, B.D. and Kanade, T., An Ieraive Image Regisraion Technique wih an Applicaion o Sereo Vision, Proc. DARPA Image Undersanding Workshop, pp.11-130, 1981 [1] Haralick, R.M., Lee, C.-N., and Oenberg, K. Analysis and soluions of he hree poin perspecive pose esimaion problem, Proc. CVPR 91, pp.59-598, 1991.