Stable real-time AR framework for training and planning in industrial environments

Transcription

1 Inroducon 1 Sable real-e AR fraework for ranng and plannng n ndusral envronens Sable real-e AR fraework for ranng and plannng n ndusral envronens L. Vacche, V. Lepe, M. Ponder, G. Papagannaks, D. Thalann, N. Magnena-Thalann, and P. Fua. Absrac Augened Realy syses can be effecvely used o enhance anufacurng and ndusral processes. However, no all he exsng prooypes of AR syses can be used n ndusral envronen, due o heavy consrans such as low robusness or cubersoe equpen. Our Augened Realy syse reles on purely passve echnques o solve he real-e regsraon proble and can run on a porable PC. We cobne a powerful VR coponen-based sulaon fraework wh Copuer Vson echnques, urnng no an Augened Realy syse. The resulng syse allows us o produce coplex renderng and anaon of avaars, and o blend he no he real world. The syse racks he 3D caera poson by eans of a naural feaures racker, whch, gven a rough CAD odel, can deal wh coplex 3D objecs. The rackng ehod s robus and can handle large caera dsplaceens and aspec changes. The arge applcaons of our AR syse are ndusral anenance, repar and ranng. The rackng robusness akes he AR syse able o work n real envronens such as ndusral facles and no only n he laboraory. Inroducon Vrual and Augened Realy are becong nexrcably negraed srands of he new eergng dgal vsualzaon fabrc. Wh he adven of he recen powerful, low cos consuer graphc copuers, becoes possble o buld hghly realsc real-e AR and VR sulaons. A he sae e, recen developens n huan anaon have led o he negraon of vrual huans no synhec envronens. As he deand for Augened Realy syses grows, so wll he need o allow hese vrual huans o coexs and nerac wh real objecs and scenes. Ths s especally rue n case of ndusral and anufacurng applcaons where AR syses are hghly suable for ranng, ergonocs evaluaon and rapd valdaon of prooypes before leng he no he anufacurng phase. Vrual eachers can be used o deonsrae coplex achne operaon o he novce users. Slarly, AR s an deal approach for desgnng objecs by havng a vrual huan neracvely perforng evaluaon ess on an objec coposed of real and vrual coponens. Includng real achnery and surroundngs no he neracve sulaon ncreases reals. I decreases e ha would be oherwse requred o odel coplex vrual envronens. Fnally elnaes he copuaon coss nvolved n renderng he. Fg. 1 llusraes hs approach: a vrual worker deonsraes how o use a achne n a coplex ndusral envronen. The user of he AR syse can change he pon of vew a any e, and snce he caera poson s correcly regsered o he real scene, he vrual worker wll always be correcly blended no he sreang vdeo. Anoher applcaon of our syse s shown n Fg. 2, where a vrual huan gudes he user hrough an unknown buldng. We herefore vew our conrbuon as wofold: an accurae real-e vson-based caera racker, whch s responsble for he regsraon of he vrual huans no he sreang vdeo and does no requre engneerng he envronen; s negraon no an exsng VR coponen-based sulaon fraework (VHD++), ha provdes huan-copuer nerface and renderng of realsc vrual huans.

2 Inroducon 2 Sable real-e AR fraework for ranng and plannng n ndusral envronens Thanks o he srong coponen-based characer of he VHD++ sulaon fraework developen of he rackng coponen as a VHD++ plug-n was sraghforward. Fg. 1. Frs and second pcures fro lef: a vrual huan deonsraes he use of a real achne (offlne es requrng wo achnes). Thrd and forh esng he real-e rackng wh basc renderng n a facory (one achne). Fg. 2. The sae vrual huan gudes a vsor hrough an unfalar corrdor. Relaed Work Many papers, such as (Druond and Cpolla 2000), (Neuann and You 1999), (Son e al.), (La Casca e al.), have been publshed on rackng, and soe characerscs such as accuracy and speed, can now be found n any exsng syses. The robusness s uch ore challengng. Prevous AR syses requred o odfy he envronen by nroducng arkers, as was done n (ao e al.) for nsance. Many oher ehods, n order o rack he caera dsplaceens, explo ceran feaures of he objecs of he scene. These feaures can be edges, lke n (Druond and Cpolla 2000) or n (Marchand e al. 1999). Unforunaely edge-based ehods canno be effecvely used n real ndusral envronen snce he usually cluered background nroduces oo srong nose. Our AR syse akes use of an approach o real-e rackng based on naural feaure pons, whch are auoacally deeced and racked as hey appear. Conrarly o prevous ehods, whch also consder naural feaure pons, our ehod s no led o pecewse planar scenes bu can handle arbrarly coplex scenes, wh no ls o he caera dsplaceen. Many oher algorhs oban hgh accuracy, even whou an a pror knowledge, by achng naural feaures such as neres pons. For exaple (Fzgbbon and Zsseran 1998) process he age sequence herarchcally o derve robus correspondences and o dsrbue error over he sequence. Consderng speed as no crcal ssue, hese algorhs ake advanage of e consung bu effecve echnques such as bundle adjusen. Many oher ehods perfor he sae ask for real-e applcaons bu end o be less relable snce hey can no rely on bach copuaons. Those ha work whou he a pror knowledge are no really praccal: for exaple (Azarbayejan and Penland 1995) assues absence of correspondences errors, and (Beardsley e al.) assues he caera cener oon o check f he correspondences respec he eppolar consran. Ths paper s organzed as follows: n he nex secon we presen he VHD++ coponen-based sulaon fraework, hen n Secon Sable real-e caera rackng we descrbe our rackng algorh, and fnally we wll presen our experens and resuls.

3 3 Sable real-e AR fraework for ranng and plannng n ndusral envronens VHD++ for AR Applcaons Fg. 3. VHD++ Developen Fraework. Fg. 4. VHD++ archecure overvew by exaple. VHD++ for AR Applcaons The VR par ha has been used n our applcaon exaple presened n Fg. 4 enables negraon of heerogeneous sulaon echnologes, such as real-e 3D renderng, skeleon and skn anaon, behavoral conrol, ec. VHD++ vrual huans show large range of anaon capables, as nroduced n (Ponder e al. 2003). The VHD++ fraework consues an exendble, real-e, audo-vsual sulaon engne wh specal suppor of advanced real-e vrual characers sulaon echnologes. AR syses rely heavly on he nerplay of copleenary heerogeneous echnologes. Because of ha nerdscplnary characer, he AR doan can be vewed as a elng po of varous echnologes, whch are non-rval o pu ogeher (Azua 1995). The VHD++ s a coponen-based sofware developen fraework ha suppors coposon of hgh perforance, real-e, neracve, audo-vsual applcaons. I provdes an exendble se of funconal coponens lke 3D renderng, 3D sound, advanced synhec characer sulaon, AI, behavoral conrol, neracve scenaro auhorng, dagnoscs, neworkng, rune daa base anageen, ec. VHD++ developen ehodology reles on assve desgn (rune engne) and code (pluggable coponens) reuse. In effecs applcaons are beng coposed ou of he coponens raher han developed fro he scrach. Fg. 3 shows a hgh level absracon of he VHD++ archecural odel. Pluggable, cuso coponens encapsulang heerogeneous sulaon echnologes are arked as he hn spkes cong ou of he kernel. Soe of he are conneced o he recangles represenng devce absracons. Movaon behnd VHD++ s based on he followng observaon relaed o he VR/AR syses. I occurs ha on he low, syse nfrasrucure level os of he VR/AR syses feaure srong concurrency, nework dsrbuon, suppor of varous npu and oupu devces and of course real-e perforance. I occurs as well ha os of he syses use que slar ses of low level fundaenal coponens responsble for daa loadng, conanen, seralzaon, sharng of resources, ulaskng, synchronzaon, e schedulng, even handlng, neworkng, brokerng, ec. On he hgh, applcaon level he VR/AR syses end o draw fro a connuously growng specru of heerogeneous and copleenary sulaon echnologes whch repea frequenly n dfferen confguraons dependng on parcular applcaon requreens and conex (e.g. caera rackng, 3D renderng, 3D sound, collson deecon, physcs, skeleon anaon, skn deforaon, face anaon, cloh sulaon, crowd conrol, behavoral conrol, ul-odal neracon, ec). In effec of he above observaons he VHD++ seancs has been developed around he followng four key seancal eleens:

4 4 Sable real-e AR fraework for ranng and plannng n ndusral envronens vhdrunesyse (whole dagra); vhdruneengne (ouer ebossed crcle); vhdservces (spkes cong ou of he sharable daa); vhdproperes (sharable daa on he dagra). Sable Real-e caera rackng Fg. 5. Machng of neres pons: he crosses are he deeced neres pons n he pcure. Fg. 6. The odel we used for rackng he corrdor. I was exruded fro a CAD plo and soe deals have been added. I ook abou 1 day of work o be desgned. I s a convenonal VRML odel. In order o llusrae he respecve roles and uual relaonshps beween he above eleens, Fg. 4 depcs n a scheac way an exaple of a VHD++ based applcaon, ncludng s possble physcal deployen sraegy. The exaple shows our arge vhdrunesyse feaurng wo vhdruneengnes hosng a oal of fve plug-able vhdservces dsrbued here over wo copuer nodes. Each of he copuer nodes feaures soe npu/oupu devces ha are used hroughou he sulaon. We can easly agne for exaple ha sarng fro he lef, he frs vhdruneengnea hoss soe servces responsble for ul-odal neracon usng conneced VR npu devces and behavoral conrol of he sulaon. Lower rgh vhdruneengnb hoss for exaple a vhdservce responsble for real-e vson based caera rackng and passng of he low bandwdh caera arx updaes o he vhdservces hosed on he prevous vhdruneengnea whch akes care of he 3D renderng, synhess of real and vrual ages, 3D sound, characer anaon. I s one of he deployen scenaros for our VR/AR applcaon exaple. Exsng vhdservces ay be easly adaped o new requreens hrough dervaon and overrdng of vrual ehods. If necessary, developers ay provde easly new vhdservces ha wll be added o he global pool and hen avalable for fuure reuse. Sable Real-e caera rackng Our rackng ehod s suable for any ypes of 3D exured objecs ha can be descrbed eher as a wre frae odels or a rangulaed eshes. I sars fro 2D achng of neres pons, and hen explos he o nfer he 3D poson of he pons on he objec surface. Ineres pons are pons where a dsconnuy occurs n he age sgnal, and can be deeced usng he ehod proposed n (Harrs and

5 5 Sable real-e AR fraework for ranng and plannng n ndusral envronens Sable Real-e caera rackng Sephens 1998) or (Sh and Toas 1994). They are a relable prve o rack, snce hey canno be confused wh any oher pars of he exure. Moreover, beng a local feaure ha can be noralzed wh respec o he lunance, neres pons do no depend on lghng condons. Fg. 7. The back-projecon process. Fg. 8. Exaples of Face-ID ages. When he neres pons on he objec are racked s possble o rereve he caera dsplaceen n he objec coordnae syse usng robus esaon. In he followng paragraphs we wll descrbe n deal our rackng algorh. Frs we presen a spler verson ha racks he caera dsplaceen frae by frae. Ths ehod works well bu suffers fro error accuulaon for long sequences. We show how o preven hs proble by consderng key-fraes (defned offlne or onlne). Ths ehod s ore coplex bu allows us o consder sequences whou duraon resrcon. Inalzaon We use copuer vson echnques o roughly copue nrnsc paraeers by eans of a calbraon grd. The algorh sars when he user oves he caera or he objec close o a known poson ha ay be shown on he screen. I s poran o sress ha a rough, approxae adjusens s suffcen. The achng algorh receves as npu he ncong age and a boosrap reference frae; f he fraes are close enough, he pon achng nuber ncreases above a gven hreshold and he rackng sars. Fg. 9. Ths sequence has been racked usng a very sple odel of one of he objecs n he envronen (four fraes of he sequence). Fg. 10. Ths pcure shows sx fraes of he racked sequence of he corrdor (700 fraes).

6 6 Sable real-e AR fraework for ranng and plannng n ndusral envronens Sable Real-e caera rackng Fg. 11. Onlne and Offlne eyfraes. a. Tracked caera dsplaceen wh four offlne keyfraes and one onlne keyfrae. The doed arrows represen he caera dsplaceen fro one frae o he nex, and he nuber shows whch keyfrae s beng used. 1 o 4 are he caera posons of he offlne keyfraes. When he curren caera poson ges oo far fro any known offlne keyfrae, a new onlne keyfrae denoed onlne s generaed. b. Choosng he bes keyfrae beween 1 and 2. C s he prevous caera poson. Sple recursve rackng Frs, we deec he sronges neres pons n he curren source age usng he Harrs corner deecor (Harrs and Sephens 1998). Feaure deecon ay also be perfored by eans of he ehod proposed n (Sh and Toas 1994). The sronges pons are he age areas havng color nensy sgnfcanly dfferen fro he neghborhood: we show he neres pons deeced n real ages n Fg. 5. Le he neres pons deeced a he e be: { 0 n} =... (1) Gven a prevous frae, le be he se of 2D pons ha we deeced n and M be her 3D poson. Assung ha he roaon and ranslaon paraeers vecor [ R T ] s known n he prevous frae, bu new pars of he objec ay have appeared, we wan o ake no accoun he new 2D neres pons. So we back-projec he n order o fnd her 3D coordnaes M, keepng only he neres pons ha are on he objec surface and dscardng all he ohers. To do so, we frs use a Face-ID age o deec on whch face of he 3D odel each 2D pon les. Soe exaples of hs knd of age are shown n Fg. 8. Tha age s generaed by encodng he ndex of each face f as a unque color, and projecng he whole odel no he age plane, usng a sandard OpenGL renderng. Once he face-id s known, we fnd he nersecon wh he found face and he lne passng hrough he caera cenre of projecon and he 2D pon n he age plane. To copue hs nersecon we cas a ray fro he cener of he caera Cop, passng hrough he 2D neres pon on he age, and hen nersecng he objec odel n he pon M (see Fg. 7). Gven he wo arces of nrnsc (A) and exrnsc ( [ R T ] ) paraeers ha defne he projecon arx as: [ T ] P = A R (2)

7 7 Sable real-e AR fraework for ranng and plannng n ndusral envronens Sable Real-e caera rackng We can copue he r r vecor, whch gves us he drecon of he ray. I can be expressed as: r (3) = 1 ( AR) where: ( u v ) T = 1 (4) The opcal cener Cop (he orgn of he ray) can be copued usng: Cop = R T T (5) The nersecon wh he odel, n he case of a rangle esh, can be copued by eans of he effcen algorh presened n (Moeller and Trubore 1997), however can be any knd of lne o geoerc shape nersecon algorh. The algorh n (Moeller and Trubore 1997) s a very effcen copuer graphcs algorh ha copues he nersecon beween a rangle esh and a ray. I can be exended o hgh polygon nuber eshes allowng rackng also coplex objecs whou slowng down he racker. Beng he 3D poson M n ha frae known, we have wo ses of pons, respecvely 2D and 3D: such ha: M { 0 n... 1} { M L M n } 1 = 0 1 = 1 (6) [ ] T M = A R 1 (7) where M and R andt, he caera roaon and ranslaon esaed for he prevous frae, are expressed n he objec coordnae syse. A s he nernal paraeers arx. We are lookng for he R and T arces for he curren frae. We ach he 2D pons beween and, choosng for each pon n he se 1 he one n he se ha axzes a correlaon easure ha s nsensve o llunaon changes (Harley and Zsseran 2000).

8 8 Sable real-e AR fraework for ranng and plannng n ndusral envronens Sable Real-e caera rackng Fg. 12. Fro he lef: he frs age has o be ached wh he second one, bu he wo ages are no close o each oher. The pon paches of he frs age wll be skewed as shown n he hrd age o ake he achng possble. As a resul, os of he curren age pons are ached o he prevous age pons : j (8) Snce us re-projec on j we should have: [ ] j R T M A = (9) j Therefore also he 3D pons belongng o M can be assocaed o he 3D pons of way he 3D coordnaes of he unknown pons: M 1, gvng n hs j M = M (10) The 3D pons are he sae for boh he ages f he 2D pons have been correcly ached. Once all he 2D-3D correspondences are done, we have enough nforaon o copue he caera poson n he objec reference syse. Ths s done usng he algorh proposed n (Deenhon and Davs 1992) and he robus esaor RANSAC o dscard ouler aches (Harley and Zsseran 2000). Usng RANSAC n our case eans ha we consder several sall ses of four rando pons o copue a eporary pose, fnd a dsplaceen [ R T ] and re-projec he whole se of pons usng hs [ R T ]. Only he confguraon ha re-projecs he os of pons wll be acceped. Ths algorh s effcen o consder a large nuber of ses of four pons and does hs robus esaon n a reasonable e, whch eans we can do hree hundred eraons n around en llseconds. In hs way we can deec only he correc pons, and dscard all he oulers. eyfrae based rackng In shor, he sple ehod presened n he prevous par works wh very good precson whou jerng. However, he sple recursve approach s oo weak fro he pon of vew of error accuulaon, and s no suable for a real-e envronen. Thus, one us prove he ehod wh soe addonal nforaon. Ths can be done usng soe pror knowledge, suppled by he keyfraes. Ths secon explans how o use keyfraes n order o rack any sequence wh no drf and no ls on he caera poson. Durng he ranng sage he user creaes offlne keyfraes, and n he rackng sage he prevous

9 9 Sable real-e AR fraework for ranng and plannng n ndusral envronens Sable Real-e caera rackng nforaon s used o rack. Durng he rackng he caera ay ove oo far away fro any known keyfrae: In ha case a new onlne keyfrae s added o he ohers and wll be re-used when he caera poson passes close o a second e. Durng he offlne sage, he user s asked o choose a se of ages represenng he scene fro any dfferen pons of vew or, a leas, he posons ha he caera wll probably reach. Usually s enough o record a vdeo sequence all around he objec and o choose a frae for soe caera dsplaceen. Whle rackng he sequences presened n hs paper we only used 4 keyfraes. Coplex aspec changes such as 360 degree roaons and 6 degrees of freedo ay need fro 10 o 20 keyfraes. Afer he reference age choce he user s asked o accuraely calculae he [ R T ] for each age. There are any ehods o calculae he [ R T ]. In our early es sage we were usng a ool bul by us usng he ehod descrbed n (Deenhon and Davs 1992): s enough o ge he 2D poson of 4 known pons n every age o calculae he objecs pose. The user can even ake use of coercal posproducon ools, such as he ones of RealVz or 2D3. The coercal producs can rereve he objec poson over he whole sequence wh good accuracy, snce hey work offlne. R T s known for every keyfrae, he user has copleed he offlne sage. Then he syse When [ ] perfors neres pon deecon and back-projecs he pons ha le on he objec surface o copue her 3D posons. Vsbly creron for keyfrae choce The frs sep of he rackng algorh s o choose he bes keyfrae. Ths choce s a crcal ask on whch he qualy of he achng depends. An aspec of a keyfrae us be as close as possble o he curren frae. As shown n Fg. 11.b, sply evaluang he caera poson s no enough. The pon C represens he curren caera poson, and 1 and 2 are wo keyfraes. Jus akng he keyfrae ha nzes he eucldan dsance eans ha he closes keyfrae s 1. However s aspec s no as close as 2, whch s furher away bu has a closer lne of sgh. To correc hs proble, we should evaluae he angle beween he wo lnes of sgh. However, hs s sll no a coplee ehod, because does no ake no accoun objec non convexes and self occlusons. Insead, we use an appearance-based ehod. We use he followng crera: where ( f P) ( ( f, A [ R T ]) Area p P P f Model Area ( f, A [ R T ])) 2 Area, s he 2D area of he face f afer projecon by P. We reuse he ehod we nroduced n he prevous secon for an acceleraed OpenGL renderng of he objec odel. Every face s rendered n a dfferen color, represenng he face ndex, usng he caera R and T esaed for he prevous frae. We hsogra hs Face-ID age and copare he resul o he keyfrae hsogras, whch have been creaed offlne durng he learnng sage. We ge he conrbuon of he area of every sngle face n he odel as s reprojeced n he 2D age. Every hsogra bar represens he nuber of occurrences of every face's pxels. Ths ehod has consan coplexy, and requres only a sngle read of he age. (11) Wde baselne achng Ths secon presens our ehod o handle he perspecve dsoron on he correlaon wndow. Convenonal ehods ake use of a square b-densonal correlaon wndow. Ths echnque gves good pons achng under he assupon of very sall perspecve dsoron beween wo fraes. However, o effecvely use keyfraes, he ably o ach dsan fraes becoes essenal. Consequenly we specfy a pon achng algorh beween a square 2D wndow n he curren frae and a perspecve dsored wndow n he keyfrae age, ha we call he re-rendered' age. We skew he 30x30 pxel

10 Experens and resuls 10 Sable real-e AR fraework for ranng and plannng n ndusral envronens paches around each neres pon fro he keyfrae age n order o brng he o a poson close o he curren one. Each pach n he keyfrae s relaed o he correspondng age pons n he re-rendered age by a planar hoography. r Gven he pach correspondng o he plane π havng coordnaes π = ( n T, d) so ha for pons on he r plane n T X + d = 0, he general expresson for he hoography nduced by he plane s (accordng o (Harley and Zsseran 2000)): T 1 ( R n / d ) r (12) H = A' A beween wo dfferen vews defned by her projecon arces P = A[ I 0] and P A' [ R ] =. The hoography equaon for he general case can easly be obaned by changng he reference syse; we ge: H = A r T 1 ( δ R δn' / d' ) A P (13) T δ R = R P R ; δ = RPR + P ; r r n = T r R n d' = d R n ' ; ( ) T (14) where A [ R ] and [ R ] A are he projecon arces of he keyfrae and he prevous frae. P P P The resulng age s a re-renderng of he neres pons' neghborhood n a ore convenen poson as shown n Fg. 12. Ths ehod allows us o effecvely ach vews even where here s as uch as 60 degrees of roaon. An alernave soluon o he hoography would be o re-render a 3D represenaon of he objec usng an OpenGL exured 3D objec, bu we choose he oher way o have a beer resul around he pons. Offlne and onlne keyfraes Assung we already have a conssen se of keyfraes, n hs subsecon we show how o eploy he o rack a sequence. As shown n Fg. 11.a, whle he caera oves around he scene, he syse swches fro one keyfrae o he oher, always choosng he one ha s closes o he curren age. When he curren caera poson ges oo far fro any known offlne keyfrae, a new onlne keyfrae denoed onlne s generaed. I wll be added o he keyfrae se and reaed lke he oher ones. The creron we use for decdng o generae an onlne keyfrae s he nu nuber of nlers (correcly ached pons): when hey are less han (dependng on he ype of objec) we swch o a new keyfrae. Afer soe e he caera wll agan pass close o a known poson, re-usng he keyfraes ha have been generaed onlne. If he sequence s dffcul he syse needs ore offlne keyfraes. An neresng characersc of hs ehod s ha when soe error has been accuulaed over a par of he sequence, wll be rese o zero when an offlne frae s used. The onlne fraes can be consdered as a knd of second chance ehod used o recover when here are no offlne keyfraes, and has only o guaranee no coplee dvergence before he caera ges close o an offlne frae. Experens and resuls The whole syse runs close o real-e. Due o he e-consung algorhs we are oblged o use wo separae 2.5GHz PC achnes, one for rackng and anoher for VHD++ renderng respecvely. In hs confguraon he rackng yelds fro 15 o 25 fps dependng on he objec sze, whle renderng goes above 30 fps. Gven he hardware confguraon he curren rackng perforance can be acheved wh

11 Experens and resuls 11 Sable real-e AR fraework for ranng and plannng n ndusral envronens 320x240 ages. Once ore CPU power s avalable, we wll use bgger ages and he rackng qualy wll prove. Ang a he rackng perforance as he os crcal, we have also bul any ess by eans of he rackng algorh bu usng only a lgh renderng par. In hs case our syse can run on a sngle 2.6 GHz lapop achne usng an IEEE 1394 caera. We esed he rackng algorh n a facory envronen, where we succeeded o rack pre exsng equpen and o superpose vrual pars. We also use he sae algorh for rackng and augenng a varey of 3D objecs (e.g. ea boxes, sall oys). The sae ehod s used for face rackng and augenaon n real-e; he execuable of hs algorh s avalable for boh IEEE1394 Pon Grey Dragonfly caera or web ca a he lnk: hp://cvlab.epfl.ch/sofware/download.hl Fg. 13. Plos showng a 700 frae sequence racked usng hree dfferen ehods. The frs plo shows he jer of keyfrae ehod copared o our ehod, he second one shows he error accuulaon of he recursve ehod copared o ours. In boh plos, he bold lne corresponds o our resuls. We use our resul as a ground ruh beng vsually correc when we re-projec he odel. All our deonsraons ake use of sall porable caeras ha can be nsalled on a hele. The vsualzaon s screen dsplay, whch solves all he probles of rackng delay. Beng our caera lens of good qualy we have never addressed he radal dsoron proble; however can be easly solved by eans of he os coon caera calbraon progras (e.g. Ausrals, Inel OpenCV Calb Fler). Exaples of racked objecs and scenes are shown n he Fg. 1 and 2. Fg. 1 shows a vrual huan showng he use of a real achne n a facory. In Fg. 2 we show he resul of a sequence n whch he caera s oved hrough he corrdor and s urnng around he corner, and an avaar s walkng n fron of he user, showng he way o follow. In order o have a ore qualave analyss of our ehod, we ade work frs n soe led condons ha are closer o he ore convenonal approaches, han usng he coplee algorh. So we used a feaure achng approach o rack a scene hree dfferen es: usng only chaned ransforaons, lke a recursve racker, usng only keyfraes (offlne nforaon), cobnng boh usng our proposed ehod. The resul of he cobned ehod s used as ground ruh, snce works wh good accuracy for he whole sequence. We verfed hs by re-projecng he odel on he ages. Fg. 13 depcs he evoluon of one of

12 Bblography 12 Sable real-e AR fraework for ranng and plannng n ndusral envronens he caera cener coordnaes wh respec o he frae ndex. The frs plo copares our ehod o he keyfraes-only ehod. The keyfrae-only ehod suffers fro jer and fals a frae 295 because here s no avalable keyfrae provdng enough pon aches for ha pon of vew. The second plo copares he recursve ehod o ours. Error accuulaon n he recursve ehod does no corrup he rackng edaely, bu evenually also provokes rackng falure around frae 550. A he end, when he rackng has been correcly perfored, he res of he work s done by VHD++. The renderng odule provdes he an suppor for vdeo age synhess of vrual huans-objecs correcly regsered o he real vdeo age. Thus we have pleened successfully he negraon of he rackng and he renderng odule (currenly on he sae vhdruneengne), realzng he archecure depced n Fg. 4. Concluson In hs paper we presened a novel fraework for buldng real-e VR/AR applcaons. Our syse has no only been esed n a laboraory, bu also n a real envronen. The focus of hs work s o buld applcaons for ranng and plannng n ndusral envronen, bu can be used as well for any oher asks, such as ergonocs, repar, prooypng, eergency suaons ranng, and all he cases where a dangerous or oo expensve o creae suaon s presen and canno be reproduced by a real scene. The racker ebedded n he fraework s based on naural feaure pons, and can be used wh a large se of scenes. The odel nforaon s exploed o rack every aspec of a gven arge objec even when occluded or only parally vsble, or when he caera urns around he scene. The resul s a praccal syse ha we have been able o es n a real facory envronen. Bblography Azarbayejan and Penland 1995 A Azarbayejan and A P Penland Recursve Esaon of Moon, Srucure and Focal Lengh. IEEE Transacons on Paern Analyss and Machne Inellgence, vol. 17, no. 6, pp , Azua 1995 R Azua A survey of augened realy. Copuer Graphcs (SIGGRAPH 95 Proceedngs), Augus 1995, pp Beardsley e al P A Beardsley, A Zsseran, and D W Murray Sequenal updae of projecve and affne srucure fro oon. Inernaonal Journal of Copuer Vson, vol. 23, no. 3, pp , Deenhon and Davs 1992 D DeMenhon and L S Davs Model-based objec pose n 25 lnes of code. European Conference on Copuer Vson, 1992, pp Druond and Cpolla 2000 T Druond and R Cpolla Real-e rackng of ulple arculaed srucures n ulple vews. ECCV (2), 2000, pp Fzgbbon and Zsseran 1998 A Fzgbbon and A Zsseran Auoac Caera Recovery for Closed or Open Iage Sequences. European Conference on Copuer Vson, Freburg, Gerany, June 1998, pp Harrs and Sephens 1998 CG Harrs and MJ Sephens A cobned corner and edge deecor. Fourh Alvey Vson Conference, Mancheser, Harley and Zsseran R. Harley and A. Zsseran, Mulple Vew Geoery n Copuer Vson,. Cabrdge Unversy Press, ao e al H ao, M Bllnghurs, I Poupyrev, Iaoo, and Tachbana Vrual objec anpulaon on a able-op AR envronen. Proceedngs of Inernaonal Syposu on Augened Realy, La Casca e al M La Casca, S Sclaroff, and V Ahsos Fas, relable head rackng under varyng llunaon: An approach based on regsraon of exure-apped 3d odels. IEEE Transacons on Paern Analyss and Machne Inellgence, vol. 22, no. 4, Aprl Marchand e al E Marchand, P Bouhey, F Chauee, and V Moreau. Robus real-e vsual rackng usng a 2d-3d odel-based approach. IEEE Inernaonal Conference on Copuer Vson, ICCV 99 (1), Sep. 1999, pp Moeller and Trubore 1997 T Moeller and B Trubore Fas, nu sorage ray rangle nersecon. Journal of graphcs ools, 2(1):21-28, Neuann and You 1999 U Neuann and S You Naural feaure rackng for augened realy. IEEE Transacons on Muleda, vol. 1, no. 1, pp , 1999.

13 Bblography 13 Sable real-e AR fraework for ranng and plannng n ndusral envronens Ponder e al M Ponder, G Papagannaks, T Mole, N Magnena-Thalann, D Thalann, VHD++ Developen Fraework: Towards Exendble, Coponen Based VR/AR Sulaon Engne Feaurng Advanced Vrual Characer Technologes. Copuer Graphcs Inernaonal (CGI) 2003, o appear. Sh and Toas 1994 J Sh and C Toas. Good Feaures o Track. IEEE Conference on Copuer Vson and Paern Recognon, pages , Son e al G Son, A Fzgbbon, and A Zsseran Markerless rackng usng planar srucures n he scene. Proc. Inernaonal Syposu on Augened Realy, Ocober 2000, pp