Workshop IAR AMRA : Augmented Reality assistance in train maintenance tasks. Speaker : Jean-Yves DIDIER Laboratoire Systèmes Complexes (FRANCE) Workshop IAR -Vienna 05 Oct 2005 1/17
The AMRA project AMRA : Assistance à la Maintenance en Réalité Augmentée Augmented Reality Assistance for Maintenance Started in May 2002, finished in May 2004 A consortium of several French partners ALSTOM Transport (application provider) CEA LIST (research center) LSC (university) ACTICM (start-up) Workshop IAR -Vienna 05 Oct 2005 2/17
Challenging issues Alstom Transport's maintenance activity : Annual turnover : x100 Meuros International contracts on the long term Operating profits based on efficiency Main issues Heterogeneity and complexity of equipment Lack of local experience abroad Mass of technical manuals (sometimes abroad) How to access to adequate information? How augmented reality can help? Workshop IAR -Vienna 05 Oct 2005 3/17
Overall Architecture (1/2) Virtual World Real World Virtual Model Real device Registration by computer vision Knowledge base Object pointing Man Machine Queries Interface technical data, check lists, diagnostic procedures, 3D simulation,etc.. Workshop IAR -Vienna 05 Oct 2005 4/17
Overall architecture (2/2) Hardware Software Camera / Video Source Tablet-PC : MMI, real/virtual mixing Database Video Data Remote PC Camera REGISTRATION SYSTEM 80 2. 11 g W La n Video Stream Camera Pose Estimator Pose Estimation Object ID queries DATAFLOW MANAGER W i fi FireWire Video Stream Acquisition High Level Operator Event Processing Operator Events System Events procedures + Monitoring multimedia data (On remote server) Multimedia Streams Input events Manager Multimedia Engine for AR Graphical User Interface MMI Lan/Wan DATABASE MANAGER Tablet-PC Remote computer : Image processing Enjoying The Sight actions Database User Workshop IAR -Vienna 05 Oct 2005 5/17
Registration system (1/2) Two kinds of fiducials Colored coded fiducials, Simple fiducials Coded fiducials detection Pose initialization, Device identification Iterative pose estimation Pose refinement using simple fiducials Minimize 2D projection to find R and t Use CAD models for occluded fiducials culling Workshop IAR -Vienna 05 Oct 2005 6/17
Registration system (2/2) Processing rate 25 Hz ( P4-3GHz) Accuracy (1 m range) +/- 3 mm in the worst case +/- 6 in the worst case always detected when slant < 60 Prospects Marker less tracking initialized using fiducials Workshop IAR -Vienna 05 Oct 2005 7/17
Transforming procedures (1/2) "XMLized" procedures + Multimedia data picture "Paper" technical manuals sound video CAD model Workshop IAR -Vienna 05 Oct 2005 8/17
Transforming procedures (2/2) XML Markups hierarchy procedure proc_title pre_requirement tools safety cautions 1..* media +mid +mfile +mtype +pid fid content idmedia 0..1 title phase consumable req_spare consumables spares phases tool available_media req_conditions +mid note did quantity reference name Workshop IAR -Vienna 05 Oct 2005 detail +link spare +sid 9/17
Augmented 3D models (1/2) VRML97 / X3D standards with Java/ECMA scripts for customized behavior, 3D models extracted from CAD models, need : Polygonal reductions, Scene graph hierarchy, Interactivity, for browser use, A human hand is still needed for this part!!! Workshop IAR -Vienna 05 Oct 2005 10/17
Augmented 3D models (2/2) New nodes for augmented reality (PROTOS) Graphical assistance to explain the procedure's steps Animations Scene graph Groupe BoxCovers Groupe BoxCover1 BoxCover1 Remover TerminalBoxCover Transform BoxCover 2 Ancre BoxCover2 Transform Vi s BoxCover2 BoxCoverFader BoxCover Material Node hierarchy Unscrewer Vis1 Vis1 Unscrewer Vis2 Vis2 Unscrewer Vis3 Vis3 Event routes Workshop IAR -Vienna 05 Oct 2005 Unscrewer Vis4 Unscrewer Vis5 Unscrewer Vis6 Vis4 Vis5 Vis6 11/17
Multimedia Engine Videos Image 3D Models Sound Scenario (XML) 3D animations (VRML/Javascript) Multimedia Engine (Qt/OpenInventor/NJS) Video stream Media types Registration data (position+orientation) Workshop IAR -Vienna 05 Oct 2005 Operator Events 12/17
Man Machine Interface Overview Interacting with models Workshop IAR -Vienna 05 Oct 2005 13/17
Early subjective evaluation (1/2) One operator : subjective questionnaire Workshop IAR -Vienna 05 Oct 2005 14/17
Early subjective evaluation (2/2) First results Usability Comfort Animations understanding Visual rendering quality Tablet-PC Interface Benefit of using AR Recommendations Real/Virtual Mixing understanding use of visual hints for spatial positioning tablet-pc too heavy, use of pocket-pc recommended, touchscreen and stylus pen not recommended for maintenance environment. Workshop IAR -Vienna 05 Oct 2005 15/17
Conclusion Prototype fully operational, Real-time registration, Lightweight XML description of maintenance procedures, Database customization for AR A real need of extended field studies!!! Workshop IAR -Vienna 05 Oct 2005 16/17
Future works Alstom : AR media integration in technical database (TrainTracer) speech recognition for browsing, use of VRML/X3D models, use of XML specification for maintenance procedures. LSC : investigating optical see-through systems and architecture for AR systems CEA : more ergonomic studies and registration on embedded systems coupled with cameras Workshop IAR -Vienna 05 Oct 2005 17/17