Situated Visualization with Augmented Reality. Augmented Reality

Transcription

1 , Austria 1 Augmented Reality Overlay computer graphics on real world Example application areas Tourist navigation Underground infrastructure Maintenance Games Simplify Qualcomm Vuforia [Wagner ISMAR 2008] 2 1

2 AR System Overview Virtual content Registration of virtual content Situated visualization SPATIAL MODEL Real world model Pose tracking User input and camera movement 3 Situated Visualization Examples Quadcopter flight planning Underground infrastructure visualization [Zollmann TVCG 2014] [Mendez CG&A 2008] 4 2

3 What is Expected of AR today? Must run on standard devices Smartphones, tablets, eyeglasses Localization in real-time 30Hz, <100ms latency Accurate to 1cm / 1 degree Work anywhere Indoors and outdoors Point-and-shoot Without any user training 5 Challenges of Outdoor AR 1. Mobile devices User go anywhere - big database, only mobiles Mobiles: slow, little memory, poor GPS, tiny camera Server Connection unpredictable, high network latency 2. Localization Database Wide area means large database Environment: feature-poor, repetitive, temporal variations Searching the database takes long Creating the database takes a lot of effort Incremental updating of database typically unsupported 3. The user does not understand what works 6 3

4 Approaches for AR Outdoor Localization How to get 30Hz pose tracking on a smartphoneclass device everywhere? Better visual matching strategies Sattler/Leibe/Kobbelt, RWTH Parallel tracking + mapping + localization Overcome latency, scalability via server Collect more images to overcome narrow FOV/user Prune search space Priors from sensors Priors from environment structure 7 Parallel Tracking and Mapping Video stream FAST New frames SLOW Simultaneous localization and mapping (SLAM) in small workspaces Tracking Map updates Mapping Klein/Drummond, U. Cambridge Tracked local pose 8 4

5 Parallel Tracking, Mapping and Localization Video stream Wide-area visual feature database CLIENT New frames SERVER New keyframes FAST SLOW SLOWEST Tracking Map updates Mapping Global pose Matching Tracked global pose 9 Video stream Geo-Registered Content Wide-area visual feature database Georegistered content New frames New keyframes Tracking Map updates Global pose Mapping Global pose Matching Tracked global pose Annotations in local reference frame 10 Langlotz Proc. IEEE 2014] 5

6 Tracking and Mapping on Mobiles Panoramic SLAM Only rotation, user must stay in one place Works instantly Full 6DOF SLAM User can move freely Needs baseline walk several meters [Wagner IEEE VR 2010] 11 Panorama Localization Study [Arth ISMAR 2011] Monotonic increase of success with wider angle Localization success rate is strongly correlated to aperture angle Steep angles (looking down the street) poor matching success Rates saturate around 180 (due to missing information at steep angles) 12 6

7 Both 6DOF and Panoramic SLAM Switch dynamically Triangulate later 53% success Hybrid SLAM 6DOF Keyframes Triangulated Infinite Features Local Pano Maps 98% success [Pirchhheim ISMAR 2013] 13 Outdoor Localization from SLAM [Ventura TVCG 2014] [Ventura CVPR 2014]

8 Tracking Through Global Model Occlusion Prune Search Space with Sensor Priors GPS: only search near position prior GPS Compass: only search in approximate heading Accelerometer/Gravity: Only consider features with right orientation Gravity [Arth ICPR 2012] 16 Compass 8

9 Improvement with Sensor Priors 15% higher success rate of localization Much faster 17 Prune Search Space with GIS OpenStreetMap now available everywhere! Offline: [Arth com.geo. 2013] Align features with facades during reconstruction Reconstructed model is less distorted Online: Compute visibility sets [Arth. ISMAR 2009] Visibility from facades (GPU) Prune database using visibility Detecting building outlines in image [in progress] 18 9

10 User Generated Situated Visualization Video stream Wide-area visual feature database Georegistered content New frames New keyframes Tracking Map updates Global pose Mapping Global pose Matching Tracked global pose Annotations in local reference frame 19 User Generated Content - Examples Annotated environments User generated video [Langlotz IEEE Perv. Comp. 2012] [Langlotz OzCHI 2012] 20 10

11 Future Social Augmented Reality Infrastructure Content is tagged User name Content type Position... Uploaded to a web community server Content can be accessed via http using tags Big Content Provider Small Personal Servers 21 Conclusions and Future Work AR will establish itself as a new medium People have to learn how to use and understand such a new medium Many open challenges Scalable infrastructures (e.g., sustainable AR city models) Content creation tools (tap into legacy databases like GoogleMaps) Content complexity (AR use cases that provide benefit over simply QR-codes) Robustness of the user experience (tracking failures etc.) Market fragmentation (one app per AR experience) Ergonomics (return of the glasses?) 22 11

12 Questions? 23 References [Arth com.geo. 2013] Clemens Arth, Jonathan Ventura, and. Geospatial Management and Utilization of Large-Scale Urban Visual Reconstruction. In Proc. 4th International Conference on Computing for Geospatial Research and Application (COM.Geo 2013), San Jose, CA, USA, July [Arth ICPR 2012] Clemens Arth, Alessandro Mulloni, and. Exploiting Sensors on Mobile Phones to Improve Wide-Area Localization. In Proc. International Conference on Pattern Recognition (ICPR) 2012, Tsukuba, Japan, November [Arth ISMAR 2009] Clemens Arth, Daniel Wagner, Manfred Klopschitz, Arnold Irschara, and. [Arth ISMAR 2011] Clemens Arth, Manfred Klopschitz, Gerhard Reitmayr, and. Real-Time Self-Localization from Panoramic Images on Mobile Devices. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, October [Langlotz IEEE Perv. Comp. 2012] Tobias Langlotz, Daniel Wagner, Alessandro Mulloni, and. Online Creation of Panoramic Augmented Reality Annotations on Mobile Phones. IEEE Pervasive Computing, 11(2):56-63, April [Langlotz OzCHI 2012] Tobias Langlotz, Mathäus Zingerle, Raphael Grasset, Hannes Kaufman, Gerhard Reitmayr: [Langlotz Proc. IEEE 2014] Tobias Langlotz, Thanh Nguyen,, and Raphael Grasset. Next Generation Augmented Reality Browsers: Rich, Seamless, and Adaptive. Proceedings of the IEEE, 102(2): , February 2014 [Mendez CG&A 2008] Erick Mendez, Gerhard Schall, Sven Havemann, Sebastian Junghanns, Dieter Fellner, and. Generating Semantic 3D Models of Underground Infrastructure. IEEE Computer Graphics and Applications, 28(3):48-57, May [Pirchhheim ISMAR 2013] Christian Pirchheim,, and Gerhard Reitmayr. Handling Pure Camera Rotation in Keyframe-Based SLAM. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2013, Adelaide, Australia, October [Ventura CVPR 2014] Jonathan Ventura, Clemens Arth, Gerhard Reitmayr, and. A Minimal Solution to the Generalized Pose-and- Scale Problem. In Proc. IEEE Computer Vision and Pattern Recognition 2014, Columbus, OH, USA, June [Ventura TVCG 2014] Jonathan Ventura, Clemens Arth, Gerhard Reitmayr, and. Global Localization from Monocular SLAM on a Mobile Phone. IEEE Transactions on Visualization and Computer Graphics, 20(4): , March [Wagner IEEE VR 2010] Daniel Wagner, Alessandro Mulloni, Tobias Langlotz, and. Real-Time Panoramic Mapping and Tracking on Mobile Phones. In Proc. IEEE Virtual Reality, pages , Boston, USA, March [Wagner ISMAR 2008] Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and. Pose Tracking from Natural Features on Mobile Phones. In Proc. 7th IEEE International Symposium on Mixed and Augmented Reality, pages , Cambridge, UK, September Best Paper Award. [Zollmann TVCG 2014] AR Record & Replay:Situated Compositing of Video Content in Mobile Augmented Reality. Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI) Pages , Wide Area Localization on Mobile Phones. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 73-82,