Example AR image. Augmented Reality. Augmented Reality. Milgram s Reality- Virtuality continuum. Why Augmented Reality? Is AR easier/harder than VR?

Transcription

1 Example AR image Augmented Reality Matt Cooper Youngkwan Cho, STAR system (Many slides based on the MUM2003 Tutorials by Mark Billinghurst and Mark Ollila) Milgram s Reality- Virtuality continuum Real Environment Augmented Reality (AR) Mixed Reality Augmented Virtuality (AV) Reality - Virtuality (RV) Continuum Virtual Environment Why Augmented Reality? Virtual Reality is Ideal for: Replacing the Real World Simulation, Training, Games Augmented Reality is Ideal for: Enhancing the real world Sophisticated interaction in the real world Intelligence Amplification Adapted from Milgram, Takemura, Utsumi, Kishino. Augmented Reality: A class of displays on the reality-virtuality continuum Augmented Reality Virtual Reality: Replaces Reality Immersive Displays Augmented Reality: Enhances Reality See-through Displays Characteristics Combines Real and Virtual Images Interactive in real-time Registered in 3D Is AR easier/harder than VR? Rendering: easier There s s less of it! But we need faster updates Display (resolution, FOV, colour): easier? Tracking and sensing *Much* harder: Greater bandwidth requirements (video, range data, etc.) Support occlusion, general environmental knowledge A big problem for registration! Portability: VE: User stays in one place in the VE AR: User moves to task In the real world

2 Additional problems of AR Computer graphics: faster updates Objects must appear in the right place in the real world Tracking must be: more accurate With respect to the real world Faster Stay aligned with the real world So artificial objects are correctly registered A Brief History of AR (3) 1996: UNC hybrid magnetic-vision tracker 1998: Dedicated conferences begin Late 90 s: Collaboration, outdoor, interaction 2000: Augmented sports broadcasts A Brief History of AR (1) 1960 s: Sutherland / Sproull s first HMD system was see-through Applications Medicine Manufacturing Training Architecture Museum A Brief History of AR (2) Medical X-ray vision for surgeons Aid visualization, minimally-invasive invasive operations. Training. MRI, CT data. Ultrasound project, UNC Chapel Hill. Early 1990 s: Boeing coined the term AR. AR. Wire harness assembly application Early 1990 s: UNC ultrasound project 1994: Motion stabilized display 1994: Fiducial tracking in video see-through AR Courtesy UNC Chapel Hill

3 Assembly and Maintenance 1996 S. Feiner, B. MacIntyre, & A. Webster, Columbia University Application: broadcast augmentation Adding virtual content to live sports broadcasts First down line in American football Hockey puck trails, virtual advertisements National flags in swimming lanes in 2000 Olympics Commercial application Princeton Video Image is one company inc.com/ 1993 S. Feiner, B. MacIntyre, & D. Seligmann, Columbia University Applications: annotated environment Public and private annotations Aid recognition, extended memory Libraries, maps [Fitzmaurice93] Windows [Columbia] Mechanical parts [many places] Reminder notes [Sony, MIT Media Lab] Navigation and spatial information access Broadcast Examples Annotation pictures Key AR Technologies 1993 S. Feiner, B. MacIntyre, M. Haupt, & E. Solomon, Columbia University Columbia University Input Tracking technologies Input devices Output Display (visual, audio, haptic) Fast Image Generation HRL

4 Video see-through HMD AR Displays Monitors Video cameras Video Graphics Combiner Optical see-through HMD Video see-through HMD Virtual images from monitors Real World Optical Combiners MR Laboratory s COASTAR HMD (Co-Optical Axis See-Through Augmented Reality) Parallax-free video see-through HMD Optical see-through HMDs Virtual Vision VCAP Sony Glasstron Strengths of optical AR Simpler (cheaper) Direct view of real world Full resolution, no time delay (for real world) Safety Lower distortion No eye displacement

5 Strengths of video AR True occlusion rather than composited as in optical Digitized image of real world Flexibility in composition Matchable time delays More registration, calibration strategies Wide FOV is easier to support Video Monitor AR Graphics Video cameras Video Monitor Combiner (Stereo glasses) Head Mounted Displays (HMD) Display and Optics mounted on Head May or may not fully occlude real world Provide full-color images Considerations Cumbersome to wear Brightness Low power consumption Resolution limited Cost is quite high? Maintenance especially Brains and Bricks AR interface for visualizing sensor data Using portable video see-through device Commonly available technology. A mobile phone. The Virtual Retinal Display Projector-based AR User (possibly head-tracked) Image scanned onto retina Commercialized through Microvision Nomad System - Real objects with retroreflective covering Projector Examples: Raskar, UNC Chapel Hill Inami, Tachi Lab, U. Tokyo

6 Example of projector- based AR Head Mounted Projector Ramesh Raskar, UNC Chapel Hill Head Mounted Projector Jannick Rolland (UCF) Retro-reflective reflective Material Potentially portable Projection screen AR Place static (angled?) glass screen (window) between user and real world Project on screen with (angled?) displays Align displayed objects with real world by tracking user s s head Or by other means? AR Tracking Projection Screen AR Real objects Virtual object Window User (possibly head-tracked) Projector The importance of tracking Tracking is the basic enabling technology for Augmented Reality Realistic merged real-virtual environment Tracking is significantly more difficult in AR than in Virtual Environments Greater precision is required Latency can not be permitted.

7 Sources of registration errors Static errors Optical distortions Mechanical misalignments Tracker errors Incorrect viewing parameters Dynamic errors System delays (largest source of error) For an arms length display: 1 ms delay ~ 1/3 mm registration error Markers Can look like anything Can be attached to anything May not be visible in the scene: Video see-through can overpaint them Must be easily identified Must be distinct and clearly orientable Types of Trackers Mechanical Armature with position sensors Electromagnetic AC or DC field emmitors/sensors Compass Optical Target tracking (LEDs( LEDs,, beads) Line of sight, may require landmarks to work well. Computer vision is computationally-intensive intensive Acoustic Ultrasonic Inertial & dead reckoning Acceleration and impulse forces Sourceless but drifts GPS Outdoor Augmented Reality Accuracy not great Line of sight, jammable Hybrid Fiducial tracking Since we have a real world and (often) a video capture of it We can use the real world to track: Use object tracking (hard) or Use fiducial markers to provide position, scale and orientation information Natural Feature Tracking Goal: Overlay virtual imagery onto normal printed material (maps, photos, etc) Method: AR registration based on matching templates generated from image texture Hard to do reliably and *generally* Markers are easier and more reliable

8 ARToolKit Enabling technology Library for vision-based AR applications Open Source, multi-platform Solves two significant problems in AR Tracking Interaction Overlays 3D virtual objects on real markers Uses single tracking marker Determines camera pose information (6 DOF) ARToolKit Website ARToolKit Coordinate Frame Hardware Camera 320x240+ Computer Pentium 500Mhz+ 3D graphics video card Video capture card HMD (optional) Video see-through or Optical see-through Binocular or Monocular Tangible AR Coordinate Frames Typical ARToolKit System ARToolKit Tracking Pentium 4 2Ghz - $1000 GeForce4 Graphics - $200 Hauppauge WinTV capture card - $50 Marshall Board CCD Camera - $200 Sony Glastron PLM-A35 - $400 VGA to NTSC converter - $100 Total Cost ~ US$1950 ARToolKit - Computer vision based tracking libraries

9 Tracking Limitations Computer vision based Camera pose found only when marker is visible Shadows/lighting can affect tracking Tracking range varies with marker size Tracking accuracy varies with marker angle Tracking speed decreases with the number of visible markers AR interfaces as context based information browsers Information is registered to real-world context Hand held AR displays Video-see see-through (Rekimoto( Rekimoto,, 1997) Magnetic trackers or computer vision Interaction Manipulation of a window into information space Applications Context-aware information displays An ARToolKit Application Basic Outline Step1. Image capture & display Step2. Marker detection Step3. Marker identification Step4. Getting 3D information Step5. Object Interactions Step6. Display virtual objects AR Interfaces as 3D data browsers 3D virtual objects are registered in 3D See-through HMDs,, 6 DOF optical, magnetic trackers VR in Real World Interaction 3D virtual viewpoint control Applications Visualization, guidance, training 3D D AR Interfaces AR Interaction Virtual objects displayed in 3D physical space and can be freely manipulated See-through HMDs and 6DOF head- tracking are required 6DOF magnetic, ultrasonic, etc. hand trackers for input Interaction Viewpoint control Traditional 3D user interface interaction: manipulation, selection, adding, removing, etc. Kiyokawa, et al. 2000

10 Augmented Surfaces Images are projected on a surface back or overhead projection Physical objects are used as controls for virtual objects Tracked on the surface Virtual objects are registered to the physical objects Physical embodiment of the user interface elements Collaborative Tangible AR: Time- multiplexed interaction Use of natural physical object manipulations to control virtual objects VOMAR Demo Catalog book: Turn over the page Paddle operation: Push, shake, incline, hit, scoop Tangible AR: Generic Interface Semantics VOMAR Interface Tiles semantics data tiles operation tiles menu clipboard trashcan help Operation on tiles proximity spatial arrangements space-multiplexed Space-multiplexed Interface Data authoring in Tiles Lessons Learned Face to face collaboration AR often preferred over immersive VR AR facilitates seamless/natural communication Remote Collaboration AR spatial cues can enhance communication AR conferencing improves video conferencing Many possible confounding factors

11 Promising Research Directions Natural Feature Tracking Outdoor AR UI Design Other Modalities HMD Design AR on Everyday Devices