Augmented Reality Tactile Maps for Pre-Journey Learning

Transcription

1 Augmented Reality Tactile Maps for Pre-Journey Learning Ryosuke Ichikari and Takeshi Kurata National Institute of Advanced Industrial Science and Technology (AIST), Japan 24 March 2016, 11:00 CSUN Conf. San Diego, CA 1

2 Walking Assists for the visuallyimpaired people Assisting technologies when they are walking White cane Guide dog Assisting technologies with computer Remote Sensing by RGB(-D) cameras or LRF Voice navigation system based on localization technology such as GNSS Pre-Journey learning and re-viewing before/after walking Tactile Map 2

3 Tactile Map Maps with tactile presentation for the visually-impaired people UV offset printing:printing transparent plastic on the color maps PIAF(Picture in a flash):using heat sensitive papers which can swell black-printed area (lines/images) UV Offset printing PIAF 3

4 Problems about tactile map Difficulties for constructing mental image of the contents with haptic sense Needs experience and training Limitation of the amount of contents Production issues Cost: UV offset printing is expensive Design issue Needs know-how to design contents Tactile Map Automated Creation System (TMACS) [Watanabe et. al, ICCHP2014 ] 4

5 Proposed system:utilizing AR for tactile map Augmented Reality; AR Seamless integration of real world and virtual world Visual augmentation is main stream, but AR concept can be applied for audio sense AR Tactile Map(Proposed System): Audio-visual AR [Higa et. al 2007] Provides audio-visual feedbacks in response to interactive operations Enables dynamic and multimodal contents Can be utilized for learning tactile map 5

6 Requirements for AR tactile map Defined requirements by hearing to the visuallyimpaired people Augmentation for the tactile map by audio representation (for completely blind people) Enlarging/enhancing display for people with week sight Providing POR(Point of reference) information Helping comprehension of the tactile map with interactive training Intuitive interaction with the system Simple and inexpensive configuration with off-the-shelf products Hands free arrangement 6

7 Augmentation for the tactile map by audio representation Adding audio representation makes the information more reachable The amount of the contents by the dynamic audio representation is not limited by physical tactile map Dynamically Providing information according to focused points and surrounding area 7

8 Enlarging/enhancing display for people with week sight The people with week sight might perceive map if the contents enlarged/enhanced. Image modifications in responding to the users potential modifications: optical zoom, edge enhancement, contrast adjustment, color adjustment 8

9 Providing POR(Point of reference) information Target contents of the AR tactile map: POI(Point of Interest)and POR(Point of Reference) POR:Specific point location where it is easily recognized for confirming routes such as characteristic shape and material of ground (steps, stairs, sloop, door), sound/noise, and scent/odor) [Okuno et. al, CSUN2015] [Denoncin,CSUN2014] [Theberge, CSUN2015] Integration with POR/POI data base Voice navigation system with POR/POI data base [Okuno et. al, CSUN2015] Utilizing as the communication tool between visually-impaired people and sighted people at Mapping Party 9

10 Intuitive interaction with the AR tactile map Interactivity is important for learning tactile maps and dynamic representation of the contents Trial-and-error with interactive system is short cut for learning tactile map Inputting interface by actually touching tactile maps Most intuitive and the simplest Perceiving and inputting at the same time Needs functions for finger gesture recognitions 10

11 Requirements of system configuration Adopting off-the-shelf products for availability perspective For widely promoting the system Consists of only simple and inexpensive products Enabling hands-free arrangement From the visually-impaired people opinions For touching as many as possible points of the tactile map 11

12 System configuration System configuration of the prototype system Tablet PC For Visual-Audio feedbacks RGB-D camera:depthsense DS325 (ToF, Range:15cm ~100cm) For finger gesture recognition and robust extraction of hands area, measurement of the distance between the map and finger PC for recognition Arm for fixing tablet Tablet mount parts 12

13 Related Works Image to Sound Converter [Kuroda] Image is converted to sound by changing key and difference of L/R sound volume according to the pixel s coordinate Transparent tactile sheet can be overlaid Display size and UI are limited by touch display Kiko-AR by cybernet systems Mobile AR system based on Metaio junaio with audio presentation Image recognition and text-to-sound Collaborating with a printing company to produce printed material with tactile presentation and AR augmentation. Need to shoot with smartphone s camera Characteristics of AR Tactile Map: - Various sizes and types of tactile maps can be used - Intuitive interface for the system are focused 13

14 RGB-D camera Fixed on backside Recognition Finger points tracking Gesture recognition Gesture command, Finger points, lon/lat Client side AR tactile app Implementations Recognition of Tactile Map Tracking Position Of the Tactile Map ID of Map Lat/Lon POR/POI data Server Parts POR/POI management parts Dababase POR data Tablet PC Temporal POR database Camera Image POR data Visual Feedback(AR, Enlargement/Enhancement) Accepting finger interaction Audio feedback (POR/POI) POR/POI, Lat/Lon data POR/POI data collection app. 14

15 Recognition of gesture Recognizing Tapping, Double Tapping, Searching gestures as the interaction commands Algorithm for detecting Tapping Extracting hand area by RGB-D camera with background differencing of depth image and distance threshold Detecting state transition within time window (1 2 1) Distance between map and finger Size of time window 2 1 1Finger on the map 2Finger in the air for tapping time 15

16 Variety of finger gestures Tapping For getting user s focused points in the map at the intended timing Equivalent to Clicking of mouse Double Tapping For another command Equivalent to Double Clicking of mouse Searching For continuously getting user s focused area Equivalent to Moving Cursor of mouse 16

17 Converting coordinate from the map coordinate to latitude/longitude Tapping points are detected by counting pixels with the patterns and calculating center of gravity Estimated points are on the coordinate of RGB-D image, they are required to be converted to latitude/longitude RGB-D images are rectified by perspective transform with known points of corners. Pairs of points between the RGB-D image coordinate and coordinates on latitude /longitude for the final conversion At least 3 pair between map coordinate and Latitude/longitude 17

18 Automatic identification and tracking of tactile map Originally defined manually. ORB Feature point detector/local feature descriptor is used for identifying tactile map by RGB image Comparing mean distance of feature descriptors Corners of the tactile map can be tracked by estimating homography matrix between rectified image templates and input image At lees 4 pairs are required for calculating homography matrix RANSAC is used for robustly calculating homography matrix ZNCC(Zero mean normalized cross correlation) is used for verifying similarity of the contents in the rectangle. 18

19 Result of the automatic recognition and tracking 19

20 Enlarging the map with text-to-speech reading as the reaction of Tapping gesture Entrance Convenience Store Park Museum

21 Mapping party fo People are gathered for making map (Mapping Party) Previewing by printing as tactile map Walking assistance by voice navigation

22 Virtual Mapping Party with AR Tactile Map Panoramic image and 3D-Sound Recording Mapping Party Anytime, Anywhere Pre-learning of the walking route with the AR Tactile Map Requests for Mapping Utilized at voice navigation

23 Workshop on Virtual Mapping Party Held at Miraikan, Tokyo, Japan March 18 th -19 th, h x 6 times Collecting POR/POI by experiencing panoramic images and 3D sounds with HMD(Oculus, GearVR. Etc.), PC and headphone Confirming and requesting POR/POI by AR tactile Map

24 Watching panoramic image with 3D sound for collecting POR/POI

25 Usage of the AR tactile map for virtual mapping party Allowing the visually-impaired people to join the mapping party by gesture Search: Confirming POR/POI on the tactile map with sound for telling existence of POR/POI where user touches Tap: Confirming POR/POI with Text-to-Sound when user taps the specific point Double Tap: Requesting the POR/POI for the specific points of the map

26 Summery of a questionnaire for the AR tactile map Comments about gesture interface It is fun to search POR/POI by actually touching. It took time to get used to double-tap, but finally I mastered. It s good concept, but needs more accuracy of detection. It lacks feedbacks to tell whether the system detected gesture or not. It was difficult to use. Other techniques can be used for more robust detection of gesture. Touch panel with tactile presentation(sheets?) can be used for robustly detecting gesture. It is hard to distinguish between detection error and nothing there to be displayed.

27 Summery of a questionnaire for the AR tactile map (cont.) Comments about requesting function Requesting function is good because the it can reflect visually-impaired people s demand. Filtering technique might be required for extracting important requests It s better to add function to send request by voice recognition. Currently only points of the request are defined, contents of the request might be needed (Just want to get POR/POI or feel scared) Comments about the contents It seems helpful for the visually-impaired people to check obstacles such as trees.

28 Summery of a questionnaire for the AR tactile map (cont.) General Suggestions and Comments Scale size of the map is not fitted. Considering to accuracy of the gesture recognition, the tactile maps should be bigger in this case. It seems interesting if it can realize dynamic tactile presentation in some day. In addition to requesting, other opportunities for the visually-impaired people to join the events seem to be needed. Functions to increase the number of participants such as position based SNS seem to be required.

29 Conclusions AR tactile map: Audio-visual augmentation for tactile map by RGB-D image recognition and text-to-speech. Audio-Visual feedbacks by AR for dynamic presentation Intuitive gesture interface by using RGB-D camera Future works Increasing accuracy of the gesture recognition Covering individual difference of gesture Noise and unclear situation (sunlight, close to paper) Virtual Mapping Party with the AR tactile Map Acknowledgement This research is conducted as a part of the project sponsored by JST RISTEX. 29

30 Thank you! Ryosuke Ichikari, Ph.D. Postdoctoral Researcher at AIST Contact: 30

31 31

32 プロトタイプでの 実 装 の 詳 細 Client 側 AR 触 地 図 アプリ Androidアプリとして 実 装 カメラ 映 像 を 用 いた 触 地 図 のIDとカメラ 触 地 図 間 の 位 置 関 係 の 認 識 ユーザの 注 目 箇 所 をジェスチャ 認 識 部 とUDP 通 信 音 声 提 示, 視 覚 提 示 ( 拡 大 ) 等 で 情 報 提 示 指 ジャスチャ 認 識 プログラム Windowsアプリとして 実 装 DepthSenseSDKを 用 いてRGB-Dカメラ 映 像 を 処 理 指 先 の 位 置 の 座 標 とコマンド 動 作 を 認 識 位 置 を 緯 度 経 度 情 報 へ 変 換 してClientアプリに 送 信 32

33 サイトワールド2015でのデモ デモの 様 子 33

34 デモ 展 示 等 で 判 明 した 課 題 カメラ 触 地 図 間 の 位 置 関 係 がずれる アームに 意 図 せず 接 触 自 動 的 なキャリブレーション 方 法 の 必 要 性 タップ 動 作 の 個 人 差 の 問 題 個 人 のタップ 動 作 をシステムへ 登 録 する 必 要 性 34

35 関 連 事 例 画 像 聴 覚 化 装 置 *( 画 像 聴 覚 化 研 究 所 ) きっかけ: 視 覚 障 害 者 に 美 術 館 で 絵 画 を 体 感 できるように 音 程, 音 が 聞 こえる 方 向 により 画 像 を 聴 覚 化 触 図 シートを 重 ねることで, 触 覚 も 併 用 可 能 AR 触 地 図 : 触 地 図 の 自 由 度, 簡 素 な 機 器 構 成, 直 観 的 インタフェースを 重 視, 画 像 聴 覚 化 装 置 の 音 自 体 に 位 置 情 報 を 関 連 付 けることは 参 考 になる [* 出 展 : 画 像 電 子 学 会 第 7 回 VHIS 研 究 会 (2015 年 )] 35

36 AR 触 地 図 とのインタラクション RGB-Dカメラを 用 いて 触 地 図 を 触 る 動 作 を 認 識 前 提 : 触 地 図 とRGB-Dカメラの 位 置 関 係 は 真 上 から の 撮 影 位 置 で 固 定 背 景 差 分 が 適 用 可 能 距 離 チャネル(D:depth)を 用 いることで 触 地 図 の 色 や 手 の 肌 の 色 に 影 響 されない 認 識 触 地 図 と 手 の 接 触 の 有 無 を 認 識 36