A MARKER-LESS AUGMENTED REALITY SYSTEM USING IMAGE PROCESSING TECHNIQUES FOR ARCHITECTURE AND URBAN ENVIRONMENT

Transcription

1 S. Chien, S. Choo, M. A. Schnabel, W. Nakapan, M. J. Kim, S. Roudavski (eds.), Living Systems and Micro-Utopias: Towards Continuous Designing, Proceedings of the 21st International Conference of the Association for Computer-Aided Architectural Design Research in Asia CAADRIA 2016, , The Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong. A MARKER-LESS AUGMENTED REALITY SYSTEM USING IMAGE PROCESSING TECHNIQUES FOR ARCHITECTURE AND URBAN ENVIRONMENT YUSUKE SATO, TOMOHIRO FUKUDA, NOBUYOSHI YABUKI, TAKASHI MICHIKAWA and ALI MOTAMEDI Osaka University, Suita, Osaka, Japan sato@it.see.eng.osaka-u.ac.jp, {fukuda, yabuki, motamedi}@see.eng.osaka-u.ac.jp, michikawa@ceids.osaka-u.ac.jp 1. Introduction Abstract. In this study, Augmented Reality (AR) system is proposed to be used for outdoor renovation and maintenance projects of buildings. The research proposes an outdoor marker-less AR system that considers the mobility of users and their long relative distance to target buildings where 3D virtual objects should be augmented on. The proposed system uses local feature-based image registration technology and Structure from Motion (SfM) which reconstructs 3DCG models using photographs from multiple viewpoints. A case study has been performed for a research building renovation scenario at Osaka University. The case study verified the performance of image registration and tracking, and confirmed the applicability of the method. Keywords. Architecture and urban environment; Augmented Reality (AR); image registration; Speeded-up Robust Features (SURF); Structure from Motion (SfM). In recent years, in order to reduce carbon footprint, renovating existing buildings (for improving safety and strength or, adding new functions) has become more desirable than reconstruction. In comparison to reconstruction after demolition, renovation is expected to preserve resources and to reduce generating environmentally hazardous substances (Egbu, 1997; Juan, 2009). Using new materials, which are more environmental friendly, makes buildings more sustainable and reduces environmental burdens. For that, it is ben-

2 714 Y. SATO, T. FUKUDA, N. YABUKI, ET AL. eficial to enable design stakeholders asses final visualizations of their design as they sometimes show hesitation towards new design challenges. The use of Augmented Reality (AR), which can integrate augmentations with real world scenes, has become common in architectural and urban design fields (e.g., Wither et al, 2009; Schubert et al, 2015). AR can help visualizing full scale building projects at the design stage before the construction, and can be used to visualize some properties of the building at the maintenance and operation stage. For that, AR super-imposes 3DCG (3- Dimensional Computer Graphics) model objects on a live video feed of the surrounding landscape acquired by a camera. AR is similar to Virtual Reality (VR) technology. However, it does not require 3D virtual models of the environment. With the use of the AR system, the time and the cost required to perform a 3DCG modelling of present surroundings are drastically reduced. However, considering possible lack of familiarity of architectural designers with the Information Technology (IT) tools, it is necessary to build simple, easy-to-use, and low-cost system which its usability does not rely on designer s IT skills. In this study, AR system is proposed to be used for outdoor environment specifically for renovation projects at the design stage. In order to realize an accurate outdoor AR, precise tracking of geometric registration between a live video and 3DCG models still remains as an important challenge. In the case of an outdoor AR, the relative distance between AR users and visualized 3D models tends to be large. The reference point for geometric registration is the sensor s position in sensor-based methods (such as GPS or 3D sensor) (Watanabe, 2011), or the position of an artificial marker in markerbased methods (which is an example of vision-based methods) (Yabuki et al, 2011). In sensor-based methods, the reference point of geometric registration is positioned near an AR user, hence, the sensing error highly influences the error of a 3DCG model registration. Meanwhile, in artificial marker-based methods, it is possible to place large markers near target objects. However, the marker needs to be properly captured by the AR camera, which imposes limitations on the user s movability range and requires installation of large markers. Therefore, development of marker-less AR systems, which do not require artificial markers, continues to advance (for example, Klein and Murray, 2007; Ventura and Hollerer, 2012). Yabuki et al (2012) developed a geometric registration method which corresponds point cloud data to natural feature of the real world. However, in this method, natural features which are used for geometric registration need to be continuously captured by the AR camera, and special equipment is required for the acquisition of point cloud data as the system needs to use a large number of point cloud data acquired by a

3 A MARKER-LESS AUGMENTED REALITY SYSTEM USING 715 3D laser scanner. In contrast, Structure from Motion (SfM) technique, which reconstructs 3D shapes from photographs taken from multiple viewpoints, is another method to create point cloud data (Hartley and Zisserman, 1994). It is an emerging, easy, and low-cost alternative method for creating models (Nagakura et al, 2015). Therefore, in this study, a novel marker-less AR system that uses local feature-based image registration and SfM technology is developed. The proposed system supports free movement, has less limitations, requires less efforts, and is less expensive for outdoor AR applications. The developed system has been applied to a renovation design project for verification. One of the features of the proposed system is that it does not require special equipment (such as sensors for geometric registration between augmentations and the real world) as it uses sensor-based registration. Additionally, the system does not require artificial markers which decreases user s flexibility. Finally, the point cloud data which is used in this system is reconstructed by SfM and does not require special hardware such as laser scanners. 2. System overview The proposed method has two main steps; preprocessing, and the real-time processing. In the preprocessing step; first, a 3D model of the environment is reconstructed by the SfM method using photographs taken from multiple viewpoints. The position and orientation data of all viewpoints are stored in a database. Second, coordinates of 3D virtual objects that should be augmented are defined relative to coordinates of 3DCG models of the environment reconstructed by SfM and are stored in a database. Finally, key points and features of each photograph used for the SfM are extracted and saved in a text file. Additionally, file paths of all photographs are saved in a separate text file. In this research, an advanced local feature detector and descriptor namely Speeded-Up Robust Features (SURF) is used (Bay et al, 2008). SURF is an improved and speeded up version of Scale-Invariant Feature Transform (SIFT) that is used for interest point detection. In the real-time processing step; first, the files created in the preprocessing step are imported. Second, features of the live video images are extracted using SURF in real time, and extracted features of the live video images are compared with features of stored images in the database. Finally, 3D virtual objects are precisely rendered in an AR display using position and orientation data of the camera and by finding the most similar image in the database. For tracking, motion vectors are calculated using optical flow. Then, external parameters are calculated using internal parameters, position

4 716 Y. SATO, T. FUKUDA, N. YABUKI, ET AL. of points on the screen and the world coordinates of their corresponding points. To realize this processing in real-time, three speed enhancement techniques are employed. In the SURF processing, for comparing key features between live video images and all stored images in the database, approximate nearest neighbor search by Locality Sensitive Hashing (LSH) algorithm (Indyk and Motwani, 1998) is applied instead of full search by linear search algorithms. And, a multithreading technique is applied when the features are extracted in the SURF processing. The live video images are also trimmed to process only their central area for the image registration processing step. The overall view of the proposed system is shown in Figure 1, and the system flow is shown in Figure 2. Figure 1. Overview of the system. Figure 2. Marker-less AR system process MARKER-LESS AR SYSTEM DESIGN 3D model is shown on the screen using Equation (1) (Zhang, 2000). In the equation, s denotes expansion or reduction factor, u and v are the position of

5 A MARKER-LESS AUGMENTED REALITY SYSTEM USING 717 object drawing on the screen coordinates. The value f denotes the focal length, and c x and c y define coordinates of the centre of the image. These values are called internal parameters of a camera. R is rotation matrix and T is translation matrix of a camera. These values are called external parameters of a camera. The values X, Y and Z denotes the position of object in the world s coordinates.! s " u v 1 $! = % " f 0 c x 0 f c y $! R 3 3 T ( 3 1 ) % " Internal parameters are possible to be calculated through camera calibration, and remain unchanged unless the camera lens is changed. In contrast, external parameters represent the position and posture data of a camera, and are specific to each photograph s viewpoint. Therefore, external parameters need to be calculated while using the AR system. In case of maker-based AR systems, these parameters are calculated based on the position of a marker, however, in case of marker-less AR system, another calculation method is required. Accordingly, in this study, external parameters of each photographs viewpoints are calculated by the SfM SYSTEM IMPLEMENTATION In this study, the proposed system was implemented using C++ and by combining multiple open source software applications and libraries. In the preprocessing, OpenMVG (Open Multiple View Geometry. Ver. 0.7) which is an open source software was used for carrying out SfM. Additionally, Trimble SketchUp (Ver ), which is a 3D modelling software, was used for defining coordinates of 3D virtual objects. In the main processing, OpenCV (Open Source Computer Vision Library. Ver ), which is another open source image processing library, was used for capturing real-time camera stream and image matching. Additionally, TBB (Threading Building Blocks. Ver. 4.4), which is an open source parallel processing library, was used for parallel processing. FreeGLUT (Ver ), which is toolkit for OpenGL (Open Graphics Library), was used for drawing 3DCG model on AR display. 3. System verification To verify the applicability of the proposed system for renovation projects, a verification experiment was conducted for an existing building. The research X Y Z 1 $ % (1)

6 718 Y. SATO, T. FUKUDA, N. YABUKI, ET AL. building M3 of Osaka University Suita Campus was selected as the target. The experiment was carried out on November 26 th and 27 th, PROTOTYPE EXPERIMENTATION For the renovation case study, the design of awning (sunshades) for a part of M3 building is chosen. In the AR scene, the prototype system allows users to change the material of to-be-installed awnings (sunshades). As for design alternatives, blue and white colour materials and different angles (10, 20 and 30 degrees) for shades were prepared. A sample photo of the experiment environment is presented in Figure 3. The procedure for the verification study is presented in the following steps: 3D shape of M3 building was reconstructed by SfM. In this study, SfM processing was carried out using images captured by a digital camera. One hundred images (resolution ) of M3 building were used for SfM. Moreover, drawing position of the awning was assigned to the reconstructed 3D shape. The photography route for SfM is presented in Figure 3. Examples of images used for SfM is presented in Figure 4 and 3D shape of M3 building reconstructed from SfM are presented in Figure 5. Drawing position of awnings is presented in Figure 6. Initializing marker-less AR system including M3 building contents was confirmed. The display of AR models on the same route where the images are captured for SFM was confirmed. Changing colour of awning material, and changing the angle of awnings were confirmed. Tracking 3DCG model on the same route where the images are captured was confirmed. The application of the developed marker-less AR system for building renovation was verified. Figure 3. Experimental photo (left), Photography route of SfM and Tracking route of experimentation (right).

7 A MARKER-LESS AUGMENTED REALITY SYSTEM USING 719 Figure 4. Examples of images used for SfM. Figure 5. 3D Model reconstructed from SfM. Figure 6. Drawing area of 3D virtual models (left), Photomontage: Ideal image of drawing 3D virtual models (Sunshades: 3D virtual models; Others: Photo) (right).

8 720 Y. SATO, T. FUKUDA, N. YABUKI, ET AL RESULT AND EVALUATION The experiment verified that the developed system performs registration, tracking at the photography route, and changing colour and installation angle of sunshades. The result of feature-based matching with SURF between live video image and stored images in the database is presented in Figure 7. The results of registration and changing colour and installation angle by typical two patterns are presented in Figure 8. The continuous AR screen shot of tracking while moving on the route is presented in Figure 9. Although developed marker-less AR system was able to register and track, visualizing error occurred in the registration process. The average error of AR drawing position were 14.5 pixels in the horizontal direction and 27.8 pixels in the vertical direction. In addition, the ratio of error for the AR display were 1.6 % in the horizontal direction and 4.6 % in the vertical direction. In comparison with marker-based AR systems, the proposed marker-less AR system has two main advantages: (1) installation of artificial markers which affect the landscape view is not required; (2), the user does not require to constantly capture an artificial marker while tracking. Hence, it provides users with more freedom of movement LIMITATIONS The proposed system is currently only capable of performing geometrical registration in the photography route. To solve this issue, new photography routes can be added to expand the coverage area for the system. Additionally, 3D virtual objects are sometimes shaking on the screen because of movement of the user s hand while tracking. In order to compensate user s hand shaking while tracking, the average of motion vectors in plural frames when calculating external parameter can be used. It is necessary to develop and implement new algorithms for such enhancement. Figure 7. Example of feature-based matching with SURF: live video image (left) and stored image (right).

9 A MARKER-LESS AUGMENTED REALITY SYSTEM USING 721 Figure 8. AR screen shots of changing colour and installation angle patterns. Figure 9. AR screen shots of tracking while user s walking: AR display for users (above) and motion vectors (below). 4. Conclusions and Future Work In this research a marker-less AR system with local feature-based image registration technology and SfM, which reconstruct 3DCG model by using photographs from several viewpoints, was developed. The developed AR system has been verified in a case study. In the case study of visualizing sunshades on M3 building located in Osaka University Suita Campus, registration, tracking, changing colour and installation angle of sunshades were tested. The study showed that it is necessary to precisely identify positional relationship between 3D virtual objects and real world scenes, and to improve the precision of registration and tracking as future work. In addition, it is possible to use this AR system at only the route in which photographs for SfM are taken. Hence, when using the AR system in other routes, camera pa-

10 722 Y. SATO, T. FUKUDA, N. YABUKI, ET AL. rameters are required to be adjusted. Further studies are needed in order to solve these challenges. Acknowledgement This research has been partly supported by JSPS KAKENHI Grant Number References Bay, H., Ess, A., Tuytelaars, T. and Cool, L.: 2008, Speeded-up robust feature (SURF), Computer vision and Image Understanding, 110, Egbu, C. O.:1997, Refurbishment management: challenges and opportunities, Building Research and Information, 25(6), Hartley, R. I., and Zisserman, A.: 2004, Multiple View Geometry, Cambridge University Press, Cambridge, UK. Indyk, P. and Motwani, R.: 1998, Approximate nearest neighbors: towards removing the curse of dimensionality, In Proceedings of the 30 th annual ACM Symposium on Theory of Computing, STOC 98, Juan, Y. K.: 2009, A hybrid approach using data envelopment analysis and case-based reasoning for housing refurbishment contractors selection and performance improvement, Expert System eith Applications, 36(3), Klein, G. and Murray, D.: 2007, Parallel Tracking and Mapping for Small AR Workspaces, ISMAR '07 Proceedings of the th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nagakura, T., Tsai, D. and Choi, J.: 2015, Capturing history bit by bit architectural database of Photogrammetric Model and Panoramic Video, In Proceedings of the 33rd ecaade Conference, Vol. 1, Schubert, G., Schattel, D., Tönnis, M., Klinker, G., Petzold, F.: 2015, Tangible Mixed Reality On-Site: Interactive Augmented Visualisations from Architectural Working Models in Urban Design, Communications in Computer and Information 527, Ventura, J., Hollerer, T.: 2012, Wide-area scene mapping for mobile visual tracking, In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, Watanabe, S.: 2011, Simulating 3D Architecture and Urban Landscapes in Real Space, Springer-Verlag, Berlin. Wither, J., DiVerdi, S., Hollerer, T.: 2009, Annotation in outdoor augmented reality, Computers Graphics 33, Yabuki, N., Miyashita, K., Fukuda, T.: 2011, An invisible height evaluation system for building height regulation to preserve good landscapes using augmented reality, Automation in Construction, 20(3), Yabuki, N., Hamada, Y., Fukuda, T.: 2012, Development of an accurate registration technique for outdoor augmented reality using point cloud data, 14th International Conference on Computing in Civil and Building Engineering. Zhang, Z.: 2000, A flexible new technique for camera calibration, IEEE Trans. Pattern Anal. Mach. Intell., 22(11),