Augmented Reality for Museum Artefact Visualization

Proc. 4th Irish Workshop on Computer Graphics, Eurographics Ireland Chapter, Coleraine, Northern Ireland, 29-30th April, 75-80, (2003). ISBN: 1649-1807. Augmented Reality for Museum Artefact Visualization M. White, F. Liarokapis, J. Darcy, N. Mourkoussis, P. Petridis, P. F. Lister University Sussex, Engineering and Information Technology, Centre for VLSI and Computer Graphics, Falmer, BN1 9QT, UK Abstract This paper describes an experimental augmented reality based system for overlaying computer generated information on the real world. Museum artefacts are digitised; 3D models are then created and rendered in an augmented reality environment providing the opportunity for museum visitors to visualise virtual artefacts in the context of real artefacts and other contextual information. Viewing 3D models, images and text in the same augmented environment enhances the museum visitor s experience. Our work is focused on three areas: photo-realistic and efficient 3D modelling of museum artefacts; on the description of the artefacts and their digital representations through the use of XML based on metadata standards; and on the rendering and mixing of real objects with synthetic ones in real time performance. The potential of our system is illustrated through a simulation scenario developed on an experimental AR tabletop laboratory environment. Categories: Virtual and augmented reality, computer graphics, XML, museum artefacts, 3D modelling. 1. Introduction Most museums hold countless archives or collections of artefacts, which they cannot exhibit in a low cost and easy way; typically museums simply do not have the space to exhibit. Our system allows museums to digitise artefacts, create 3D models, manage the resulting digital information using XML and visualise this information in an augmented reality (AR) environment. Museums can digitise their collections in several ways: using state-of-the-art photogrammetry software 1, hardware photogrammetry systems 2, laser techniques 3, or manually using 3D Modelling software such as Maya or 3ds max. Photogrammetry and laser based modelling currently represents cutting-edge research and manual 3D modelling is extremely labour intensive requiring great expertise. The ARCO project is focusing on developing simpler photogrammetry and manual modelling techniques focused on the museum end user with little experience. In this paper we discuss a manual modelling process using 3ds max that is aimed at the inexperienced museum user. We also describe how digital representations of the museum artefacts are created and managed through the use of EXtensible Markup Language (XML) based on metadata standards, and finally we describe how the museum user can visualise the digital representations by mixing real artefacts with virtual models and other digital information in real time AR presentations. Augmented reality can be defined in two different ways 4. The first definition refers to the class of display systems that is based on Head Mounted Display (HMD) 5. The second definition refers to systems that utilize an equivalent of an HMD, thus encompassing both large screen and monitor-based displays 6. Our prototype system is mainly focused on developing an AR environment based on large screen displays because this allows multiple museum visitors to experience the visualisation. HMDs are restricted to single user are generally too expensive for this type of application, but the system does not restrict their use. A simple interface is developed to experimentally explore the

potential of AR in a museum environment by mixing digital information in an AR tabletop environment with real objects. The museum user can interact with the AR environment by moving predefined markers that are assigned to virtual artefacts, writing on the table top, and using the keyboard and mouse. 2. Related Work Several European Union research projects have been undertaken in the field of virtual heritage. The ARCO 7 project is developing a system for museums that allows museums to create virtual exhibitions on the web and includes components for creating digital or virtual representations of museum artefacts, object relational database management system, and generation of dynamic VRML or X3D content in virtual and augmented environments. The 3D Murale 8 project is developing and using 3D multimedia tools to measure, reconstruct and visualise archaeological ruins in virtual reality using as a test case the ancient city of Sagalassos in Turkey. Media and textual information about archaeological content is stored in a database. This content is structured by metadata information. Metadata information will make this content available by remote Internet access through the use of search engines for archaeological researchers and members of the public. The Charismatic 9 project focused on generating tools to allow the definition of virtual environments that represent reconstructions of large geographic areas. Virtual people and events are recreated to offer users a creative and entertaining way to enjoy heritage as well as preserve it. The Seattle Art Museum Project 10 was developed to demonstrate a virtual dig experience, based on two software packages, Hi-Space 11 and ARToolKit 12. This project allows a persons hand or hand held objects to be tracked over a table using an overhead camera. While users move a real brush over the table surface, the location of the hand and brush is found and used to change the projected image using IR reflected light. The Archeoguide 13 project provides an interactive AR guide for the visualization of archaeological sites. The system is based on mobile computing, networking and 3D visualization providing the users with a multi-modal interaction user interface. The European Museums' Information Institute (EMII) 14 project aims at facilitating on-line access to the cultural heritage of Europe's museums, creating long term partnerships between museums throughout Europe and finally at providing a European focus for international initiatives. This paper discusses some initial experimental work in 3D modelling of museum artefacts, describing digital artefacts in XML and visualising using augmented reality. The main objective of this experimental work is to enable us to make judgements on what functionality White et al / Augmented Reality for Museum Artefact Visualization is possible and can be implemented in the ARCO system, i.e. what will work and what will not work. 3. System Components Our experimental system has been implemented with of-the-self hardware components based around an HP workstation with two 2.4GHz Xeon processors and 1024 MB of memory. The workstation or PC is equipped with a Wildcat 6210 graphics card, a Hauppage WinTV-PVR video capture card, and a low cost bullet camera. Our system also uses standard display technologies such as PC monitors or projectors to provide a cost effective method of presenting a simple spatial AR application, or a relatively cheap CyVisor HMD for a single user immersive experience. A goal is to demonstrate that PC based augmented reality architectures can be constructed with relatively cheap components. In this case the hardware just described cost: HMD 1,500, Bullet Camera 90, Workstation 6,000 (since reduced to 4,000). These sort of costs are affordable for small regional museums such as our partners in ARCO, Sussex Archaeological Society who own and manage six regional museums in Sussex, England. The software architecture is built around several main components: the ARToolKit tracking library 12, our own custom OpenGL rendering engine, XML schemas, and 3ds max scripts to aid the modelling processes. Threedimensional objects are generated using a customised version of the high performance software package, 3ds max, where the user interface is customised with scripts for modelling museum artefacts. Virtual models of museum artefacts are rendered using the OpenGL API. The ARToolKit tracking functions grab and decompose the live video into images and calculate the camera calibration parameters in real time in order to correctly overlay the virtual models in the real world tabletop environment. We are also in the process of implementing a Microsoft Foundation Class (MFC) windows library to replace ARToolKit s GLUT. Figure 1 illustrates our experimental software and hardware system components.

White et al / Augmented Reality for Museum Artefact Visualization model so that the virtual models of museum artefacts can be rendered to overlay the real marker. It is also important to make sure the virtual model s coordinate system is coincident with the marker, i.e. make sure the y coordinate is set to zero and the x, z coordinates are not offset from the model to any great degree. Otherwise the model appears to be rendered in space rather than on the marker. Figure 1: Software and hardware system components In addition to developing a manual modelling process based on plug-ins for 3ds max we are also experimenting with a Roland Picza 3D Laser Scanner (cost 7,500) and a MicroScribe G2X (cost 2,500), which will be controlled from plug-ins in 3ds max, thus providing the museums with a unified modelling environment. If using an HMD, a bullet video camera is used to video capture the real world in which the marker is placed. However, for monitor based AR we have also tested a number of USB web cameras (D-Link, Creative WebCam Vision, PC Line) and a Sony Digital Handycam. All cameras were calibrated using the ARToolKit calibration software and different configuration files for each type of camera were extracted. However, calibration in ARtoolKit appears to be not very efficient and we are currently investigating other methods, i.e. Tsai s Camera Calibration approach15. 4. The OpenGL graphics engine performs the actual rendering of the virtual models into the real world, and both real world and virtual models are displayed in an MFC window environment. Various computer graphics techniques can be implemented so that an augmented photo-realistic visualization can be achieved such as, matching virtual lighting to real lighting, texture mapping, environmental mapping, shading, etc. An important feature to add is shadows so that the user can get a more realistic registration of the virtual model in the real world. 5. Creating Virtual Museum Artefacts While our system provides for visualising virtual museum artefacts in an augmented reality environment, museums still have to create the virtual models. This can be a very expensive and tedious process. Thus, part of our research has focused on trying to develop cost effective modelling processes using de-facto standard modelling packages like 3ds max. Realising, that 3D modelling experts are not cheap, we are implementing a 3D modelling tool and processes based on 3ds max, but focused on the museum user. The default 3ds max user interface, see Figure 2, is very complex and quite System Operation The captured video scene is sent to the graphics card after being processed by the Microsoft Vision SDK software. Then the ARToolKit tracking libraries image process to calculate the video camera's position and orientation relative to marker. Each image is searched for square regions. When all the squares in the image are detected, the pattern inside the square is captured and matched against some pre-trained pattern templates for each square. When a match is detected the position of the real video camera relative to the physical marker is calculated. It is important to calibrate the camera as accurately as possible otherwise ARToolKit tends to track any black square in the scene making is difficult to correctly assign virtual models to different markers. Once the markers are tracked properly the video camera real world coordinates relative to the marker are calculated and then translated into the OpenGL camera daunting especially for the novice museum user. Figure 2: Default 3ds max modelling interface We have reduced this screen complexity by applying good interface design and layout techniques16, 17, illustrated in Figure 3.

White et al / Augmented Reality for Museum Artefact Visualization has a polygon count of less than 2,000 polygons, whereas the 3D model of the Bronze-age mortar has over 65,000 polygons. Similarly, a laser scanned artefact can produced hundreds of thousands of polygons if you are not careful. The museum user needs modelling guides to help determine appropriate 3D models for their virtual exhibitions. In terms of suitability for the AR environment where we are looking to achieve refresh rates of at least 25 fps it is clear that the axe head model is more desirable. It is this type of low-polygon 3D model that our customised interface will encourage simply by providing easy access to the tools necessary for features such as mesh optimisation and smoothing, etc. Figure 3: Customised 3ds max modelling interface We also gained practical experience by developing a large set of virtual museum models. Customisation of the 3ds max user-interface is an ongoing process, guided by the need to produce high quality low-polygon models for visualisation in our system, and driven by requirements captured during modelling exercises carried out by actual novice museum users. In this simpler modelling interface we have only exposed the functionality required for this specific modelling task. In this case, the modelling task was to create an interface focused on creating virtual models that exhibited symmetry. For other specific modelling classes other interfaces could be developed, e.g. laser scanning or Microscribe interfaces. A set of about 30 models was created using the simple modelling interface and the development of each model provided valuable feedback into the interface design process. Figure 4 illustrates two of the models created with this simple modelling interface. 6. Describing the Virtual Museum Artefact as Metadata Our experimental system is able to display virtual museum artefacts as 3D text and 3D models in an AR environment. In our experiment system we refer to all the computer-generated information (i.e. text, 3D models) as virtual multimedia content (VMC) 18 and in ARCO simply as media objects. Virtual multimedia content is the digital representation of the museum artefact. The 3D model can also be a 3ds max project type or a VRML model type. An obvious requirement is the need for a multimedia database connected to the modelling and visualisation systems. This is being implemented in ARCO, but not needed in our experimental system. Initially, we are exploring the management of the test models using XML and a file system. Thus, we have developed a simple XML Schema to describe our data model as metadata. Figure 5 illustrates part of our metadata element set that describes the actual museum artefact, which we also refer to as a cultural object. There are several key metadata elements defined that may be used during visualisation by the museum user to present useful information in the AR environment. Figure 4: Bronze-age Axe Head (<2,000 polygons), Medieval mortar (>65,000 polygons) A crucial aspect in our 3D modelling process is the size in terms of actual polygons of the final model to be used in the AR environment. By creating our own customised modelling interface, it is possible to include tools and functionality that will aid the model creator in the production of 3D models with low polygon counts. For example the 3D model of the Bronze-age axe head

White et al / Augmented Reality for Museum Artefact Visualization real objects representing museum artefacts are placed (in this case the real artefact is a painted wooden carving of a native American Indian), and the user arbitrarily places a marker. Figure 5: XML Schema of the Cultural Object 19 Metadata is defined as the structured description of a data object 20. The structure of the descriptive record varies according to the data model one describes and the domain where the development system applies. There are numerous international metadata standards, projects and initiatives that are used in the cultural heritage community, our system has adopted elements from several key standards to describe its data, e.g. the Art Museum Image Consortium (AMICO) 21, Museum Documents Association s (SPECTRUM) 22, and Dublin Core Metadata Initiative (DCMI) 23. In order to describe the data model and the processes potential users of the system go through when creating the VMC, an element set has been defined from these standards and the technical metadata associated with it. 7. An Augmented Visualization Scenario Figure 6 represents a tabletop AR scenario where a museum user is presented with an interactive augmented presentation composed of 3D models and real physical objects the model could be an actual 3D model, 3D text or a video texture; in this case we have the virtual north American Indian having looking at his real twin brother. In this experimental scenario a museum researcher is looking at a tabletop environment where Figure 6: Augmented presentation scenario Using the AR interface the user can associate a particular virtual model to the marker. The user, e.g. a museum researcher, can now examine the virtual model in the context of the real artefact using either an HMD or the monitor. This scenario can be extended by adding other markers, hence more virtual models or physically annotating the table-top imagine the tabletop is a flat white board. A standard I/O interaction (mouse or keyboard) interacts with the 3D model in real time. Specifically the virtual objects can be easily manipulated in x, y and z-axis in six-degrees-offreedom. In addition different modes of rendering can be switched (i.e. wireframe or solid mode), different texturing techniques can enhance realism (simple or complex texturing) and finally the lighting conditions can be controlled. This scenario could be further extended replacing the table-top with an interactive whiteboard allowing for example a museum curator to visualise the virtual object in the context of the real, annotate the whiteboard with other data, e.g. metadata, and update the metadata based on the ensuing study. 8. Conclusions Our experimental system has been developed to prototype a simple augmented reality environment for visualising virtual museum artefacts with other computer generated objects and metadata. This system has been successful as a pilot experiment for ARCO, and is currently being enhanced as part of the ARCO projects augmented reality interface (ARIF). We have described the basic components including a userfriendly 3D modelling interface used to develop virtual museum artefacts and an interactive AR interface for

visualising the generated 3D models. The experimental system links VMC, describe as an XML file defined by an XML schema, which is visualised by the AR application. Future work will develop a more robust object relational database to enhance the system, switch between virtual reality and augmented reality visualisations, provide stereoscopic rendering, and allow more sophisticated user interactions using a 3D space mouse and other input devices. Acknowledgements Part of this research was funded by the EU IST Framework V programme, Key Action III-Multimedia Content and Tools, Augmented Representation of Cultural Objects (ARCO) project 7 IST-2000-28366. References 1. ImageModeller http://www.realviz.com/products/im/index.php, (last visited 05/03/03) 2. ARCO http://www.arcoweb.org/textversion/description/description3.htm l, (last visited 05/03/03) 3. Leroy, M., et al. The Digital Michelangelo Project: 3D Scanning of Large Statues, Computer Graphics (Proc. Siggraph 2000), ACM Press, New York, July 2000, pp. 131-144. 4. Milgram P and Colquhoun H., ATaxonomy of Real and Virtual World Display Integration. Mixed Reality Merging Real and Virtual Worlds, Ohta Y and Tamura H, 1999 Ohmsha Ltd, Chapter 1, 5-30 (1999). 5. Azuma, R., A Survey of Augmented Reality, Teleoperators and Virtual Environments 6, 4 (August 1997), 355-385 (1997). 6. Milgram P and Kishino F., A Taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, vol.e77-d, no.12, 1321-1329 (1994). 7. ARCO Consortium, Augmented Representation of Cultural Objects, http://www.arco-web.org (last visited 05/03/03) 8. 3D Murale 3D Measurement & Virtual Reconstruction of Ancient Lost Worlds of Europe, http://www.brunel.ac.uk/project/murale/home.html (lasted visited 05/03/03) 9. CHARISMATIC, http://www.charismaticproject.com/index.html, (last visited 05/03/03) White et al / Augmented Reality for Museum Artefact Visualization 10. The Seattle Art Museum Project. http://www.hitl.washington.edu/people/grof/shared Space/Download/Projects/SAM/ 11. The Human Information Workspace(HI-SPACE) http://www.hitl.washington.edu/people/rmay/hs_i nfo.html, (last visited 05/03/03) 12. AR ToolKit, http://www.hitl.washington.edu/people/poup/resear ch/ar.htm#artoolkit, (last visited 05/03/03) 13. Vlahakis, V., Karigiannis, J., Ioannidis, N., Tsotros, M., Gounaris, M., Sticker, D., Daehne, P., Almeida, L., 3D Interactive, On-site Visualization of Ancient Olympia, Proceeding of the First International Symposium on 3D Data Processing Visualization and Transmission (3DPVT 02), IEEE, (2002). 14. EMII. http://www.mda.org.uk/200106c.htm (last visited 07/01/03) 15. Tsai Camera Calibration Software - http://wwwcgi.cs.cmu.edu/afs/cs.cmu.edu/user/rgw/www/tsai Code.html, (last visited 05/03/03) 16. Shneiderman, B., Sparks of Innovation in Human Computer Interaction. Ablex Publishing Corp., 1993. 17. Galitz, W. O., The Essential Guide to User Interface Design: An Introduction to GUI Design Principles and Techniques, 2nd Edition, Wiley 2002 18. F. Liarokapis, N. Mourkoussis, P. Petridis, S. Rumsey, P. F. Lister, M. White, An Interactive Augmented Reality System for Engineering Education, 3rd Global Congress on Engineering Education UICEE (2002) 19. ARCO Consortium, Deliverable 9, Report on XML Schemas, XSL Stylesheets and X-VRML Technology, pp 8 20. Gill T., Metadata and the World Wide Web, Introduction to Metadata Pathways to digital Information, Baca M, Getty Information Institute, pp 9, 1998 21. AMICO http://www.amico.org (last visited 07/01/03) 22. SPECTRUM http://www.mda.org.uk/ (last visited 07/01/03) 23. DCMI http://dublincore.org (last visited 07/01/03)