PROCEEDINGS OF THE INSTITUTE OF VEHICLES 2(102)/2015

Transcription

1 PROCEEDINGS OF THE INSTITUTE OF VEHICLES 2(102)/2015 Marcin Januszka 1, Wawrzyniec Panfil 2 AUGMENTED REALITY TECHNIQUES FOR VEHICLE MAINTENANCE 1. Introduction Nowadays, designed and manufactured products are becoming more and more complex. This is particularly noticeable in the field of mechatronic devices, especially vehicles. Knowledge associated with such complex technical objects (e.g. with their servicing and maintenance) is also becoming increasingly complex and difficult to interpret. Systems supporting servicing of such devices must meet the increasing demands, what in effect should give savings of time and costs. This can be achieved through more efficient delivery of knowledge and information to the service personnel. New demands force the replacement of text semantic information grains with visual and interactive information grains [8]. Augmented reality systems are the perfect solutions, which allow for the integration of text grains and interactive models in a common interface. Augmented reality systems allow to superimpose on a current state of a real technical object some virtual objects related to information which is important in the process of such exploitation. Augmented reality technology was born as a result of drawbacks and imperfections of virtual reality techniques. The main idea of AR is to facilitate the execution of a task by the user in the real world and in the context of real objects, that do not provide a virtual reality techniques. For example, AR system can provide technicians with information on how to properly perform the process of installing a door in a car in the context of a real and not a virtual car [6]. This goal is achieved through the provision of information by several communication channels. It is thus possible visualization (via video channel) of assembly procedures of a virtual door on the real car, as well as providing additional sound information. All information is precisely provided at the moment when there is demand for it (just-in-time) and in a proper place (directly on the real vehicle for which the maintenance work is carried out). In this paper the authors present a prototype augmented reality system supporting servicing and maintenance tasks of various vehicles. Augmented reality mode which is used to visualize the data and knowledge connected with a particular vehicle can be more intuitive than traditional ways of delivering knowledge and information using the printed instructions or instructions on a flat computer screen [5]. 2. Augmented reality-based systems for machinery maintenance One of the first AR system supporting maintenance of devices is presented in [2]. KARMA, despite its simplicity, resulting from the application of visualization based on a simplified edge models (Fig. 1), well supported a user in his servicing and maintenance tasks. 1 Marcin Januszka, PhD Eng, Assistant Professor, Institute of Fundamentals of Machinery Design, Silesian University of Technology 2 Wawrzyniec Panfil, PhD Eng, Assistant Professor, Institute of Fundamentals of Machinery Design, Silesian University of Technology 37

2 Fig. 1. KARMA one of the earliest AR system [2] Another example of the use of augmented reality in helping to service activities is ARVIKA [3]. The authors developed a system of several scenarios of application, and one of them is Troubleshooting and service on production machines intended to support personnel and users of machines in their start-up, operation, maintenance and repair. Such a support may take place directly (on the machine) or by interaction with a service center. Examples of applications of the ARVIKA system are aiding a design process of factory installations, maintenance and repairs of the NATO helicopters or servicing and repairs of sliding roofs of BMW cars. System STARMATE [9] is provided to assist the user during the assembly and disassembly of complex technical facilities and their maintenance and service, as well as a training tool for service staff. This system allows employees to perform tasks that require high qualifications (often exceeding their qualifications). Usually their qualifications raise thanks to use of the system. The system works according to the learn by doing idea. The authors [9] present examples of applications in disassembly and repair of jet engines, where the user is aided by the AR system indicating following steps of the task, disassembled parts of the engine, necessary tools, as well as technical documentation of the engine. Another, more recent, examples of the AR systems are VTT PALMOS [7] and context-based system [10]. VTT PALMOS (Plant Model Services for Mobile Process Maintenance Engineer) system was developed for modern maintenance works carried out in industrial plants by staff of these plants or external service providers. The system allows to solve such problems: How quickly locate the plant equipment requiring maintenance? How to get a concise and clear information on plant equipment and its state? How to connect and use the knowledge derived from multiple sources of information without having to use a wide variety of computer software? The second system [10] is a system based on knowledge. In this system content and placement of additional information presented the user are dependent on the currently running tasks/activities. Context in this system is understood as a state of an object, where the object can be the device, place of a service, a person performing it, etc. The system is considered as a context-based, if it can gather relevant information, draw conclusions and adapt to a current situation. The advantage of the system is that it can dynamically adapt to changing situations while performing maintenance and service tasks. In the literature, especially in the online sources, you can find many other interesting examples of use of augmented reality in assisting maintenance and service activities, e.g. with use of mobile devices [11], ARMAR system applicable in operational use and repair of military objects [12], in automotive [13, 14, 15]. 38

3 3. Requirements and assumptions for vehicle service supporting system It is assumed that the AR prototype system supporting maintenance of vehicles should allow correctly and quickly perform the process of service (e.g. the repair or replacement of worn out parts). It should be possible for standard and emergency (when it is not possible for prior training) vehicle servicing situations. A person without a proper knowledge (prior training), i.e. home user, should be carried out step by step by the system providing instructions concerning the way and procedure of the service. Instructions visualized in AR mode may contain: visual simulation of the activity (e.g. a way of some part disassembly), information about the kinds of tools used during the execution of the necessary actions (e.g. a type and size of a key needed for a disassembly), including visualization and simulation of usage during the execution of the task, audio records of how to implement a task (with the possibility of enriching videos). For a system to be used by any user (not only professional, but also home user), it is required that the hardware and software components are not expensive and publicly available. It is assumed therefore to develop a system that can be run on many devices, including mobile devices, i.e. a tablet or smartphone. 4. The prototype vehicle service supporting system In this study was developed a prototype system based on augmented reality techniques, allowing for supporting the various activities (operational and servicing). Implementation has been made for two types of vehicles: a mobile robot and a car. Instructions supporting people engaged in service have been limited to a few selected procedures. i.e. procedure for DC motor replacement of a mobile robot (service activity), cruise control start-up procedure or headlights operation procedure (operational activity). The user of the system is supported by appropriate instructions displayed using elements presented in the form of text, interactive three-dimensional models, movies. Their visualization is carried out at the site of the real vehicle or it s component for which the service is performed. Step by step real-time instructions should enable the efficient conduction of operations also by people without prior training. This way of instructions presenting is more intuitive than the presentation of the instructions in the traditional form (as printed instructions) Hardware components In the case of systems based on augmented reality technology a proper selection of optimal hardware solutions is essential, depending on the purpose of the system. Display devices are one of the most important elements of AR-based systems. In an AR-based system some additional computer-generated elements are added to a real environment that is observed by eyes. There are many ways to present computer-generated virtual content to the user of an AR-based system. By using advanced displays [1], i.e. Head Mounted Displays (HMDs), Hand Held Displays (HHDs), Binocular Omni-Orientation Monitors (BOOMs), and even the standard monitors or projectors, we can present what we normally do not see. In recent years, more and more knowledge presentation using AR techniques could be done through mobile display devices, i.e. displays in tablets or smartphones. The use of mobile devices is both the most effective and cheap for home users. Therefore, the system presented in this paper uses a PC-based tablet both as a 39

4 display device and at the same time as a computing unit (computer). However, the system can be run on any device with Windows, Android, ios operating systems. Another very important element of the developed AR system is suitable tracking system that allows properly integrate virtual objects with images of real world. As in the case of display devices, there are many types of tracking systems (see [1]). The simplest and one of the cheapest solution is a system based on image analysis and processing. The most important component of this system is advanced software. However, for proper operation of the system also some widely available components (especially cameras), are required. In the developed AR system, a HD camera (installed on the tablet) captures images of the real environment (real world), and sends these recorded images in real time to a user s computer. The images are analyzed and searched on the characteristic elements (graphical markers) and then tracked. Next, for such images virtual objects are added Software components The presented system consists of several software components (e.g. a CAD system, a PDM system, an UML editor, a knowledge base, a tracking system and a visualization system) that are integrated with a single simple application for management (Fig. 2). The system is available in a desktop version (for personal desktop computers) and in a mobile version (for tablets or smartphones). The mobile version of the entire system is stored in the form of a compiled executable application or as server-based application to download and install on the user s device. The most important component of the AR system is a software that allows to accurately track the real objects and present virtual objects on the real image. Virtual objects are carriers for conveying knowledge and information. In the developed system, software that uses image analysis has been applied for tracking. There are two techniques used for a tracking: a tracking based on 2D images from a camera and a 3D object tracking with the use of the SLAM (simultaneous localization and mapping) method. Both methods are supported by gyroscopes in order to increase the quality of tracking. Software used by the authors, that uses and implements these methods, is called Metaio SDK (software library) and Metaio Creator (creator for AR scenes). Fig. 2. Components of the system 40

5 Metaio software is also used both during the development and the presentation of an AR scene with the knowledge useful during maintenance operations. The elaboration of the AR scene involves determining an appropriate orientation, a position (relative to the real objects, that are tracked), a size, and other parameters of the virtual objects. The last stage - the application of knowledge - is to present knowledge to the people involved in maintenance tasks. The most important knowledge and information necessary for the operation of the system should be obtained from a design process of a vehicle. During this design process some virtual 3D CAD models are developed. In the presented system, due to the lack of such models, to reconstruct them reverse engineering methods have been applied. In the process of reconstruction and 3D modelling a system CATIA V5 was used. The 3D models were then stored in Product Data Management (PDM) system. A repository that contains virtual models of the tools used during the operation and animated models of typical maintenance operations (models with move of a hand, operations of unscrew the left / right, clamping, gripping, etc.) (Fig. 3) is also stored in the PDM system. Due to requirements of the AR scene all models are converted to the format supported by the used software (mainly the FBX format). Fig. 3. Examples of models of tools and maintenance tasks from repository Except that the user of AR system can see virtual 3D and interactive models superimposed on the real world, he/she can also see knowledge connected with maintenance activities in form of diagrams. Activity diagrams are created in accordance with the notation of the Unified Modeling Language (UML). This knowledge is stored in a knowledge base in two different forms: graphical (as.png graphical file format), and textual (as an XML file). For the purposes of the research carried out by the authors only a graphical form (with the use of UML diagrams shown in AR mode) was applied. The XML file can be used in future knowledge processing systems or knowledge based systems (expert systems). Presentation of knowledge with diagrams and 3D models may be supplemented with additional sources of knowledge, i.e. audio descriptions, photos, videos that could be also stored in the PDM system Functionality of the system If someone needs to perform some maintenance activities, he/she should launch the main application for the AR system management (for PC) or standalone application (for mobile devices). When the application is running, a searching for an appropriate instruction of a maintenance task is available. If the user finds a right instruction and points a main unit (tablet, smarthone, PC camera) at a vehicle (often at a fragment of the vehicle), for which instructions should be displayed, then the images with an indicated key parts of a vehicle are automatically 41

6 displayed. The user can analyze (regarding to position and part name etc.) these parts. Next, by pressing "Start" button it is possible to run the instruction that presents maintenance activities. The user sees on the screen the visualization of a first operation for a procedure (e.g. unscrewing the fixing screws). At the same time on the screen could be presented (Fig. 4): UML diagram with an indicated current operation from a whole procedure (UML activity block with the name of the operation), models (that are usually animated) of parts (components) related to the operation, tools that are necessary to carry out the current operation, animation of activities performed by the user (also with the use of 3D models). Additionally, the user can listen to the audio description of the operation or refer to additional sources of knowledge, i.e. movies, pictures, drawings, if they have previously been prepared for the operation. After the initial operation, the user can - by selecting a button from the UML diagram - go to the next operation and so on until the end of the procedure. Fig. 4. Example of elements presented to the user on a screen The system presented in this paper allows the user to develop both the basic and advanced skills in maintenance of vehicles with a maximum reduction of the risk of damages (caused by a lack of knowledge). The system allows the user to get acquainted with a complex construction of a vehicle and with main components that are parts of the vehicle (Fig. 1). However, the most important function of the AR system is presentation of maintenance instructions regarding to a vehicle, including: operating instructions for operators, maintenance instructions for technicians (repairs, part replacements). The functionality of the AR system allows a presentation of the visible interior of the vehicle. Moreover, it is possible to visualize the position of individual elements (Fig. 5), including those which are normally not visible from outside (e.g. are hidden under the covers) or those for which an access is difficult. Such information is a hint for an inexperienced user, what is the exact location of a part (useful e.g. in the case of need of part replacement or its repair). Likewise, various types of data and information (e.g. parameters, marks etc.) connected with selected components may be also displayed. 42

7 For example, information regarding to a part mark allows proper selection of a replacement part during exchanging procedure. Fig. 5. Presentation of elements positions and descriptions The most important feature of the system is connected with a possibility of maintenance tasks supporting (Fig. 6). Generally, the user of the AR system is supported by displaying the appropriate instructions, using interactive 3D models and diagrams superimposed on a physical object. Step by step" instructions with visualization of realtime tasks allow to efficiently carry out complex maintenance operations, including those with no prior training. This way of instructing the user should be more intuitive than the presentation of the instruction in the traditional form (as printed instructions). Instructions visualized in AR mode may contain knowledge for which different virtual objects could be used as carriers (see section 3). Knowledge and data for the system are derived from the design stage in which many virtual 3D CAD models are elaborated. Fig. 6. Examples of visualisations of maintenance tasks 5. Evaluation of the AR system The aim of the validation studies in relation to the developed system was to evaluate the operation and usefulness of the system in the context of its use to facilitate maintenance activities for vehicles with the use of AR techniques. To evaluate the system quantitative (time of maintenance tasks) and qualitative measures (satisfaction of the user) were selected. Experimental studies have been carried 43

8 out in order to evaluate the functionality and usefulness of the developed system using AR technology. Experiment consisted of four maintenance tasks (for two types of vehicles, i.e. a mobile robot and a car), including: engine replacement for a mobile robot (task connected with robot service), a light bulb replacement in a car (task connected with car service), launching cruise control in a car (task connected with car operation), running lights in a car, in the correct order, i.e. traffic lights, high beam, position lights, front fog lights, rear fog lights (task connected with car operation). 10 random people were invited to take part in the experimental study. Before the experiment they have never been dealing with vehicles involved in the experiment. Each participant carried out two maintenance tasks using the AR system and another two tasks using traditional printed user's manual supplied with the vehicle. User randomly sketched which of two tasks were carried out by them with the use of the AR system, and which another two without the AR system. The validation of the system was assessed with the use of induction method, according to the following plan: 1. A short training for participants, explaining the purpose of the experiment; 2. Carrying out the maintenance tasks for two types of vehicles; 3. An evaluation of the system in terms of functionality and usefulness of the AR system. The evaluation of the system has both quantitative and qualitative form. Quantitative assessment was made as a result of direct observation of time of the task. A test solely included the duration time of operator actions, assuming that the other times (e.g. time of preparation and time to search for information, etc.) are at a very similar level, both in the case of the AR system and the printed manual. To assess the quality of the AR system a personal interview method was used. The participants evaluated the usefulness and functionality of the AR system and traditional manuals, during the interview, in three categories: an efficiency, a quality of visualization, an overall evaluation of the usefulness. The participants responded to statements using a three-point Likert scale. The scale rating was determined by evaluation from -1 negative rating, 0 I do not have an opinion, up to 1 positive rating. The average saving times thanks to the using of the AR system for computer-aided maintenance are presented in Fig. 7. Analyzing the duration times of maintenance operations realized by individual users it can be seen that by using the AR system it is possible to achieve a significant shortening of the tasks execution. The mean shortening of the execution time of the tasks achieved the level of 23.7%. This result is largely consistent with the results for this type of maintenance tasks presented in [4]. In the case of qualitative assessment the AR system received higher average scores than traditional printed user manuals in all three categories. The results of the qualitative assessment are presented in Fig. 8. It can be seen that only in the category of quality of visualization results were most similar. This situation was probably due to some disadvantages of AR technology in the field of visualization, i.e. limitations related to image resolution, and tracking stability during the superimposing the virtual objects on the real-world images. 44

9 Fig. 7. The average duration of the maintenance tasks and time savings Fig. 8. Comparison of quality assessment of the AR system and the traditional user manual. Summarizing, the participants of the experiment positively evaluated the AR-based system aiding maintenance of vehicles. This group clearly indicated the usefulness of this system in the maintenance tasks. 6. Summary The need for the presentation in a comprehensive manner of knowledge related to the handling of complex technical means, which are vehicles, requires advanced visualization methods. A proper visualization allows more flexibility and freedom in decision making, speeds up the maintenance and helps to avoid errors. In this respect great potential involves the application of the augmented reality techniques, which attempt is presented in this article. As demonstrated by studies concerning the use of AR system supporting a user in performing servicing and operational activities on different groups of vehicles, it should reduce the time of these activities (in relation to the time they will be fulfilled with supporting by traditional printed user manuals). A significant benefit of the use of the augmented reality is also that the operating instructions developed in this way are much more readable for people using them. The knowledge represented in the form of 3D models can be accurately presented at the time of a demand for it and in such a way that the user is not disturbed during his duties. These 45

10 advantages have a positive effect on the shortening of executed operations, and thus allow for some costs savings. As part of a future work is expected to conduct verification tests on a larger group of people, as well as among professional technicians in garages. Of particular interest may be the results of a group of professional mechanicians who have a lot more knowledge and experience than the standard user of the vehicles. The goal is to demonstrate the usefulness of the system also for such users. Benefits of using augmented reality techniques allow us to hope that one day such systems become widely used and will contribute to the further rationalization of maintenance and servicing of vehicles, despite still present some technological (quality of visualization and tracking accuracy could be improved) and economic (currently high cost of equipment) problems. References: [1] Azuma R.T.: A Survey of Augmented Reality. Teleoperators and Virtual Environments 6 (4), 1997, s [2] Feiner S., Macintyre B., Seligmann D.: Knowledge-based Augmented Reality. Communications of the ACM, 36(7), s , 1993 [3] Friedrich W.: ARVIKA-augmented reality for development. Production and Service, International Symposium onmixed and Augmented Reality, s. 3 4, 2002 [4] Januszka M.: Metoda wspomagania procesu projektowania i konstruowania z zastosowaniem "poszerzonej rzeczywistości", Zeszyty, Nr 147, Politechnika Śląska, Instytut Podstaw Konstrukcji Maszyn, Gliwice, 2012 [5] Moczulski W., Panfil W., Januszka M., Mikulski G.: Applications of augmented reality in machinery design, maintenance and diagnostics. In: R. Jablonski, M. Turkowski, R. Szewczyk, eds., Recent Advantages in Mechatronics, Springer- Verlag, Berlin Heidelberg, s , 2007 [6] Reiners D., Stricker D., Klinker G., Muller S.: Augmented reality for construction tasks: Doorlock assembly. 1st IWAR98, pp.31 46, 1998 [7] Savioja, P., Järvinen, P., Karhela, T., Siltanen, P., Woodward, C.: Developing a mobile, service-based augmented reality tool for modern maintenance work. In: Proc. of the 2nd Int. Conference on Virtual Reality, Beijing, s , 2007 [8] Skarka W., Moczulski W., Januszka M.: Interaktywne technologie w procesie kształcenia. Szybkobieżne Pojazdy Gąsienicowe (29) nr 1, 2012 [9] Schwald, B., et al.: STARMATE: using augmented reality technology for computer guided maintenance of complex mechanical elements. In: Proceedings of ebusiness and ework Conference (e2001), Venice, s , 2001 [10] Zhu J., Ong S. K., Nee A. Y. C.: A context-aware augmented reality system to assist the maintenance operators, International Journal Interact Des Manufacturing, vol. 8, s , 2014 [11] Augmented reality in action - maintenance and repair {Available : [12] Augmented Reality To Help Military Mechanics Fix Vehicles {Available : [13] End of the mechanic? BMW smart glasses make it possible for ANYONE to spot and fix a car engine fault just by looking at it {Available : 46

11 Smart-glasses-make-possible-fix-car-engine-just-looking-it.html} [14] Metaio and Volkswagen announce MARTA augmented reality service support for new XL1 concept car {Available : [15] I-Mechanic, the AR App that turns yourself into a Mechanic {Available : Abstract Knowledge of modern technical means, connected for example with their using or servicing becomes increasingly complex and difficult to interpret. Systems supporting servicing of such technical means are expected to fulfill still increasing requirements, which in consequence should result in time and money savings. The paper presents an augmented reality (AR) system based on a mobile device such as tablet. The main task of the system is to aid a normal use and maintenance tasks of the technical means by enhancing the image captured and displayed on the tablet with virtual 3D objects. These objects are properly located with respect to the real objects. The aiding consists in displaying invisible parts, activity diagrams of the tasks, and tools necessary during the tasks. Verification of the system have been carried-out on two objects, i.e. a mobile robot and a car. In the first case the task was to replace a drive of the mobile robot. In the second case the task was to replace a bulb or to launch a cruise control system or switch the lights. There was done a quantitative assessment of the AR system based on comparison of the time of these operations in carried-out in a conventional manner (with use of printed manuals) and using the developed AR system. There was also made a qualitative assessment of the AR system based on subjective evaluation of system s users. Both assessments (quantitative, qualitative) proved beneficial effect of using the AR system to perform maintenance and simple car driver tasks. Keywords: augmented reality, vehicle maintenance, computer-aided vehicle maintenance, interactive systems, knowledge-based systems, visualization TECHNIKI POSZERZONEJ RZECZYWISTOŚCI DO WSPOMAGANIA OBSŁUGI POJAZDÓW Streszczenie Wiedza dotycząca współczesnych środków technicznych, związana np. z ich obsługą serwisową/techniczną lub operatorską staje się coraz bardziej złożona i trudna w interpretacji. Przed systemami wspomagania obsługi takich środków technicznych stawiane są coraz wyższe wymagania, które w efekcie mają na celu uzyskanie oszczędności czasu, a tym samym kosztów. W artykule zaprezentowano mobilny system poszerzonej rzeczywistości (ang. Augmented Reality, AR) wykorzystujący tablet, którego głównym zadaniem jest wspomaganie obsługi serwisowej i operatorskiej poprzez wzbogacanie obrazu rejestrowanego i wyświetlanego przez ten tablet obiektami wirtualnymi. Obiekty te są odpowiednio zorientowane względem obiektów rzeczywistych. Wspomaganie polega m.in. na wyświetlaniu niewidocznych obiektów, diagramów czynności dla wykonywanych zadań, a także narzędzi potrzebnych w trakcie tych czynności, Weryfikację działania systemu przeprowadzono dla dwóch obiektów, 47

12 tj. robota mobilnego oraz samochodu osobowego. W przypadku robota mobilnego dokonywano czynności serwisowej polegającej na wymianie jego silnika napędowego, natomiast w przypadku samochodu dokonywano czynności serwisowej (wymiana żarówki), a także czynności operatorskich (uruchomienie tempomatu oraz włączenie świateł). Dokonano oceny ilościowej systemu AR na podstawie porównania czasów wykonywania wspomnianych czynności w sposób tradycyjny (z zastosowaniem instrukcji drukowanych) oraz z wykorzystaniem opracowanego systemu. Dokonano również oceny jakościowej systemu na podstawie subiektywnej oceny jego działania poprzez jego użytkowników. Obie oceny wykazały korzystny wpływ zastosowania systemu AR na wykonywanie czynności obsługowych. Słowa kluczowe: poszerzona rzeczywistość, komputerowe wspomaganie eksploatacji pojazdów, systemy interaktywne, systemy bazujące na wiedzy, wizualizacja 48