Augmented and Mixed Reality Gaming

Michael Howell University of Southampton Southampton UK Mh15g09@soton.ac.uk Augmented and Mixed Reality Gaming ABSTRACT In this paper the strength of augmented reality (AR) in game entertainment is described. The technology that is best for AR gaming will be reviewed. The hardware for AR gaming will be evaluated. The software Philosophy that is best for AR gaming will be debated. Genre is a factor in entertainment and games and it can define a new technology, thus the possible genre strengths and weaknesses will be discussed. Finally the method of inputs that complement AR gaming will be analyzed. Categories and Subject Descriptors [Augmented reality] Human Factors, Design. General Terms Documentation, Design, Reliability, Experimentation and Human Factors. 4 th December, 2012 2. Transparent screens or camera based system Keywords Augmented reality, Video games, Entertainment, Ergonomics, Portable computing, Human computer interactions and Multimedia. 1. INTRODUCTION Augmented Reality (AR) delivers graphically rich data on virtual overlays on to the real world. It doesn t intend to completely exclude the perception of the world, unlike virtual reality which aims to block it out completely. AR is a new and interesting area in human-computer interactions and multimedia; it has many uses in many areas. The area this paper will focus on, is the games entertainment. The game industry has driven many areas in technology and could do the same for AR. AR gaming has been in development for over ten years. It still hasn t taken off which is unusual for a promising area of technology. Many attempts have been made to commercialize AR technology and AR gaming. Only recently has any AR been brought in to the mainstream with the advent of smarter potable computing such as smart phones, tablets and portable consoles. It may be due to the accuracy of the systems and the technology, so what systems are currently most well adapted for mainstream games will be appraised. There are many forms of games genres and where a new technology falls, defines that technology for its life span in the games industry. It is important to determine what box R gaming fits into. So this paper will try to determine the area AR gaming will fall and how it may define this technology in the future. Figure 1. A diagram of the distances the 4 types of technology. [6] There are 4 types of AR technology based around their form of display. Each has a different target vision as seen in Figure 1. A diagram of the distances the 4 types of technology.. They all have positives and negatives, I will try to identify which is currently best suited to AR gaming. 2.1.1 HMD Figure 2 pictures of current HMD [6]

The head mounted system (HMD) either reflects the image into the user s eye off angled glass as seen in the images in Figure 2 or now transparent OLED screens are possible, which is a physically transparent screen as seen in Figure 4. In theory this is a great system, which displays AR in the simplest form. It mirrors the AR concept perfectly; the users vision wouldn t be impaired which is ergonomic. Unfortunately most HMD are still big heavy and uncomfortable. It also currently has a downside; it is extremely susceptible to glare and fading in sunlight. In AR Quake, a AR game based on the classic open source game quake, which is a first person shooter. AR quake had a problem with visibility "In order to be visible outdoors to the user, [the creators] had to modify the skin textures of the monsters, which are normally quite dark and not visible on a transparent HMD."[1] (See Figure 3) 2.1.3 Physically transparent screen system Figure 3 An image taken from inside AR quakes HMD [7] The other form of HMD is a camera and traditional Display system, which captures the outside world via a camera, the image is processed and the graphics added and displayed to the user via a screen in both eyes. This system reduces the glare and fading problems, it also allows for completely opaque virtual objects adding to the immersion of the game. It does however current restrict the users viewing angle with most systems cutting out peripheral vision. This affects the user experience peripheral vision is an asset that adds realism to a game. Peripheral vision is responsible for feelings a speed and movement in the user. Wide angle Virtual Reality goggles are not yet on the market. Head mounted display systems would not work for mass-market games due to its current limits and is not currently viable for mainstream AR games. Maybe this will change in the future when products such as Google glass are widely available. Figure 4 an OLED screen (http://singularityhub.com/2010/01/19/samsungs-laptop-withan-amazing-transparent-oled-screen-video/) This system makes use of the now available fully transparent screens now emerging. The lead technology is OLED screens, which are completely transparent conventional screen seen in Figure 4. With this technology, it would be easier to implement an AR system, as a perception of the world is already visible thus there is no need for a camera or processing of the image. It would still need some form of model or computer vision, however, to allow the graphics to align with the real world. Like the head mounted system, it is prone to glare and fading, which makes it hard to implement outdoor games. The OLED technology is not yet viable for the mainstream market as it is still under development and is extremely expensive. Another down fall of this technology is that it can t display completely opaque images making it difficult to implement completely solid objects, which would add to fidelity of the game graphics. 2.1.4 Camera screen based system 2.1.2 Hand held Transparent Hands held transparent system devices now come in two forms physically transparent and camera and screen based systems, which give the illusion of transparency when viewed from and angle. Both have merits and shortcomings. Is either better suited to current mainstream AR gaming? Figure 5 Pictures of Handheld devices [6]

This system makes use of existing mainstream technology as seen in figure 5. Most portable computing, smart phones tablets and games consoles, have built in cameras and vibrant screens and the processing power needed to perform the basic image manipulation required for AR gaming. This requires a lot more processing power than the transparent system, but this seems to not be a problem anymore with smart phones and tablets with GPUs and multicore processors now commonplace. This power enables computer vision algorithms to be processed in real-time with the image gained from the camera. This system also allows for completely opaque graphics unlike transparent screen system. This adds a large amount of realism to the AR graphics. It how ever is limited in its current 3D capability. This system by far is the current best option for augmented reality games. It is widely available, relatively cheap and has graphical benefit over the other option. 2.1.5 Suitability for gaming Both systems have merits, for current gaming it is clear that camera based systems are better suited for the current market for a number of factors, the price and wide availability along with its technical benefits make it the choose for further development of AR gaming. 2.2 Projection systems This comes in two forms held projection systems and spatial system. They both use projection systems such as flat 2d and 3D holographic projection system. Both of these systems are future technologies and are not current systems that could be used to implement mainstream games. 2.2.1 Hand held projection system such a system in AR gaming is difficult to imagine possible social 2D games, as the main asset of this technology is that it is easier to share the experience with multiple people due to its larger visibility. 2.2.2 Spatial system Figure 7 Pictures of spatial systems [6] Spatial system is a design mentioned in D.W.F. van Krevelen and R. Poelman, A Survey of Augmented Reality Technologies, Applications and Limitations [4] it is a system that is not built into portable device. It utilises projection systems built into the environment itself see figure 7. This differs from all the other systems outlined here. Unfortunately this is a far future idea incorporating holo-projectors so this is not a currently viable option for AR games in the near future but offers an alternate in the future that could be extremely flexible and cost efficient. 3. Marker vs. Marker-less A criterion for Augmented reality, as described in Augmented reality Games by Leino et al. (2008)[5], is that it must accurately align the graphics with the world. Marker and Marker less are techniques for aligning augmented graphics with the real world. 3.1 Marker Figure 6 a hand held projection systems [6] Hand held projection system uses a hand held device that contains a projection device Figure 6. There are very few devices that are capable of this; the Samsung galaxy beam (http://www.samsung.com/global/microsite/galaxybeam/) is one such device that has the potential to perform this form of AR. It doesn t currently seem to be a viable option for games, as it is susceptible to glare and can t be outdoors and is difficult to automatically align the data with the world. Also user interaction with a system such as this is not currently possible. The uses for Figure 8 an example of marker technology (http://ael.gatech.edu/lab/research/handheld-ar/arf-iphone/)

The marker tactic is a popular way to display graphics that need accuracy. The most common technique uses a camera and a fixed monochrome image on paper or material, similar to a bar code or quick reaction code, as a fixed point to which the graphics can be oriented, see figure 8. The device is able to do this with simple edge detection algorithm and with some simple mathematical functions that know the size and geometry of the marker. The orientation of the surface marker is lying on can be gained. With this plane objects in virtual space can be scaled and orientated, displaying them accurately on top of the real world surface. This process is not extremely CPU or GPU intensive and even low powered smart phones can easily process this and run a games graphics in parallel in real time. This technique is the one of choice for close and accurate graphics, such as the graphics found in traditional games. It is ideal for board games; it is possible to pack an array of games on to one board with this technology. AR gaming could give conventional and traditional gaming a whole new way to interact with game. 3.2 Marker-less future of AR and I have no doubt that one day soon even it will be accurate enough for main stream use. But currently its inaccuracy is too great, for the system to be ergonomic. The ease and robustness of the marker-less systems are integral to their adoption into the mainstream. 3.3 Evaluation Currently marker based systems are the most likely option for implementing popular games. I think the strength and flexibility of markers could be utilized to create very interesting games. Working to the benefits of marker systems would create new and interesting areas in games. The HCI benefits of using markers needs better investment from the industry. 4. User interaction The interaction with the device while playing games is significant; a games input have to be intuitive and simple yet powerful and flexible yet quick and ergonomic. Games as they currently are have a multitude of input devices from the cutting-edge like Microsoft s Kinect to the simplistic button. AR devices have a unique interaction problem. It s difficult to type when moving so forms of interactions are better suited to AR than others. Some forms of interaction are unique to the sort of portable device capable of AR games currently. Figure 10 represents the current input devices used by AR gaming. Figure 9 pictures of marker-less systems The marker-less techniques more convenient but add complexity to the algorithm. Current marker-less techniques trade convenience for accuracy, which is not a negative point; it s proposed for different more expansive games, where accuracy is not as essential. Large outdoor games could utilize this technology as seen in the popular apps shown in figure 9. As processing power and real time computer vision improves it can be utilized in more areas where marker is currently the only option. Computer vision is improving vastly and as convenient AR gets more and more accurate more people will feel the benefit. AR games can be more spontaneous and take the technology away from serious uses to more entertaining uses. Marker less is the Figure 10 A Graph of inputs of AR games by year.[2] 4.1 Gesture Gesture controls are a popular form already in use in most portable devices. Gestures are favored for their intuitive style. Gesture interaction is a complex area of study in its own right so outlining some of the best interactions that directly relate to AR gaming. Microsoft s Kinect has revolutionized body gestures for games; it has the ability to map the entire body. A Kinect like system for portable devices would open up many possibilities. Another very innovative gesture I found was in Augmented Reality Window: Digital reconstruction of a historical and cultural site for smart phones [4] in this application to clarify the picture the user simulated wiping the phone clean. A very novel and intuitive gesture, which was found to bring emotion to the application experience adding to its value as an interaction. 4.2 Voice Talking to the device to interact with it seems like the perfect way to interact with AR. A device that is smart and context aware, as an AR system should indeed be would solve the problem. In Iron Man the 2008 movie by marvel (http://www.imdb.com/title/tt0371746/) Tony Stark communicates with J.A.R.V.I.S (Just. A. Really. Very. Intelligent. System.). His

voice activated artificial intelligence system that interacts and displays information on his HMD. In theory this is a great way to interact, if the AI system can understand syntax and be contextually aware. Voice activated systems like apples SIRI are becoming a viable alternative to some traditional inputs, adding another input system that is hands free. Current systems are getting extremely good at understanding preset commands, which could possibly be good enough as one of multiple inputs, but not as the sole input device. 4.3 Touch Touch screens are freely available in smart phones and are great for user input in the correct circumstances. Touch screens that lack haptic feedback are difficult for users to learn to use without visual conformation making it had to use them on the move. Touch screens can be used to interact with the virtual objects. This systems maybe the current best option for input of AR gaming systems, it has weaknesses but used in conjunction with outer input it would be part of a strong team. 4.4 Point Using the devices contextual awareness to point and select objects is intuitive and ergonomic if implemented correctly. This is seen in games such as alien invaders and tower defiance where the device is aimed at the AR objects. Yelp also takes avenge of this interaction when panning around a view. The user can aim or point he device at a location the virtual objects can then be manipulated. 4.5 Physical This is an interaction almost unique to augmented reality and is especially effective in some roles. This takes advantage of computer vision used in the detection of markers to over lay graphics onto inanimate tokens. This allows the plain token to become multimedia rich and for the user to interact physically with objects in the virtual element. Touching inspires emotion, emotion leads to a richer user experience. This interaction method works with games that simulate traditional board games but with richer media. Real-time strategy games benefit immensely from this user interaction like a general moving the troops on this strategy map. 4.6 Evaluation There are many forms of input for AR and I have picked out the strongest candidates for the job. Any of the user interactions can t be used solitarily. A strong user experience would need to take advantage of a range of these input options. Physical interactions need more investment and development, as a technology that has the possibility to revolutionize gaming interaction. 5. Genre The category a games technology is placed in often defines that technology for its life span. The social element of gaming leads to stigmas and perception that can be hard to reverse. Anticipation of what genre suits AR and works to it core strengths is vitally important. AR gaming had already developed two distinct groups location based and static marker based systems. Both have strengths and weaknesses and at there core serve two distinct purposes. Location based is a social game type needing multiple players in a location to serve its purpose. Static marker serves as the more traditional gaming model for the more serious genre. The AR gaming may have its own sub categories, but it must fit into the industries categories or blaze its own new path. Chek Tien Tan and Donny Soh[2] also attempt to fit AR gaming into a category. But they find it difficult with a lack of solid industry material. The majority of the currently available applications were developed for research purposes, not for mainstream use. This technology can be achieved through existing, cheap hardware and should therefore be utilised. Even with this lack of material it is apparent that AR is well suited to puzzle and real time strategy from the graph in figure 11. Both take advantage of marker technology, which allows rich intuitive user experience. This game type is suited to AR graphical flexibility and a user interaction that was hard to gain prior to this kind of physical interaction. Exploration games such as environmental detective use the location-based model. Unfortunately no application I have found has utilized both technologies simultaneously. AR gaming definitely has its niches, which fit well into the current gaming market serving both serious and casual gaming type. This places AR gaming in a strong and healthy position. Figure 11 A graph of genre of AR gaming by year. [2] 6. Conclusion I perceive at this point in time that AR gaming will be mostly on portable hand held devices, like smart phones, tablets and portable consoles, like the Nintendo DS and Sony PlayStation portable. All these devices have the same core architecture of camera-based systems with a multitude of built in sensors and online-based Meta data systems. This will be the base line on which AR gaming could thrive. The Nintendo ds PSP along with smart phones are starting to build a line of AR applications and games. All have still to go mainstream they lack the accuracy and core Meta data to be successful in the market, which has high standard. The technology is catching up with ideas of AR and is now a commercial viable option that can compete in may battle grounds. With proper investment from the industry AR games could open a new sector to push the technology forward. Since the advent of the smart phone AR has gone from strength to strength it is on the verge of becoming a new industry leader if more backing and investment is given. I believe AR gaming could greatly benefit the games and entertainment industry, leading use in the future of a media and information rich world. 7. ACKNOWLEDGMENTS My thanks to ACM SIGCHI for allowing us to modify templates they had developed. I would like to thank Doga Kuyucu and Bhairav Sharad for proof reading. 8. Bibliography [1] Butchart, Ben. "Augmented Reality for Smartphones A Guide for developers and content publishers." 01 03 2011. scribd. 01 11 2012

<http://www.scribd.com/doc/56146988/8/markerlessvs-marker-based-ar>. [2] Chek Tien Tan, Donny Soh. "AUGMENTED REALITY GAMES: A REVIEW,." 15 10 2010. gamesstudio.org. 01 11 2012 <http://gamesstudio.org/chek/wpcontent/uploads/2011/02/tanandsoh_gameon.pdf>. [3] Ebling, Maria R. and Ramón Cáceres. "Gaming and Augmented Reality Come to Location-Based Services." Pervasive Computing 9.1 (2010): 5-6. [4] Kang, Jiyoung and Jung-hee Ryu. "Augmented Reality Window: Digital reconstruction of a historical and cultural site for smart phones." Mixed and Augmented Reality - Arts, Media, and Humanities (2010): 67-68. [5] Ollo Leino, Hana wirman, Amyris ernandez. "Augmented reality Games." books.google. 01 11 2012 <http://books.google.co.uk/books d=ncnkiykf 8C&lpg=PA228&ots=gdXBu_oY5&dq= Augmented%20reality%20gaming&lr&pg=PP1#v=one page&q&f=false>. [6] Poelman, D.W.F. van Krevelen and R. "A Survey of Augmented Reality Technologies, Applications and Limitations." The International Journal of Virtual Reality (2010): 1-20. [7] Thomas, Wayne Piekarski and Bruce. "ARQuake: the outdoor augmented reality gaming system." ACM 45.1 (2002): 36-38.