Augmented Reality and its uses in Computer Gaming

Transcription

1 Augmented Reality and its uses in Computer Gaming Mark Higgins 3 rd Year Computer Game Development School of Computing, University of West of Scotland Markh888@hotmail.co.uk Abstraction The purpose of this paper is to examine the current uses of Augmented Reality(AR) in the computer games industry. It will look through the different categories of AR and see how each are used to enhance game play and to also increase the level of immersion of computer games. This paper will also take a look at the future potential of AR and try to see what it can possibly be used for in the years to come. Introduction Augmented Reality was first developed back in the 1960s and has continued to make a large amount of progress over the years with the intention of merging the real and digital world. There is a growing interest to bring more physical interaction and movement into the way we play our computer games and Augmented Reality has the potential to offer that to us. As we have seen over the past few decades the popularity of computer games has grown to a point that it is now rivalling all other forms of entertainment. Augmented Reality has the potential to be the next stage of development in this fast flowing and highly creative industry.

2 Contents Page Page 1 Abstraction and Introduction Page 2 Contents Page Page 3-7 Augmented Reality o Surface o Pattern o Outline o Location o Holographic Page 7-9 Strengths and Weaknesses of Augmented Reality o Surface o Pattern o Outline o Location o Holographic Page What does Augmented Reality offer Computer Games? o Game/Real World Environment Interaction o Immersive Game play o Social/Group Games o Mental and Emotional Attachment Page Why is Augmented Reality so Appealing? Page 13 Is There a Market for Augmented Reality Games? Page Currently Under Development Page 15 Into the Future o Indoor/Outdoor First Person Shooter o War Games o Smaller Games

3 Augmented Reality Augmented Reality is the term for a live view of the real world that has been changed by adding computer generated graphics in real time. Augmented Reality is different from Virtual Reality. Virtual Reality creates an entirely new digitally created environment for the user to look round and explore. Whereas Augmented Reality uses the real world around us that can be seen through cameras and adds it s content on top of it, so it looks like it is really there. Different types of Augmented Reality display the graphics and information in different ways. Some still use large head mounted displays or digital goggles worn by the user to see the images. Others use projectors to display the information onto blank surfaces such as a wall, transforming the surface into something like a touch screen computer. Also some types of AR systems use webcams or hand held devices that have cameras to capture the real world environment. A computer then combines the cameras input with the AR graphics and displays it onto a screen. There are five easily distinguishable types of Augmented Reality. They can be given the names: Surface, Pattern, Outline, Location and Hologram/Holographic. Surface This form of Augmented Reality uses surfaces that are not commonly thought of as computing surfaces. Such as tabletops, walls, floors etc. Good examples of this technology include Microsoft Surface that was first launched April 17 th 2008 by Microsoft. Also SmartTable that was developed and released by the Malaysian company SmartSurface in May This company has also gone on to create a 6.5m display at the Malaysian Institute of Microelectronic Systems (MIMOS) called SmartWall.

4 Microsoft Surface [1] These devices might not be thought of as using classic Augmented Reality, that is to say projecting images into a view of the real world. But they do not require any sort of human controlled mouse or keyboard. The devices have a wide flat touch screen surface and everything is controlled by the user s fingers. Touch screens have been available for over 30 years since the first was developed back in 1974 by Sam Hurst, but these new systems are very different from a simple touch screen computer. The surface uses cameras under the touch sensitive area. They are able to sense with the use of pressure sensors to recognise foreign objects placed onto it. For example, putting a paint brush onto the surface opens up a Painting application. The tables are also able to recognise and handle the input from several different people at once. Pattern The Pattern form of Augmented Reality recognises certain objects, shapes or patterns as where it will display the computer generated graphics or information onto. This method uses small, hand held cameras or webcams to capture the surrounding environment, a computer to create the AR and a monitor or television to display the output. The special markers that are used to tell the computer where to display the images act like barcodes that appear on the back of packets of food. The markers are small cards which are usually black and white with very simple block shapes making it easy for the computer to register them. The camera takes in the image and the computer recognises the pattern that is on the card. Each pattern corresponds to a different 3D model, some information that changes the task that they are being used for or a small animation.

5 2D Matrix Marker [2] 2D Matrix Marker displaying 3D image [3] Outline Outline is a form of Augmented Reality that allows the user to fully interact with the digital environment in the game or system that they are using. The Augmented Reality software tracks the movement of either the whole, or certain parts of the user s body as the input for the system. Probably the most well known and anticipated technology that uses this form of Augmented Reality is the upcoming release from Microsoft: Project Natal. Microsoft s Project Natal [4]

6 The Project Natal system is able to detect the full body of the player in front of it, allowing for every small movement to be completely tracked in real time. This allows players use their entire bodies to play games, not just control it from small hand held devices. The system not only tracks the player s small body movements, but their movements in the 3D environment. The system is also able to detect if the player moves left, right, back, forward, jumps or crouches. This form of Augmented Reality is a big development in how players find themselves immersed in the games that they play. Instead of sitting down on a chair or couch the player has the ability to completely control the movement of the game s character. Playing a sports game the player can kick a football by swinging their foot, hit a tennis ball by moving their hand or throw a virtual baseball. Location This form of Augmented Reality is most useful as a way for finding directions or information about the surrounding area. The Augmented Reality is tied into a GPS system that tracks the real world location of either the user or the device they are using to access the software. A device that is putting this form of Augmented Reality to use is the iphone. There are many apps available that are all used to help direct people where they are going. For example: o Le Bar This app s development was funded by Stella Artois and will direct the user towards the pub that they are closest to. The app uses the camera and displays the directional information onto a view of the streets around the user. o Car Finder This app is funnily enough used to find your car. The app remembers the location of your car and then displays how far away and in what direction it is in, on the device s screen. Another well developed form of this Augmented Reality is the game ARhrrrr! It was developed by the Georgia Tech Augmented Environments Lab for NVidia s hand held device, Tegra. The player is looks at the view of a few small city streets that have zombies walking through them. The player s device plays the part of a helicopter gunner trying to save the civilians from the zombies. The game monitors where the player is viewing and lets him move around and see this 3D environment from any angle.

7 ARhrrrr! Real world view[5] ARhrrrr! Augmented Reality game view[6] Hologram/Holographic This form of Augmented Reality is mostly still at the research stage. The idea is that instead of the graphics of the game staying inside the screen, this AR would use 3D holographic technology to display it all in the real world. The player would also be able to interact with these 3D holograms using movement tracking, just like in the previous forms of Augmented Reality. Holographic Augmented Reality has not yet becoming part of computer games. However there are some interesting developments in the field of holographic projections that show that the inclusion of Holographic Augmented Reality into computer gaming could soon be possible. LM3Labs have created an interactive hologram technology called Airstrike. Airstrike projects a hologram that the user can control with basic and movements. Moving your hand down will gradually make the hologram disappear and moving your hand up will make it reappear. Another company that is working on high definition holographic projections is Musion. All the images created by Musion all look amazing. However they are so detailed the hardware needed to create them is extremely bulky and expensive. Therefore at the moment the inclusion of Holographic Augmented Reality into the mainstream computer games market is not possible. Strengths and Weaknesses of Augmented Reality As listed above Augmented Reality can be categorised into five separate groups: Surface, Pattern, Outline, Location and Hologram/Holographic. Each category has its own strengths and weaknesses that it would bring when it is integrated with computer games.

8 Surface Strengths o Surface Augmented Reality would offer lots of opportunities for communal games. As the surfaces allow for input from several people at once, you could play with more than one other person at the same time, all on the same surface o As the surfaces are able to register objects that are placed onto them, they would be an excellent tool for people who enjoy playing table top games such as Dungeons and Dragons, or model based games like Warhammer. Instead of the players having to calculate the game s outcomes in their heads the computer could do it for them. This would allow for a much more dynamic, interactive and immersive game. o The surface Augmented Reality would add a new perspective to computer games that player s may not have seen before, using their hands directly. If you were playing a shooter you could shoot the gun by pressing your finger to where you want it to shoot. Weaknesses o The main weakness of these devices would have to be availability. Looking at the website for the Microsoft Surface, it looks like it s only available for businesses or for developers looking to create applications for it. Obviously these are just meant to be big fancy toys that look impressive at a company s open day. It looks like this form of technology is still quite a long way from becoming wildly available to the general public. o Another weakness of this technology is that it is very unmovable. These large table units aren t going to be able to be moved very far or easily. This is a disadvantage to most types of computer games. Current game consoles are all easily moveable, for example if you want to go play in a different room of the house, or take it round to a friend s house to play. Pattern Strengths o This type of technology is fairly easy to develop. You don t need to buy any extra or super expensive hardware to make it work. All you need is the Augmented Reality software, a webcam and a piece of paper with the marker on it. o Pattern Augmented Reality would be useful for the people who enjoy table top games. Instead of having lots of little plastic models, all you would need is a little piece of paper with a marker on it and let the computer add the graphics. o A good thing about this technology is that it is already well developed and already commercially in use. For example Adidas released a shoe that when held in front of a webcam would unlock several interactive games on their website. All of these games also used Augmented Reality and the shoe became the player s controller for the game. Weaknesses

9 o Games using this form of Augmented Reality are still not very common and many of them are still just in the experimental stage and not yet commercially available. For example the game mentioned above, ARhrrrr! looks like a good game to play but it is just a part of an experiment or research in a technology lab. o It is possible for these systems to display quite detailed graphics onto the markers, however they still aren t able to compete with the believable game play and HD graphics that are available from normal console games. Outline Strengths o The main strength of this type of Augmented Reality would have to be the potential for the development of active gaming. As the system is able to track full body motion, then all games using it will require players to be on their feet. They will be moving around a lot more than they would with conventional games consoles. o Outline Augmented Reality will also for a much greater sense of immersion into computer games. As the game is able to track body motion, it would be able to display any sort of character on top of that movement letting the player control the avatar that is being displayed. o As shown by Project Natal the hardware for this type of Augmented Reality is already able to be very compact and used in conjunction with games consoles that are already widely in use. Players do not have to go out and spend large amounts of money on a large piece of equipment to play it. Weaknesses o It seems to be quite easy for this technology to track the movement of a single player. However it is only going to become more and more complex if you want to include more players into the games. o The games that are completely controlled by the player s body movement would be quite constricted in terms of explorable game area. If the player is controlling where the character is moving, then they are going to be limited, mainly be the size of room that they are playing in. The space of the room will also be restricted down to the area that is visible to movement tracking system. Location Strengths o The main strength of this type of Augmented Reality is that it makes information much more easily accessible to the user. It saves the user spending a lot of time either on the internet or phoning around to look for this information. For example the Le Bar app mentioned above gives the user exact directions to the nearest pubs in the area. The app is also able to give user reviews about the venue. o Also this type of Augmented Reality would be fairly easy and cheap for companies and venues such as pubs, restaurants or nightclubs to develop or become included in for advertising purposes. Looking through the camera of their iphone the user would

10 be able to see whatever adverts or deals the venues are offering, and how far away they are from the place. o A good thing about this Augmented Reality is that there is a very small learning curve; only a basic level of knowledge would be needed to use it. All the user needs to be able to do is turn on the application on their phone and look at the screen as they move the phone around them. Weaknesses o Currently the technology that puts this type of Augmented Reality to use, such as the iphone or other small hand held devices are still fairly expensive and are only used by the minority that can afford them. o This form of Location Augmented Reality completely relies on the use of GPS location technology. If for any reason the device cannot receive a signal from the GPS satellite then the app using it would not be able to work properly. o If the applications were being used to display adverts or special deals then they would have to be updated fairly regularly to make sure that it is display correct information What does Augmented Reality offer Computer Games? Immersion is only one way for players to enjoy a game. For the purposes of examining the possibilities of AR in gaming we characterize the experience of playing games as four aspects; physical, emotional, social, and mental. Motivations for Augmented Reality Gaming There are many things Augmented Reality can add to the experience of playing computer games: Game/Real Environment interaction, immersive game play, also mental and emotional attachments to the game characters and environments. These are things already available from current games, but Augmented Reality has the potential to improve and add to these areas. Game/Real Environment Interaction Computer game players have always enjoyed being able to explore areas and interact with objects within virtual game worlds. Augmented Reality takes this idea to a new level and combines the game s world with the player s real world. Most computer game play has confined the player to sitting in a chair and staring at a television screen. The Outline form of Augmented Reality could be seen as the leader of the category of games which is steering games away from this stereotype. With the use of

11 motion tracking, the player has full control over what their game character is doing, allowing players to experience a digital world, through their actions in the real world. Immersive Game Play Computer games up to now have offered very good mental immersive game play. Meaning that it s easy for your mind and imagination to become immersed into a game s world, but your body is still sitting on a chair looking at a screen. Augmented Reality takes immersive game play to the next level. If the player is controlling the character through full body motion, or even hand movements they are going to feel like they are much more included in the game than they would be with just a hand held controller. It won t be long till Augmented Reality shooter games, or even RPGs are available. Players will then find it even easier to become immersed into the games they are playing as there will be a real sense of competition or hard earned achievements available through those sorts of games. Social/Group Games Augmented Reality also has a great potential for games that include large groups or teams. From simpler forms of Augmented Reality that could let players play table top games with interactive 3D graphics, to large scale shooter or RPG games. Any type of game that lets more than one player join in there is going to be lots of interaction between the players. For example, The Outdoor Augmented Reality Application: ARQuake that was developed by, The Wearable Computer Laboratory, University of South Australia. The player carried a portable computing unit on their back, which calculated the input and outcomes of the game and displayed the Augmented Reality graphics to the players through a set of digital goggles. The gear used and carried by each player for this experimental game was very bulky and looked quite heavy, not something for long length games. ARQuake Equipment [7]

12 It is easy to see that large scale Augmented Reality games of this sort are still far from being available to play. It is also interesting to imagine what could be possible with this sort of outdoor Augmented Reality technology in the not so distant future. Mental and Emotional Attachment Computer games using Augmented Reality that offer good and believable emotionally rich content could still be quite a long way off. Looking at systems like Project Natal the main games that they have been demonstrating have been basic swing my arm to hit a ball type games. They also seem to be using a fairly plain looking avatar to be controlled by the player. Therefore an attachment to characters may be quite difficult. Consider some of the famous computer games characters of the past: Sonic the Hedgehog, Super Mario or Lara Croft. All these characters have spawned a massive industry around themselves that has lasted for decades. They have all sold millions of games, millions of pounds worth of merchandise and in the case of Lara Croft a couple of feature length films. Each of these characters has become loved by the computer game community, and still sells millions of games because people enjoy playing the characters. Also just about everybody has heard of, or seen a picture of these characters, even if they have not played one of the games. Thinking about these famous computer games characters just shows how far Augmented Reality has still got to go to be seen as an equal contender against such competition. It is true that a lot of games companies will probably try to carry their old characters into new Augmented Reality games to add a new dimension to the game play. However it is intriguing to imagine the brand new characters that games companies may develop in the future. Why is Augmented Reality so Appealing? To answer this question we need to think about why people play games in the first place, in the real world and on computers. One major feature that people enjoy about games is that it is so easy to become immersed in what you are playing. Every form of entertainment is the same. If you enjoy something, you become focussed upon it and forget about everything else. Films, books, television, sports, board games, card games, are all ways that people use to escape from whatever else is happening in their lives. And computer games are no different. Real world games offer us more than just entertainment. They offer physical, emotional, social and mental interactions with our environment around us, and with other people. From a very young age children are taught how to interact with their world through games and playing, a practise that carries throughout our lives in some form or another.

13 Sadly, through the development of computer games the idea of physical and even social interaction has been lost. Computer game players now are almost always confined to sitting with their consol in front of a television screen and barely reacting to anything that is happening around them, except to what is on the screen. The most social interaction available within computer games is talking to other people through a headset over the internet. It s just not the same as having someone playing the game next to you. Computer Gamer Evolution [8] The image of sitting in a dark room hunched over in front of a television has become a stereotype of computer game players and is mainly true of the small percentage of the population that spend lots of their spare time playing games. But as we have seen with the release of the Nintendo Wii there is a large percentage of the public that enjoys the kind of game play the Wii provides. The Nintendo Wii has seen a lot of people starting to play that before haven t necessarily been targeted as consol game players: young girls, middle aged men and women, large groups and families. The Wii has even enticed some of the already hardcore game players into enjoying a lighter and equally as fun game playing experience. Is There a Market for Augmented Reality Games? It is obvious that more and more people are beginning to enjoy and want more active computer gaming. Especially nowadays when there are so many concerns about the growing number of young children being described as unhealthy and overweight due to lack of exercise. This is why I believe that Augmented Reality will soon become the future of how we play computer games, and why it is so appealing. Consol games just now are mainly controlled by just your hands, pushing a small number of buttons. This will change when

14 Augmented Reality becomes implemented. Games may still be controlled by just your hands but it is possible the games could take their instructions from the movement of your body. Augmented Reality has the potential to reintroduce the lost physical and social interactions to our computer games. All ready I have seen just as many adverts in pubs for Karaoke Nights as I have Nintendo Wii Competitions. It will allow us to continue playing games in groups, in more interesting and entertaining ways. Currently Under Development There are several Augmented Reality systems that are in development and are soon to be released that have had quite a lot of media coverage. The two main products would have to be Project Natal developed by Microsoft for the Xbox360, and the Playstation Move developed by Sony for the Playstation 3. Both of these systems are meant to be a way for Sony and Microsoft to break into the casual gaming market that Nintendo has ruled over with the Nintendo Wii for over five years now. Both of the new systems use forms of Augmented Reality which is an improvement over the Wii, which only uses movement tracking to play its games. Project Natal uses a form of the Outline style of Augmented Reality, fully tracking the player s body movements to allow them full control over the characters in the games. This is a large improvement over the Wii because the player does not have to mess around with hand held controllers any more, the player just controls everything with their movements. The Playstation Move uses a different form of Augmented Reality compared to Project Natal. The Move still uses hand held controllers that are extremely similar to the Wii controllers Nintendo Wii Controller [9] Playstation Move Controller [10] As you can see the Playstation Move has a large ball at the end of the controller, which is the reference point for the game, which uses a form of Pattern Augmented Reality. The

15 game recognises the ball as where the players hand is, allowing the player to interact with the digital world of the game. However the one of the most impressive Augmented Reality technology that is being developed at the moment is: the Sixth Sense device, which was created by Pranav Mistry, a student from MIT. Sixth Sense is a device designed to enhance personal computing and is small enough that it hangs around the neck of the user. With the use of a small projector, the Sixth Sense is able to use any blank surface as an interactive screen. Sixth Sense uses more than one category of Augmented Reality that is specified above. It uses the Surface form of Augmented Reality to project its information onto blank surfaces. It also uses a combination of Pattern and Outline Augmented Reality. The user wears coloured markers over the tips of three fingers on each hand. The system recognises the colour markers as controlling points. These let the user move and control anything that is being displayed by the device with their fingers. Into The Future It is possible to see that Augmented Reality is soon going to become a part of real life, no longer a thing of fantasy or science fiction. So having looked at what has already been done with Augmented Reality the biggest question that presents itself is: What is going to be next? As Project Natal and Playstation Move have not officially been released to the public yet, they can be considered as future technology. However what is most interesting is to think of new uses for Augmented Reality that has not yet been developed yet. Indoor/Outdoor First Person Shooters As shown above with the ARQuake experiment there has already been a bit of research done in this area. However, looking ahead there is a great potential for this technology to be implemented into the area of first person shooter games. There are already a lot of people who enjoy playing games such as Paintball or Lazertag; both involve team games of shooting the enemy team. So taking it a step further: with the addition of digital goggles and Augmented Reality. The people hosting these types of games could make the game environment or even the enemy team look like anything they wanted. Imagine, in real life the game is being held in a large warehouse with a few walls and things to hide behind. Then when the player puts on the goggles they can see what looks like a battlefield on an alien planet and they are being attacked by alien creatures. Looking even further into the future, these types of games could even be played without the use of the digital goggles. Holographic Augmented Reality may eventually be developed

16 to a point that these games could produce completely computer generated enemies with advanced enough AI to play real life first person shooters. Thinking of Augmented Reality like this still feels like science fiction and the most obvious example of this would have to be the Holodeck that is used in Startreck. War games Large multiplayer games using Augmented Reality would not have to be constricted to just shooter games. Even to this day there is still large numbers of people who spend their time doing historical re-enactments, often with re-enactments of battles. Augmented Reality would allow these people to add a new dimension of realism to their re-enactments. Augmented Reality could also allow for large fantasy style battle style games. Players could join a battle as a warrior, wizard, assassin or anything else they could think of. There probably is not a single computer game player, or young person who would not enjoy running around a simulated battlefield with a computer generated sword and shield, or casting a spell to throw a big ball of fire at someone. Smaller Games It s nice to think of large scale games like those first person shooter games. But Augmented Reality could also make big improvements to games that are played by a smaller number of people. For example, board games. Imagine playing chess with computer generated pieces. Pieces that could fight, punch and kick each other instead of the player simply taking them off the board. The example of this idea would be from Star Wars when Chewbacca is playing the game with 3D characters on the board. Augmented Reality in Star Wars [11]

17 Entertainment for Children An idea that would be good for children would be either educational games, or just casual fun games based on cartoons or storybooks. Children would be able to interact with the Augmented Reality system and see themselves controlling their favourite characters. They would be able to move around the game as the characters and play games, build things or simply just run around having fun.

18 Conclusion The thought behind this research was to look and see how Augmented Reality has so far been implemented into computer games and to see how it could be used in the future. It has shown that there are a small number of AR systems that are becoming available to consumers, in the form of Project Natal and the Playstation Move. However, most forms of AR have not yet progressed further than tech demos or university laboratory research projects. In conclusion, looking at the complete range of different types of Augmented Reality there is still a long way to go before it becomes part of our everyday life and the games we play.

19 References Pictures [1] Microsoft Surface - Picture taken from - last accessed [2] 2D Matrix Marker Picture taken from - last accessed [3] 2D Matrix Marker displaying 3D image Picture taken from - last accessed [4] Microsoft s Project Natal Picture taken from - last accessed [5] ARhrrrr! Real World view Picture taken from - last accessed [6] ARhrrrr! Augmented Reality game view Picture taken from - last accessed [7]ARQuake Equipment Picture taken from: Thomas.B, Close.B, Donoghue.J, Squires.J, De Bondi.P, Piekarski.W, First person Indoor/Outdoor Augmented Reality Application: ARQuake, last accessed [8]Computer Gamer Evolution Picture taken from - last accessed [9]Nintendo Wii Controller Picture taken from - last accessed [10]Playstation Move Controller Picture taken from ler/ html/ - last accessed [11]Augmented Reality in Star Wars Picture from - last accessed Quotes 1. Trond Nilsen, Steven Linton, Julian Looser, Motivations for Augmented Reality Gaming, 1.2 AR Gaming, accessed from Papers 1. Craig McMahon, Leander Hambley, Augmented Reality Gaming, accessed from

20 2. Trond Nilsen, Steven Linton, Julian Looser, Motivations for Augmented Reality Gaming, accessed from MotivationsARGaming.pdf 3. Ronald T. Azuma, A Survey of Augmented Reality, accessed from =pdf 4. Trond Nilsen, Julian Looser, Tankwar Tabletop war gaming in augmented reality, accessed from 5. C.Geiger, F.Klompmaker, J.Stoecklein, R.Fritze Development of an Augmented Reality Game by Extending a 3D Authoring System, accessed from 6. Troels L.Andersen, Sune Kristensen, Bjørn W. Nielsen, Grønbæk.K Designing Augmented Reality Board Games: The BattleBoard 3D experience, accessed from 7. Carsten Magerkurth, Adrian David Cheok, Regan L. Mandryk, Trond Nilsen Pervasive Games: Bringing Computer Entertainment Back to the Real World, accessed from =pdf 8. Adrian David Cheok, Siew Wan Fong, Kok Hwee Goh, Xubo Yang, Wei Liu, Farzam Farzbiz, Yu Li Human Pacman: A Mobile Entertainment System with Ubiquitous Computing and Tangible Interaction over a Wide Outdoor Area, accessed from e=pdf 9. Ronald T.Azuma The Challenge of Making Augmented Reality Work Outdoors, accessed from Bruce Thomas, Ben Close, John Donoghue, John Squires, Phillip De Bondi, Wayne Piekarski First Person Indoor/Outdoor Augmented Reality Application: ARQuake, accessed from