Augmented Reality: A New Layer of Service

Augmented Reality: A New Layer of Service

Augmented Reality: A New Layer of Service By the Editors at Nearshore Americas: Ask someone for a definition of augmented reality and often you will get a cinematic illustration rather than a literal definition. The movie Iron Man 2, where the main character engineers his weaponized flying suit by manipulating graphics hanging in the air, for example, or less dazzling, the Terminator's visual data display that identifies objects such as human targets. But there are also basic examples from the real world, such as the first-down line shown on the field during TV broadcasts of U.S. football games. Definitions of augmented reality run from simple to verbose. At its most fundamental, AR refers to the use of computer-generated visuals or other sensory elements to augment, supplement, or enhance reality. More succinctly, AR involves overlaying information on the real world. One of the earliest definitions came from researcher Ronald Azuma in 1997, and it still applies: AR combines real and virtual elements, is interactive, happens in real time, and appears in 3-D. Creative designers and innovative software developers are exploring breakthrough implementations for personal and business use. Forward-looking service providers are talking to clients about solutions that would benefit from a layer of augmentation. The potential for AR to spawn new companies and investment opportunities is considerable. And although some of the underlying technology is ready, hardware and software is available, and there is no shortage of real examples, AR is in its early stages. Basically, with augmented reality, we are at the same place we were with social media about five years ago, says Juan Pereyra, VP of Technology for the Creative & Social Studio at Globant, a software developer and creative services provider based in Buenos Aires. Most everyone recognizes the great potential, but no one is quite sure yet what to do with it in terms of mainstream or business applications. No one is sure how to make it as popular as, say, Facebook. The earliest practical example of augmented reality was developed at Boeing in the early 1990s, and involved projecting aircraft maintenance and repair instructions to workers wearing head-mounted displays. (A Boeing engineer named Thomas Caudell is often credited with coming up with the term augmented reality. ) With advances in this type of display, augmenting reality with text, graphics, and even video was feasible. Military applications became prevalent, with pilots wearing helmets that incorporate AR screens, or heads-up systems that flash data onto an aircraft windshield. Head-mounted or heads-up displays, like those used by the Boeing workers and fighter-jet pilots, are usually embedded in helmets or in large binocular-like goggles and stereoscopic glasses. They use special mirrors and lenses to project computer-generated text, images, or video. Newer devices combine both the digital information and the real world to present an augmented perspective. A Canadian company called Arcane Technologies has developed helmets, used by the U.S. Army, that can overlay information like satellite data or building blueprints upon a soldier's field of vision. Some researchers say that this kind of technology will eventually be incorporated into AR eyeglasses that look like an ordinary pair of spectacles.

Head-mounted or heads-up displays, like those used by the Boeing workers and fighter-jet pilots, are usually embedded in helmets or in large binocular-like goggles and stereoscopic glasses. They use special mirrors and lenses to project computer-generated text, images, or video. Newer devices combine both the digital information and the real world to present an augmented perspective. A Canadian company called Arcane Technologies has developed helmets, used by the U.S. Army, that can overlay information like satellite data or building blueprints upon a soldier's field of vision. Some researchers say that this kind of technology will eventually be incorporated into AR eyeglasses that look like an ordinary pair of spectacles. Two other display technologies are also used in augmented reality: spatial and handheld. Spatial displays will provide an experience most like AR in the movies. This technique uses projection systems to display graphical objects (text, images, virtual controls, etc.) onto physical objects. The user does not need any special equipment to see the augmented content. Because they are not tied directly to an individual user, spatial systems can scale up to serve a group of users. Spatial systems can be implemented with haptic devices or a touch interface; users interact with the projected elements by tapping, twisting, turning, or moving them. A collaboration session among design engineers, manipulating components of a new machine, for example, would be one productive application of such display technology. Handheld displays include mobile devices like smartphones and small tablets. Many of these devices incorporate the essential ingredients for practical AR, including a camera, GPS, position sensors, and a usable screen. They are portable, and thus allow the user to interact with AR on location, wherever and whenever it makes sense. However, they do currently require that the user carry the device and typically hold it out in front of him or her. Because of the sheer number of smartphones being purchased around the world, this type of display will be prominent in AR applications. It is certainly attracting many AR developers. Other key elements of AR include some sort of tracking or location hardware (e.g., GPS, RFID), input method (e.g., touch screen, haptic pinch glove), CPU, and various sensors. But the most challenging ingredient is the core software: the algorithms and intelligence to recognize objects, to sense location and understand geographic coordinates, to track the user's movements and orientation, to see where and what the user is seeing, to not only detect points of interest but to detect points of interest that interest the individual user, to align virtual objects with real. Although this fundamental software intelligence that will bring sophisticated, sci-fi-like AR to the masses currently exists mainly in R&D labs or in the minds of pioneers, there are quite a few examples of rudimentary AR available to mainstream users now. Some might seem trivial, but they do allow a glimpse into what future applications will be like. Harbinger Apps Many of the early-stage AR applications for consumers work with smartphones like the iphone and the Android. Globant's Pereyra mentions Foursquare as a spartan form of mobile augmented reality. You are out on the town, in an unfamiliar neighborhood, and the app will give you a list on your phone of all the places around you, including ones outside your viewing area, along with loads of information about them. The app requires manual check-ins, which are cumbersome and not always reliable, but it illustrates the possibilities.

Another good example is WorkSnug, an iphone app that pinpoints Wi-Fi hotspots and available places to plug in and do some work, including rental office space and coffee shops. As well as location, the app displays details about power connections, atmosphere, and noise levels. Le Bar Guide is designed to help thirsty people find a convenient drinking establishment by pointing an iphone down the street ahead. The app responds by displaying the names and locations of nearby bars, as well as reviews. Its range is only 20 feet, and some early users have complained about its accuracy, but Le Bar does demonstrate AR's function of finding specific information tied directly to geographic context. A similar but more advanced app is Monocle, which uses GPS and an online database to generate a list of local restaurants along with reviews. One of the most often mentioned examples of mobile AR is Layar, which its Netherlands-based developers call an augmented reality app that shows you the things you can't see. Point an iphone or Android phone's camera at a building and Layar will display information about it, including whether any businesses housed there are hiring, a review if it's a restaurant, and even its history. Game-playing is another domain in which AR is advancing. First-person-shooter games have long incorporated or mimicked AR by displaying coordinates or information about opponents, for example, over top the on-screen battleground or other scene of action. But that's primitive compared to Nintendo's new 3DS gaming system, which comes with augmented reality hardware cards that can create the illusion of characters or objects appearing right in the room. Microsoft's Kinect video controller for the Xbox is the basis of some portentous strides in the AR user interface, allowing control of objects by gesturing and giving players the chance to see themselves within the game onscreen. This sort of natural UI has long been a goal of interface designers, and if it can be perfected for games, it can also be adapted to manipulate objects in other AR environments. (Again, see Iron Man 2.) Forward-thinking people in the marketing industry have enthusiastically embraced the possibilities of AR. The makers of Ray-Ban sunglasses, for example, designed an AR website that allows visitors to see how they might look wearing a particular pair of glasses. Clothing companies are working on similar applications, enabling shoppers to see onscreen, at home, how they might look in that dress or suit. In Japan, a company called Dentsu developed the ibutterfly app, which displays digital butterflies that the user sees through the iphone camera viewer. The animated butterflies are promotional: They contain coupons for stores and services in the local area. Matthew Szymczyk, CEO of creative agency Zugara which developed the Fashionista dressing-room app for an online clothing boutique wrote recently: "As with any emerging technology, there's the question of ROI and how to measure it... For marketing, there are already signs that AR has increased awareness and engagement for certain product campaigns. In addition, AR can be used to increase purchase conversions in e-commerce and drive purchase intent at retail." Promise, Problems of a New Platform AR has been successfully implemented in niche applications that demonstrate the technology's usefulness, and the first wave of mainstream, consumer-oriented apps suggest the potential for wide appeal and pervasive impact in the future.

Many of the components are coming together. There are design libraries like the ARToolkit, introduced by Hirokazu Kato in 1999, that provide video tracking capabilities so that the AR system knows the user's viewpoint one of the technical challenges of AR. Companies like Total Immersion are attracting substantial funding to continue development of AR solutions. MIT Media Lab researchers Pattie Maes and Pranav Mistry have cobbled together a mobile device they call SixthSense that incorporates all the essential elements of AR: smartphone, camera, GPS, tiny projector. The unit views the surrounding environment, transmits location data, pulls down associated content from the web, and projects that information onto a convenient surface. Wearing special fingertip caps, the user can interact with the projected data. The researchers estimate the cost of the apparatus to be about $350. But before AR can become widespread and practical, many technical issues still need to be resolved. The core intelligence mentioned earlier will have to be designed and coded. Computers will have to become a lot better at recognizing objects. The big thing is, most of the objects we will want to interact with cannot be easily identified by computer, says Juan Pereyra. If I point my phone at a car I see on the street because I want to know more about it, where I can buy it, gas mileage, and so on, the app has to first of all be able to recognize that it's a blue 2011 Corolla. There's a lot of AI involved that we still need to develop. Some of the technology and algorithms are there, but important parts of the mix, like shape recognition and spatial analysis, being able to interpret all the necessary information from the physical world, those things are not there yet. Furthermore, he says, essential communications issues will have to be settled and new. There are no clear protocols for devices to communicate with each other or for people to interact with this new layer of reality. Competing operating systems for mobile devices complicate the situation. Some sort of standards and infrastructure for interoperability will be required. At software and design companies such as Globant, where augmented reality services will become part of the standard creative roster, engineers will be tasked with building effective user interfaces for client applications. UI development will involve drawing in 3-D, and things will have to appear in perspective, and behave like reality so that people will try to touch them, Pereyra says. Ultimately we will have to design not for smartphone or tablet screens but for 3-D glasses that project onto the real world. The promise of augmented reality is evident in the many successful, although niche, applications. If hardware and software challenges are solved, AR could become as pervasive and immersive as it is in futuristic movies. As with social media, there will be personal and business uses. Everyone that has a business or service should be interested in AR, Pereyra says. We do not yet have the standard AR platform, but let's imagine that we do: You're on the street. You can tap into a service that tells you not only where a particular shop is but how to get there, what's going on there today, any special deals or activities, all in the same interface. We've been thinking of an app that when you walk by a shop that offers discounts for something you're interested in, your mobile device would vibrate or beep. Marketing would have to solve the problem of getting the right service to the right person, but AR would let them put a new layer on the client's mind. Businesses would want to be on that platform. Think of whatever you do now with a computer, whatever you check on the Internet, being displayed in front of you at any time, augmenting your reality. Juan Pereyra, VP of Technology, Creative & Social Studio, Globant