Natural User Interfaces

Transcription

1 Natural User Interfaces Alexandr Goncharov Student ID: University of Southampton 2012 ABSTRACT This paper is aimed to show and analyse past, present and future of user interfaces making an accent on development of Natural User Interfaces. It is also present the newest discoveries in the Human Computer Interaction, which are able to highly participate in forming the new generation of User Interfaces. Author would try to subjectively analyse both positive and negative sides of new-coming technologies, which might be a bit complicated as there is no access to the working prototypes to most of them. General Terms Algorithms, Performance, Design, Reliability, Experimentation, Security and Human Factors. Keywords Human Computer Interaction, Natural User Interface, Augmented reality, Gestures, Near Field Communication, User Interface, Information manipulation. 1. INTRODUCTION People always look for better and faster way to work and communicate with each other. We have to admit that computers, including mobile phones, are playing a big role in these processes. Many people are not aware about NUI. In this paper it will be showed how User Interfaces evolved and the future of HCI 1. But first we need to answer what NUI actually means. Natural User Interface is the way to avoid abstractions and make people interact with computer's virtual objects in such way, as they would operate with real life objects. In other words, make communication between human and computer more natural and transparent. 2. Evolution of User Interfaces In order to understand the context of User Interfaces and see the vector of Interfaces development, it would be useful to look at the key milestones of UI evolution. Advantages and issues associated with every type of HCI model would also be covered in this section. 2.1 Batch Interface In comparison to the present interfaces, with batch interface you have no interaction with computer. All you can do is to insert a program (usually in punch cards [Figure 1]) and wait for results to be printed. It was quite a hard and time-consuming procedure and effectiveness of such systems was not very high. [1] Figure 1: IBM Punch Card Punch card provided opportunity to write programs and even make a batch of several programs performed over a data set with outputs either printed on paper or encoded in other set of punch cards. It allowed program writing and even self-modifying programs, which was a big breakthrough. But it is obvious, that array of cards with set of little holes is not the best interpretation of the program. It makes both creating and modifying of a program very hard. 2.2 CLI 1969-present First Command Line Interfaces were introduced in 1960s, but it was very simplistic model and mostly used in Teletype systems.[1] Later, in 1970, it was UNIX, which introduced real nowadays command line with pipes and shell script.[10] Which was much more sophisticated than just running a program. Command Line interface in cooperation with spreading CRT Monitors made work with computer much more intuitive and fluent. Productivity also rocketed with such kind of interaction between computer and human. 1 HCI Human Computer Interaction Figure 2: Modern FreeBSD Shell

2 There is no doubt that CLI provided a much clearer representation for computer applications development as well as using them, but it still required quite a lot of training in order to use computer. However many IT professionals still prefer CLI to Graphical User Interfaces for their speed and low system requirements. 2.3 GUI The next generation breakthrough was a system designed by Alan Kay and rest of Xerox PARC team. It was a graphical user interface, as we know it now. Before that nobody even saw a computer mouse, because Xerox has invented one. The very first Xerox Alto was introduced in 1981 and it showed the world fully functioning WIMP metaphors, which are Windows, Icons, Menus and Pointer. Figure 4: Drag and Drop feature Here we can clearly see that Natural principles allow people to operate with virtual objects in the direct and intuitive way. If you want to move file from one folder to other you just drag it from origin folder and drop into destination one. This works absolutely fine. Right till the point when you would like to do same trick with two windows opened on different computers. We have such issues as standard GUI and Operation Systems were created without such way of communication in mind. We should also consider input devises, which create another layer of abstraction for computer users. The most common example would be a computer mouse. At this moment it is the most popular pointing device, which is used by people all over world. But if you ever tried to teach an elder person to use computer, you would see that this layer of abstraction between mouse and cursor movement is not so obvious for new users. On of the reasons for this is not just physical, but perceptional as monitor and mouse are laying in different planes [Figure 5] Figure 3:Example of Xerox Environment (Cedar) All this improvements made people much easier to start using computer as well as provided very efficient and handy environment.[1] As you can see we still using this model, however reaching it s limitations. And it makes us to think about creating the brand new UI system, which would be based on NUI principles from the very beginning Reaching standard GUI limitations It is very hard to explain where is the border between current GUI and NUI interfaces, because most advanced graphical interfaces are trying to simulate Natural User Interface features [Figure 4], but I would like to show you when difference becomes visible. Figure 5: Demonstration of Display and mouse planes This makes the mouse movement unusual in the beginning, but most of people are getting used to it. For this reason, touch screen devices are proved to be much more intuitive in comparison to mouse and many other abstracted devices. [5] 2.4 NUI The reasons to develop Natural User interfaces are simplicity and intuitiveness, which allow people work with computers in much easier and effective way. There are no metaphors and virtual objects are treated much more like real ones. Often people recall Minority Report movie, directed by Steven Spielberg, as the best example of Natural User Interface.

3 Figure 7: Microsoft Surface 1 - Diagram Figure 6: Minority Report computer interface 20th Century Fox and DreamWorks The main character starred by Tom Cruise operates objects on a huge display surface using gestures only. It was also shown how fragment of clip could be copied to the external tablet-like device without interrupting playback just by dragging it. This inspired people to try and develop similar systems. 3. Present of NUI In order to make understanding easier, I would like to divide NUI into two equally important research areas: sensors (hardware) and operation systems (software). The first one is developing devices to identify, track and distinguish real world events, objects and gestures. The second one is aimed to provide the environment and react on the sensor-detected events. It might sound simple, but implementation of both is not trivial. Here are examples of current NUI implementations starting from the most familiar ones and introducing much more interesting technologies. 3.1 PixelSense There were many attempts to make interacting systems, which behave in most direct and natural way. One of the most commonly known is Microsoft PixelSense 2. While the first version called Microsoft Surface was quite exciting, but had a huge table with no space for legs. This happened, because of a big infrared camera at the bottom of the case, which was functioning as eyes of system, and projector to output pictorial representation of the screen.[9] [Figure 7] 2 Previously known as Microsoft Surface Figure 8: Microsoft PixelSense running on Samsung SUR40 As you can understand the main purpose of this approach is to provide an operational system, which is able to provide life-like interaction level to virtual data objects and make an illusion as they are lying on the table. But it is only one side of interaction. With the help of 2 million of infrared pixel-sensors all over the screen the system is capable of distinguishing unlimited (but theoretically up to 2 million) finger-points and objects (by form) on the table as well as in the close distance.[4][figure 9] In comparison to the previous version it has an LCD display, so thee is no need to make huge box. This resulted into proper design of the table with space for legs. [Figure 8]

4 In contrast to many other systems g-speak is concentrating on productive and natural way of manipulating any kind of data, starting from spreadsheets and ending with video editing with feature extraction.[7] Figure 9: Output of IR sensors data Shortcomings The PixelSense systems are mainly developed as interactive tables and accent on using it as augmented tables. However using such system for any other kind of work is not just inconvenient it is also unhealthy, as you have to bend over the table to work with it as with personal computer. 3.2 Kinect This system was designed by Microsoft for their game console [Figure 10] it has infrared projector and infrared camera to capture in order to track 3d positioning of objects and able to do 3D reconstruction from image. There is RGB camera, which provides pictorial information. There is also an array of four microphones. Figure 11: John Underkoffler presenting g-speak at TED 2010 The system make maximum use of 3D visualization of data, as it gives users more degree of freedom in comparison to the current devises.[7] Shortcomings The very first thing to understand is that g-speak is not a complete NUI solution, but mostly a software part of it. And this makes system incomplete. There is need of good controllers to make such system working. 3.4 Mezzanine This is similar to g-speak system, produced by the same company, but is much more aimed on collaboration and network communication. One of the major points to mention is that Mezzaninne works with several users, several devises and even several platforms (e.g. mobile phone) simultaneously.[8][figure 12] Figure 10: Microsoft Kinect According to research by Matthew Tang Kinect can be used for gestures recognition with accuracy rate up to 96.1%. But the software suit was fully developed by researcher. [11] Shortcomings Kinect is very good system and can be used for many purposes, however it need a good operation system based on NUI principles. It can also benefit from more accurate movement tracking. 3.3 g-speak It is a user interface, which was aimed to work as NUI from the very first steps of development. The main inspiration source of the Oblong team, who created this system, was the movie mentioned earlier. Some gestures from the movie are even been used in the real system.[figure 11] Figure 12: Work in cooperation between several platforms However this system faced an issue of interaction between big screens and users. The problem was solved with Mezzanine controllers.[figure 13]

5 Figure 15: Leap Motion device Figure 13: Mezzanine Wand Controller These controllers are changing behavior mode according to wand s rotation Shortcomings The wands used to communicate with displays are breaking the intuitive controls, which is mandatory for every NUI system. There is need of better controllers for such system in order to make it a fully-fledged Natural User Interface. Just as g-speak we can see good software solution, but not very good controls solution. 3.5 Leap Motion Here is a relatively new project of gesture capturing device. In contrast to Kinect it does not work on entire body, but on hands area. On the other hand it provides extreme accuracy down to 1/100 of millimeter. The system captures three-dimensional positioning of hands, fingers or pencil and able to recognize individual fingers as well as gestures. [3] Another good point is that device can be used with most of main operating systems and computers. Developers going to support GNU Linux based systems as well Shortcomings In spite of the announcement by Leap Motion developers that they are able to make tracking area bigger the presented device is not able to track for a big distances. 3.6 Speech recognition Talking about NUI many people forget to mention speech recognition software. However it plays a big role especially for disabled people. Such technology gives opportunity to talk with computer and command it to do certain tasks. Many disabled people found speech recognition software very helpful, although all the people can use it in everyday life.[6] Most of new generation satellite navigation systems for cars have a speech recognition feature, which allows driver to create a route without being disturbed by looking at the screen of device.[2] We can also monitor that mobile phones use speech recognition technology as well Shortcomings Some languages are easier to for speech recognition than others. This means that in comparison to gesture recognition, speech recognition cannot guarantee same level of support for all the languages.[12] 4. NUI and Disabled People Figure 14: 3D position tracking and finger identification This small device provides 8 cubic feet of interactive volume, which is not much, but enough for gestures communication.[figure 15] The project is still under massive development and does not share all the technical details. It is hard to estimate the influence the new User Interface can make on disabled people as every technology has different specifications. Kinect for example has not enough accuracy to be operated by leaps only, while Leap Motion technology allow people to operate computer with a minimum of movements, recognizing gestures. In the previous section it was clearly shown that speech recognition is another good example of NUI to be used for disabled people. 5. Conclusion As you can see the HCI sphere is rapidly developed and amount of technologies we can see nowadays is huge. However each of them have some good and bad sides. It would be amazing if we could take the best of ever system in order to create an ultimate NUI experience. The ideal device might be mixture of following: Range of Kinect Accuracy of Leap Motion OS integration of g-speak and Mezzanine Development power Microsoft PixelSense and screen capabilities of

6 In other words we need cooperation of such good sensors as Kinect and Leap motion with Operating Systems as g-speak and Mezzanine. We also should admit that some tasks would still be better and faster done with appropriate UI. As an example, server administrators prefer CLI to GUI or 2D graphics is better done in 2D user environment. Another aspect is system requirements. There is no doubt that additional layer of abstraction requires more computing power in order to function. On the other hand we can monitor that GUI developers trying to implement some NUI elements, but limitations are reached, so there is need to create new User Interface system, based on NUI principles in order to continue future development. 6. REFERENCES 1. Ceruzzi, P History of Modern Computing, MIT Press 2. Garmin Garmin Guidance. Available from accessed 15 November Leap Motion About Leap Motion. Available from accessed 9 November Microsoft Power of PixelSense. Available from accessed 9 November Murata, A. and Iwase, H Empirical Study on the Improvement of the Usability of a Touch Panel for the Elderly-Comparison of Usability between a Touch Panel and a Mouse. IEICE Trans Inf Syst, E86-D(6), Noyesand, J. and Frankish C Speech recognition technology for individuals with disabilities. Augmentative and Alternative Communication. 8, 4, Oblong G-SPEAK. Available from accessed 9 November Oblong Mezzanine. Available from ; accessed 9 November Poenaru R Microsoft Surface What s inside. Available from accessed 9 November Stephenson, N In the Beginning...was the Command Line, HarperCollins 11. Tang, M Recognizing Hand Gestures with Microsoft s Kinect. Stanford University 12. Uebler, U Multilingual speech recognition in seven languages, Speech Communication, 35, 1 2, 53-69