Computing without Mice and Keyboards: Text and Graphic Input Devices for Mobile Computing Robert Rosenberg A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy of the University of London. Department of Computer Science University College London 1998
Abstract
Abstract 3 Computers have been progressively becoming smaller and more mobile. The past few years have seen computers evolve from the desktop to the notebook to current handheld systems. Each new device is more portable and less cumbersome than last. The next step in this progression is the wearable computer: a computer consisting of parts small enough to be worn as clothing or accessories. However, the portability gained from a wearable computer is lost if the input devices are no smaller than existing ones. The main concept behind this thesis is that, in order for wearable computers to be effective, they need to throw away the mouse and keyboard interfaces in favour of a new style of input. These new interfaces should be designed to be easily transportable, quickly accessible, easily usable in a variety of environments, and minimise interference with real world interactions. In order to test this theory we have built a graphic (Biofeedback Pointer) and a text input device (Chording Glove) based on these design principles. These devices are designed to work in tandem, with the Chording Glove s text input complementing the Biofeedback Pointer s graphic input. The Chording Glove is a text input device made from mounting the keys of a chord keyboard onto the fingers of a glove. Chords are made by pressing the fingers against a firm surface. After 11 hours of use, the average input speed on the Chording Glove was 16.8wpm, with an error rate of 17.4%. The Chording Glove s entire 97 character keymap takes, on the average, 80 minutes to learn, with users needing to look up chords around 0.5% of the time after 6 hours of use. There is evidence of keymap retention after significant absence from the device, but this remains to be proven. Evidence also shows that there is no significant difference in performance on the Chording Glove while standing or sitting, implying that the device is as mobile as intended. The Biofeedback Pointer is a graphic input device controlled by wrist motion. Moving the wrist causes the pointer to move in that direction. The pointer works by measuring the Electromyograms (EMGs) of four of the main muscles used to move the wrist. This data is interpreted by a neural network which is trained for each user. The network takes about half a minute to learn to recognise the user s EMG signals. After the training the user learns to fine tune the pointer control through a biofeedback process. Fitts Law was used to compare the performance of the Biofeedback Pointer with a standard mouse. The average index of performance of the Biofeedback Pointer was found to be 1.06, while the mouse was 7.10. This is about half of the lowest performance of a common graphic input device. Using a more sophisticated neural network or better training may improve the device to a more comparable level. These results show that the design criteria established here for mobile input devices can be used to build workable computer interfaces which encompass both graphic and text input. Hence input devices are shown to exist which are transportable and usable, thereby making wearable computers a viable technology.
Acknowledgements
Acknowledgements 5 First off, I would like to give a special thanks to my supervisor Mel Slater. His constant advice and support throughout the entire course of my degree are very much appreciated. I would also like to thank Mike Craggs for building the Biofeedback Pointer hardware and by being an invaluable resource in helping me understand the practical side of bioelectric measurement. Derek Coppen and Tim Barnes also deserve credit for their help in designing and building the Chording Glove s circuitry. I am extremely grateful to the members of the Virtual Environments and Graphics Group at UCL for just being there to answer questions, offer advice, or just to bounce random ideas off. The subjects of the experiments deserve thanks for volunteering to help. The data they provided were essential to this thesis. Another special thanks go to Anu Hyttinen who one day asked, Is there anything I can do to help? Except for a few computer-generated pictures, all the illustrations in this thesis were drawn by Anu. I am indebted to her continued help despite the fact that every time she thought she was finished, I would find I needed just one more drawing. Lastly, I would like to thank my parents for their unwavering encouragement and support. A final note: The work for this thesis began at Queen Mary and Westfield College. This is where the bulk of the Chording Glove research, including the experiment, was performed. Due to a transfer in 1996, the remainder of the research was carried out at University College London.
Contents 1 Introduction 15 1.1 The Limitations of Mobile Computing........................... 16 1.2 Designing Mobile Input Devices.............................. 16 1.3 Objectives.......................................... 17 1.4 Scope............................................ 18 1.5 Contributions........................................ 18 1.6 Organisation......................................... 19 2 Input Devices and Mobile Computing 20 2.1 Introduction......................................... 21 2.2 An Overview of Text Input................................. 21 2.2.1 Conventional and Alternative Keyboards..................... 21 2.2.2 Soft Keyboards................................... 26 2.2.3 Chord Keyboards.................................. 27 2.2.4 Handwriting Recognition.............................. 30 2.2.5 Glove-Based Text Entry.............................. 32 2.2.6 Voice........................................ 34 2.2.7 Summary of Text Input Devices.......................... 35 2.3 An Overview of Pointer Input............................... 36 2.3.1 Fitts Law...................................... 37 2.3.2 Arrow Keys..................................... 39 2.3.3 Joystick....................................... 39 2.3.4 Trackball...................................... 40 2.3.5 The Mouse..................................... 40 2.3.6 Trackpad...................................... 41 2.3.7 Touch Screens................................... 42 2.3.8 Stylus........................................ 42 2.3.9 Gesture, Eye Tracking, and Voice......................... 42 2.3.10 Bioelectric-Controlled Input Devices....................... 43 2.3.11 Summary of Graphic Input Devices........................ 49
Contents 7 2.4 The State of the Art of Mobile Computing......................... 50 2.4.1 Portable Computers................................. 51 2.4.2 Handheld Computers................................ 52 2.4.3 Wearable Computers................................ 52 2.5 Summary.......................................... 55 3 Wearable Text and Graphic Input Devices 56 3.1 Introduction......................................... 57 3.2 The Chording Glove.................................... 57 3.2.1 The Hardware.................................... 57 3.2.2 Chord Recognition................................. 60 3.2.3 The Chord Keymap................................. 60 3.2.4 Uses for the Chording Glove............................ 71 3.3 The Biofeedback Pointer.................................. 72 3.3.1 Measuring Bioelectric Signals........................... 72 3.3.2 Which Muscles to Use?.............................. 73 3.3.3 Hardware...................................... 75 3.3.4 Pattern Recognition................................. 81 3.3.5 Biofeedback.................................... 87 3.3.6 Uses for the Biofeedback Pointer......................... 88 3.4 Applications......................................... 88 3.5 Summary.......................................... 89 4 The Chording Glove Experiment 91 4.1 Introduction......................................... 92 4.2 Materials.......................................... 92 4.2.1 Shift and Function Key Changes.......................... 92 4.2.2 Finger Sensors................................... 93 4.2.3 The Keymap.................................... 94 4.3 Method........................................... 95 4.3.1 Subject Selection.................................. 95 4.3.2 Tutorial....................................... 95 4.3.3 Chording Sessions................................. 96 4.4 Results............................................ 97 4.4.1 Text Entry Speed and Error Rate.......................... 97 4.4.2 Keymap Learning Rate............................... 97 4.4.3 Portability...................................... 99 4.4.4 Questionnaire Results............................... 99 4.4.5 Analysis of Questionnaire Data and Final Results................. 101
Contents 8 4.5 Discussion.......................................... 106 5 Testing the Biofeedback Pointer 109 5.1 Introduction......................................... 110 5.2 Materials.......................................... 110 5.2.1 The Mouse..................................... 110 5.2.2 The Biofeedback Pointer.............................. 110 5.2.3 The Performance Test Window Setup....................... 110 5.3 Method........................................... 111 5.3.1 The Subjects.................................... 112 5.3.2 The Performance Test............................... 113 5.4 Results............................................ 113 5.4.1 Training....................................... 113 5.4.2 Index of Performance................................ 114 5.4.3 The Neural Networks................................ 116 5.5 Discussion.......................................... 119 6 Future Directions 121 6.1 Sources of Error in the Experiments............................ 122 6.1.1 The Chording Glove................................ 122 6.1.2 The Biofeedback Pointer.............................. 123 6.2 Future Work......................................... 125 6.2.1 The Chording Glove................................ 125 6.2.2 The Biofeedback Pointer.............................. 127 6.2.3 General Input Device Research.......................... 129 6.3 Summary.......................................... 132 7 Conclusions 134 7.1 Introduction......................................... 135 7.2 Review of Design Goals.................................. 135 7.3 Summary of Contributions................................. 136 7.3.1 Review of Input Devices and Mobile Computers................. 136 7.3.2 The Chording Glove................................ 137 7.3.3 The Biofeedback Pointer.............................. 138 7.4 Conclusions......................................... 139 A Glossary 140 A.1 Special Symbols....................................... 141 A.2 Glossary of Physiological Terms.............................. 141 B Relative Character Frequency for Keymap Design 143
Contents 9 C The Chording Glove Hardware 148 C.1 Introduction......................................... 149 C.2 LED Outputs........................................ 149 C.3 Finger Sensors....................................... 149 C.3.1 Double Foil Plates................................. 150 C.3.2 Interlaced Foam.................................. 151 C.3.3 Foil Plate and Single Laced Foam......................... 152 C.3.4 Single Laced Foam and Conductive Unlaced Foam with Foil Backing...... 152 C.3.5 Single Laced Foam and Foil Cover Finger Sensor................. 153 C.3.6 Moulded Resistive Foam with Foil Backing Finger Sensor............ 154 C.3.7 Moulded Low Density Resistive Foam with Foil Backing............. 154 C.3.8 Resistive Foam with Silvered Faces........................ 155 C.3.9 Resistive Foam with Foil Backing......................... 156 C.3.10 Hard Copper Plate................................. 156 C.4 Shift Buttons........................................ 157 C.4.1 Taped Aluminium Foil Plates........................... 158 C.4.2 Taped Copper Foil Plates.............................. 158 C.5 Function Keys........................................ 158 D The Chording Glove Experiment Details 159 D.1 Questionnaires....................................... 160 D.1.1 Preliminary Questionnaire for the Glove Text Input Experiment......... 160 D.1.2 Daily Questionnaire for the Chording Glove Experiment............. 160 D.2 The Tutorial......................................... 165 D.3 The Chording Sessions................................... 178 E Calculating the Effective Width for Fitt s Law 183 F Deriving the Neural Network 185 F.1 Definitions.......................................... 186 F.2 Derivation.......................................... 186 G Technical Specifications for the Biofeedback Pointer 188 G.1 The Physiological Amplifiers................................ 189 G.2 The Analogue-Digital Converter.............................. 190 H The Biofeedback Pointer Software 191 H.1 Introduction......................................... 192 H.2 Using the Application.................................... 192 H.2.1 Button control................................... 192 H.2.2 Menus........................................ 194
Contents 10 H.2.3 Hotkeys....................................... 194 H.2.4 Main Display Area................................. 194 H.2.5 Status Bar...................................... 195 H.3 Internal Processes...................................... 195 References 196
List of Figures 2.1 The QWERTY layout.................................... 21 2.2 The Dvo rák Simplified Keyboard layout.......................... 22 2.3 A generalised split keyboard................................ 23 2.4 Top view of the K-Keyboard, with left and right sections bent downward at 45..... 24 2.5 The preferred slope of a keyboard............................. 25 2.6 Diagram of a key showing preferred sizes and spacing.................. 25 2.7 A generalised chord keyboard............................... 29 2.8 Two methods for handwriting recognition......................... 30 2.9 The five basic strokes in the unistroke alphabet...................... 31 2.10 Diagram of an action potential............................... 44 2.11 Graphs of the EMGs for low muscle use and high muscle use............... 45 3.1 Side view of the Chording Glove.............................. 58 3.2 The finger sensor...................................... 59 3.3 The hand positions in American Sign Language for the letters I, M, N, and Y...... 68 3.4 The chords which resemble the characters they make................... 68 3.5 Normalised finger work distribution............................ 69 3.6 The one degree of freedom of the elbow.......................... 74 3.7 The three degrees of freedom of the wrist......................... 74 3.8 The superficial muscles in the lower right arm which control wrist motion........ 76 3.9 The main hardware of the Biofeedback Pointer...................... 79 3.10 Diagram of the bio-amplifier setup............................. 80 3.11 The signal processing.................................... 82 3.12 The Biofeedback Pointer s simple perceptron neural network............... 83 3.13 The pointer training motion................................. 84 3.14 Error levels of the Backpropagation Neural Network................... 87 4.1 Circuit diagrams for the two types of finger sensors.................... 93 4.2 Sample text from the tutorial session............................ 95 4.3 A sample section of a chording session........................... 96 4.4 Average chording speed in WPM per session....................... 98
List of Figures 12 4.5 Average percent error per session.............................. 98 4.6 Average time spent looking up chords per session..................... 98 4.7 Distribution and mode of fatigue levels per session.................... 99 4.8 Average muscle strain over time.............................. 100 4.9 Muscle Strain vs. Fatigue.................................. 105 4.10 Chording speed vs. Musical ability............................. 106 4.11 Correlations of writing preferences............................. 106 5.1 The button arrangement in the Performance Test window................. 111 5.2 The position of the electrodes on the forearm....................... 113 5.3 Calculating the effective width of the target........................ 114 5.4 Index of performance for the mouse............................ 115 5.5 Index of performance for the Biofeedback Pointer..................... 115 5.6 The neural networks weights used in the experiment for each subject........... 116 5.7 The average and standard deviation of the weights of 4 neural networks generated by subject A............................................ 117 5.8 The total of the absolute values of all the weights for each muscle, averaged over the 4 neural networks produced by subject A........................... 118 5.9 The total of the absolute values of each frequency bin, averaged over the 4 neural networks produced by subject A................................ 118 5.10 The average of the neural networks of all the subjects................... 119 5.11 The total weights for each muscle, averaged over all subjects............... 119 5.12 The total weights for each frequency bin, averaged over all subjects........... 119 5.13 Relative performance on various graphic input devices.................. 120 6.1 Comparison of the average chording speeds including and excluding the old finger sensors122 6.2 Comparison of the average percent error including and excluding the old finger sensors. 122 C.1 Circuit diagram of the LED setup.............................. 149 C.2 Circuit diagrams for the finger sensors........................... 150 C.3 Double Foil Plate Finger Sensor.............................. 151 C.4 Interlaced Foam Finger Sensor............................... 151 C.5 Foil Plate and Single Laced Foam Finger Sensor...................... 152 C.6 Single Laced Foam and Conductive Unlaced Foam with Foil Backing Finger Sensor.. 153 C.7 Single Laced Foam and Foil Cover Finger Sensor..................... 153 C.8 Moulded Resistive Foam with Foil Backing Finger Sensor................ 154 C.9 Moulded Low Density Resistive Foam with Foil Backing Finger Sensor......... 155 C.10 Resistive Foam with Silvered Faces Finger Sensor..................... 155 C.11 Resistive Foam with Foil Backing Finger Sensor..................... 156 C.12 Hard Copper Plate Finger Sensor.............................. 157
List of Figures 13 C.13 Taped Aluminium Foil Plate Shift Button......................... 158 D.1 The Preliminary Questionnaire: Cover page........................ 161 D.2 The Preliminary Questionnaire: Page 1........................... 162 D.3 The Preliminary Questionnaire: Page 2........................... 163 D.4 The Daily Questionnaire.................................. 164 E.1 The button and the approach vector............................. 184 H.1 The Biofeedback Pointer Software............................. 192 H.2 The real-time displayed plot for one EMG signal..................... 194
List of Tables 2.1 A comparison of the normalised performance of six graphic input devices........ 39 3.1 The evolution of the Chording Glove keymap....................... 65 3.2 The chord patterns for numbers............................... 66 3.3 Mnemonic relations between shifted and unshifted characters............... 67 3.4 Common trigrams and their chords............................. 68 3.5 Chords related to other chords............................... 68 3.6 Normalised finger work distribution............................ 69 3.7 Theoretical Input Speed and Error Rate.......................... 70 3.8 The muscles controlling movement of the wrist...................... 77 3.9 The muscles used by the Biofeedback Pointer and the motions they control....... 78 4.1 The reduced keymap.................................... 94 4.2 Top 5 positive and negative comments about the Chording Glove............. 101 4.3 Experimental data for each subject............................. 102 4.4 Correlations of Personal, Experimental, and Opinion data................. 104 4.5 Summary of significant correlations............................ 104 5.1 Comparison of performance results for the Biofeedback Pointer and mouse....... 116 B.1 Character probability sorted by ASCII code........................ 145 B.2 Character probability in order of decreasing probability.................. 146 B.3 Character probability by type................................ 147 G.1 Physiological Amplifier specifications........................... 189 G.2 Specifications for the Pico ADC-11............................. 190 G.3 Output pin connections for the ADC-11.......................... 190