Character Recognition Using the Wii Remote Controller

Transcription

1 Character Recognition Using the Wii Remote Controller Graduate School of Computer Science and Engineering, The University of Aizu, Tsuruga, Ikki-machi, Aizuwakamatsu City, Fukushima , Japan Abstract In this research, we use the Wii remote controller developed by Nintendo as an input device. The Wii remote controller has many sensors and it can connect to a PC and a Wii using Bluetooth. One of the most discriminative sensors is the acceleration sensor. In addition, by using Wii MotionPlus, we can obtain angular velocity data. By these information, we can recognize the posture and trajectory of the Wii remote controller and correct the direction of acceleration by adjusting the posture. Keywords: Character Recognition, Three-Dimensional Posture, Wii Remote Controller 1. Introduction Recently, many tasks that can be performed manually have begun to be processed by using the computer since computers are capable of processing these actions at high speed and with certainty. The computer is also used in science and technology to perform tasks that are very complex and troublesome. However, today, graphical user interfaces have come to be widely used and many kinds of input and output devices have been developed. When the computer first made its appearance, the punch card was used as the input device. However, more recently, the keyboard and the mouse are the most common devices, while the pen table and devices that use brain waves have also begun to be use at the input. The usual output device is the display. Normally, the display shows a two-dimensional image. However, the computer can present three-dimensional images [1][2][3][9][11][12]. We propose that if we use a three-dimensional input device, we can perform three-dimensional intuitive operation. In this research, we use the Wii remote controller as a three-dimensional input device. The Wii remote controller is the input device for the game hardware console known as the "Wii," which was first developed and sold by Nintendo in September The controller is depicted in Fig. 1. The old approach to provide the game input involved the use of many buttons. In contrast, the Wii remote controller uses few buttons, an acceleration sensor, and an IR camera. The Wii remote controller can establish a wireless connection with a computer via Bluetooth, and therefore, we can connect to the Wii remote controller to obtain its state information. The greatest feature of the Wii remote controller is that it has a three-dimensional acceleration sensor. Only two-dimensional information can be obtained with a mouse and a pen tablet, because these devices require a flat surface. However, the Wii remote controller can input three-dimensional data and does not need a flat surface. In the recent research, some researchers using the Wii remote controller and apply to input for baton of orchestra [5][6], 3D game world [4][8] and gesture recognition [7][10]. In this paper, we propose a character recognition method involving the use of the Wii remote controller for three-dimensional posture and trajectory recognition. In this research, we obtain the Wii remote controller s state information and trace the position of the Wii remote controller; we recognize the trajectory as the character. Research on character recognition is older and it can be separated into on-line recognition and off-line recognition. Off-line recognition uses non-temporal data such as images. The Fourier transform and hidden Markov model are used for off-line recognition. On-line recognition uses time data obtained from a pen tablet. On-line data include pressure and pen altitude information, which are not directly observable and which reflect personal features. The Wii remote controller can provide information on personal features because it can input on-line data and contains many kinds of data such as acceleration, angular velocity, and button state data. International Journal of Engineering and Industries(IJEI) Volume3. Number2. June doi: /IJEI.vol3.issue2.3 19

2 Fig. 1. Wii remote controller 2. The specifics of the Wii remote controller First, we explain the specifics of the Wii remote controller. The Wii remote controller has six features: a three-dimensional acceleration sensor, an infrared camera, twelve buttons, a speaker, a rumble device and four LED lamps. In June of 2009, the Wii MotionPlus came to market. This device has a three-axis gyro sensor and the Wii remote controller can thus obtain its angular velocity. A. The Acceleration Sensor The ADXL330 sensor developed by Analog Devices is mounted in the Wii remote controller. This sensor measures acceleration with a minimum full-scale range of 3g and its nonlinearity is 0.3% of the full scale. It is a high accuracy sensor. Fig. 2 shows the Wii remote controller when it is still. The acceleration sensor is subject to little noise, and it detects gravity acceleration. The Wii remote controller maintains calibration data for the acceleration sensor, and the data can be used to calculate individual differences and secular change. We show the individual difference in Fig. 3. In these data, the Wii remote controller records a large difference over each axis and each device. Based on this figure, the acceleration data are detected at 26 points.. Fig. 2. Original acceleration on static state 20

3 Fig. 3. Individual differences among Wii remote controllers B. Infrared Camera A IR camera with built-in image processing is mounted on the top of the Wii remote controller. This image processing sensor is capable of tracking up to four moving objects. When using 8x sub-pixel analysis, the IR camera provides a resolution of The Wii has a sensor bar that includes IR LEDs at intervals of 20 cm, which is set up on the TV. We can use the Wii remote controller as a pointer by recognizing its direction with respect to the sensor bar. Fig. 4. Axis of acceleration sensor and gyro sensor C. Gyro Sensor in Wii MotionPlus Angular velocity data are obtained by connecting the Wii remote controller to a Wii MotionPlus. Angular velocity data are acquired for three axes, as shown in Fig. 4. Fig. 5 shows data for the gyro sensor in static conditions. The angular data contain more white noise than the acceleration sensor data; thus, we need to correct for the posture based on the direction of gravity, calculated using the acceleration sensor. In Fig. 5, the average yaw and roll are similar, while the pitch is different. This is because the Wii MotionPlus has 2 kinds of gyro sensors: a dual-axis gyroscope, the IDG-600 made by EPSON-TOYOCOM, and a single-axis gyro sensor, the XV3500CB made by InvenSense. Fig. 5. Angular velocity obtained by gyro sensor in static conditions. 21

4 The Wii remote controller maintains calibration data for the acceleration sensor. However, it does not have calibration date for the angular velocity. We researched the relation between sensor input and angular data by performing an experiment. As a result, angular data are calculated by the following equation. roll base, yaw base and pitch base are calculated as the average of the gyro sensor input that is obtained when a Wii remote controller is not moving. Roll = (Roll input -Roll base ) , if Roll input > Roll base (1) Roll = (Roll input -Roll base ) , if Roll input Roll base (2) Yaw = (Yaw input -Yaw base ) , if Yaw input >Yaw base (3) Yaw = (Yaw input -Yaw base ) , if Yaw input Yaw base (4) Pitch = (Pitch input -Pitch base ) , if Pitch input >Pitch base (5) Pitch = (Pitch input -Pitch base ) , if Pitch input Pitch base (6) The Wii remote controller has a fast flag to represent large-scale data. When the fast flag is 1, the angular data represented are divided by 5. Fig. 6 presents the data when the fast flag is changed, and Fig. 7 presents the data multiplied by 5, when the fast flag is one. Fig. 6. Angular velocity data and fast flag obtained by Wii MotionPlus Fig. 7. Corrected angular velocity data 3. Character recognition method To trace a Wii remote controller in three-dimensional space, we calculate the global acceleration using the original acceleration and the posture of a Wii remote controller. When we obtain the y-axis acceleration, we cannot judge the direction of the Wii remote controller s motion because the axis of 22

5 original acceleration moves with the posture. We calculate the acceleration in three-dimensional space according to the following procedure. First, we calculate the posture of the Wii remote controller based on its angular velocity and the direction of gravity. Second, the original acceleration is rotated based on the posture of the Wii remote controller, to calculate the modified acceleration. Third, the influence of gravitational acceleration is removed from the modified acceleration. After that, we obtain an acceleration that represents the movement of the Wii remote controller. Next, we calculate the velocity and position of the Wii remote controller using the modified acceleration. In this section, we explain each step. A. Calculate Posture of a Wii Remote Controller The posture of the Wii remote controller is calculated using the angular velocity, obtained from the gyro sensor. However, because the gyro sensor has a lot of white noise, we cannot obtain an accurate posture. We need to reset the posture using the direction of gravity. The posture of the Wii remote controller is represented by the yaw, roll and angular pitch, as in Fig. 8. We can calculate the roll and pitch angle based on the acceleration, and yaw is corrected using the IR camera. The roll and pitch angle are reset using the following equation, as in Figs. 8 and 9: pitch arccos( y z) (7) roll arccos( x z) (8) Fig. 8. Calculation of pitch angle using gravity acceleration Fig. 9. Calculation of roll angle using gravity acceleration 23

6 Normally, we cannot correct the yaw angle using gravity acceleration. The yaw angle is corrected using the direction of the IR sensor bar. However, it is difficult to calculate the yaw angle based on the direction of the IR sensor bar, because we cannot know how long the distance is between the IR sensor bar and the Wii remote controller. Therefore, when the center of the LEDs on the sensor bar is in front of the Wii remote controller, the yaw angle is corrected. B. Convert Local Coordinates to Global Coordinates The modified acceleration is calculated by rotating the original acceleration based on the posture of the Wii remote controller, as in Fig. 10. After calculating the modified acceleration, the gravitational acceleration is detected as the y-axis vector. We parse out the gravitational acceleration to calculate the movement of the Wii remote controller. Fig. 10. Rotated posture of the Wii remote controller and corrected acceleration 4. Experiments To verify that the posture and the movement of the Wii remote controller are acquired using the proposed method, we obtain data in various situations. Next, the character recognition is performed using some features that are calculated based on our proposed method. A. Modifying Acceleration Data Fig. 11 shows the adjustment of the original acceleration based on changes in posture. At first, the Wii remote controller is on the desk and from 253 to 561, the Wii remote controller is tilted. Fig. 12 depicts the modified acceleration. In this experiment, the modified acceleration should become 0, because the Wii remote controller does not move. However, the figure shows some noise when the posture of the Wii remote controller is changed, because we cannot erase the impact of rotating, and posture recognition is subject to some noise. Fig. 13 shows the original acceleration of the Wii remote controller when it moves left, up, right and down. When the Wii remote controller moves up, the direction of gravity is changed. This can be seen as the Wii remote controller moves up from observations 209 to 417. However, the modified acceleration that is calculated using our proposed method eliminates gravitational acceleration and posture error, as can be seen in Fig. 14. B. Discussion. In this section, we explain the results of character recognition. We test using the original acceleration, the modified acceleration calculated in Section 2, and the posture vector of the Wii remote controller. We use 500 words that are obtained from 10 people write 0 to 9 numbers at 5 times. Fig. 15 shows the result of the experiment, and it can be seen that the modified acceleration provides the best recognition rate. Fig. 16 shows the result of each number sample. In this result, the recognition 24

7 rate for the number 0 is bad because it is similar to 6. The velocity value is not obtained accurately because it is calculated using integration and posture data. Because posture data contain considerable noise and errors, the velocity data that are integrated from the posture have errors as well. Our proposed method shows the best result because some people change the Wii remote controller s posture while they are writing a character. Fig. 11. Original acceleration when the Wii remote controller is tilted Fig. 12. Modified acceleration when the Wii remote controller is tilted Fig. 13. Original acceleration when drawing a square 25

8 Fig. 14. Modified acceleration when drawing a square Fig. 15. Result of character recognition Fig. 16. Result for each number 26

9 5. Conclusion We have shown that our method can be used to recognize numbers As a result, we recognize numbers 95.56% rates using our proposed feature for numeral recognition. We recognize posture and movement of wiimote that calculated using angular velocity data. However, we cannot recognize a trajectory, the main reason being that the sensor generates noise. Analog devices generate noise such as white noise and are associated with errors such as measurement error, conversion error, and calibration error. In the future, we intend to eliminate errors in the angular velocity through noise filtering. For researching the sensor characteristics, we require a measurement instrument and we must determine the principles and characteristics of the gyro sensor. If we eliminate the noise, we can accurately obtain the trajectory of the Wii remote controller. 6. References [1] Sreeram Sreedharan, Edmund S, Zurita, Beryl Plimmer: 3D Input for 3D World, Proceedings of the 19th Australasian Conference on Computer-Human Interaction: Entertaining User Interfaces, Ambience & 3D interfaces, [2] Natsue Yoshimura, Naoaki Itakura: A Study on Brain-Computer Interface Using Transient Visual Evoked Potentials, Japanese Society for Medical and Biological Engineering, , [3] Morihiko Tamai, Wanmin Wu, Klara Nahrstedt, Keiichhi Yasumoto: A View Control Interface for 3D Tele-immersive Environments, A View Control Interface for 3D Tele-immersive Environments, , [4] Torben Schou, Henry J. Gardner: A Wii Remote, a Game Engine, Five Sensor Bars and a Virtual Reality Theatre, Proceedings of the 19th Australasian Conference on Computer-Human Interaction: Entertaining User Interface, Ambience & 3D interfaces, , [5] Bernd Bruegge, Christoph Teschner, Peter Lachenmaier, Eva Fenzl, Dominik Schmidt, Simon Bierbaum: Pinocchio: Conducting a Virtual Symphony Orchestra, Proceedings of the International Conference on Advances in Computer Entertainment Technology, , [6] Chris Kiefer, Nick Collins, Geraldine Fitzpatrick: Evaluating the Wiimote as a Musical Controller, International Computer Music Conference, [7] Henry Owen Wiggins: Gesture Recognition of Nintendo Wiimote Input Using an Artificial Neural Network, Cognitive Systems, [8] Joao Bernardes, Ricardo Nakamura, Daniel Calife, Daniel Tokunaga, Romero Tori: Integrating the Wii Controller with enjine: 3D Interfaces Extending the Frontiers of a Didactic Game Engine, ACM Computers in Entertainment, Vol.7, No.1, Article 12, [9] Shoji Yamada: Improvement and Evaluation of an EEG Keyboard Input Speed, Institute of Electronics, Information, and Communication Engineers, Vol. J79-A, No.2, , [10] Robert Nebelrath and Jan Alexandersson: A 3D Gesture Recognition System for Multimodal Dialog System, 6th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, 46-51, [11] S.Y. Chen and Y.F. Li "Vision Sensor Planning for 3-D Model Acquisition", IEEE Transactions on Systems, Man and Cybernetics, Part B, Vol. 35, No. 5, pp , Oct [12] S. Chen, Y. Wang, and C. Cattani, "Key Issues in Modeling of Complex 3D Structures from Video Sequences", Mathematical Problems in Engineering, Vol. 2012, AID , 17pages,