Handwritten Pattern Reproduction Using Pen Acceleration and Angular Velocity

Transcription

1 Handwritten Pattern Reproduction Using Pen Acceleration and Angular Velocity TOHRU MIAGAWA, OSHIMICHI ONEAWA, KAUNORI ITOH AND MASAMI HASHIMOTO Dept. of Information Engineering Shinshu University 4-7- Wakasato, Nagao City, JAPAN Abstract: - Human-computer interaction devices using handwriting movement have been developed. But almost all the pen-based devices have a special plane, which gives us a sense of touch different from that of ordinary writing with a pen on paper. We tried to reproduce the handwritten patterns using acceleration and angular velocity of pen movement. In writing, the pen is rotated by writing movement, therefore, the acceleration on the normal coordinate system is calculated from accelerometer and gyroscope outputs. By twice integration of the calculated acceleration, we succeeded in reproducing handwritten patterns. Key-Words: - accelerometer, gyroscope, handwritten pattern reproduction, human motion capture, handwriting acceleration, handwriting angular velocity Introduction A pen-based interface for human-computer interaction is widely used in the personal digital data assistants (PDA) terminals, computer aided design (CAD) tools, tablets and digitizers. It enables us to easily operate information tools and to input handwritten patterns and characters. But almost all the interfaces have the special plane[2], therefore not only the miniaturization of the devices is limited by the size of the special plane, but also the input area for handwriting is limited. Besides, in writing on the special plane, it gives us a sense of touch different from that of ordinary writing on paper. We have noticed acceleration in handwriting movement, and have applied the handwriting acceleration signals to a new pen-based input device without a special plane. In our previous studies[6][7], we could get the exact acceleration of the handwriting movement, and the patterns can be reproduced as exactly as the written pattern in the condition of keeping pen vertical. But the pen is likely to slant, and the slant angle varies as the pen moves. Therefore, the output signals of the accelerometers are affected by gravity and the pen slant. To eliminate the influence, accelerometers were attached not only to the lower pen position but also to the upper pen position[0]. Then the pen slant angle was calculated from the accelerometer outputs[], and the acceleration of handwriting movement was extracted. But for more exact reproduction of the handwritten patterns, 3D measurement and analysis of handwriting movement is essential. In a proposed system of this paper, accelerometers and gyroscopes are attached to the pen for extracting 3D handwriting movement. Accelerometers have been used in many studies for sensing the human body movement and physical activity[5][8]. The system with accelerometers and gyroscopes has been researched in the area of inertial aviation system. The sensors of inertial aviation system are highly precise ones but the system is too large to measure a human body movement. Recently a small accelerometer and gyroscope have been available, and they have been applied to inertial measurement system of the human body movement[3]. They are also small enough to be attached to the pen. The accelerometer and gyroscope attached pen in the proposed system can be dealt with as well as a usual ball-point pen. The studies using handwriting acceleration and angular velocity exclusively in the area of signature verification[4][9] were reported, and the main concerns have been extraction of the personal identification data. Until recently, the performance of the accelerometer and the gyroscope weren t good enough to reproduce the handwritten patterns. In this paper, we show how we try to measure 3D pen movement and to reproduce the handwritten patterns from the pen acceleration and angular

2 z S z S a S z a x y x y x y Fig. 3D Vector on Coordinate System S. velocity. The accelerometers and gyroscopes are attached to the pen for measuring 3D pen acceleration and angular velocity. Handwritten patterns are 2D pattern, but the pen moves 3-dimensionary in writing. The acceleration of the moving pen is calculated from sensor outputs. The velocity of the moving pen is calculated from the pen acceleration, and the pen position is calculated from the pen velocity. By the pen position signals, the handwritten pattern can be reproduced. 2 Vector Transformation to Normal Coordinates For extracting the pen acceleration while the pen is moving with rotation, the measured acceleration signal has to be transformed to the normal coordinates. In Fig., 3D acceleration vector a on the coordinate system S with axes, and orthogonal to each other is shown. The coordinates of vector a is defined as (x y z) T. Next, it is assumed that the coordinate system S is rotated around the axis as angle α, the axis as angle β and the axis as angle γ. The coordinate system S after the rotation of S coordinate system is shown in the Fig.2. The acceleration vector a on the S is defined as (x y z ) T.The coordinates transformation from the coordinates on S to the coordinates on S becomes as follows, x x y = A y z z () Fig.2 3D Vector on Rotated Coordinate System. where A is a coordinates transformation matrix. Actually, the calculation of the exact components of the matrix A becomes very complex. But in the case of sampling the data fast (almost less than [deg/sample] of angular velocity), a sine function of rotation angle can be regarded as almost 0, and a cosine function can be regarded as almost. Then the approximate matrix of the coordinates transformation matrix is given by cos β cosγ A = sin γ sin β sin β sinα.(2) cosα cos β Coordinates a of vector a on coordinate system S is transformed as a = Aa (3) where a is coordinates of vector a on coordinate system S. Coordinates b 2 of vector b on coordinate system S 2 rotated from S is transformed as follows, b sin γ cosα cosγ sinα = A b (4) where b is coordinates on coordinate system S, b is coordinates on coordinate system S and A is a coordinates transformation matrix from S to S 2. b = Ab 2

3 3-Axes Angular Velocity 3-Axes Accel. Signal LPF LPF ' Coordinates Transform Writing Acceleration Integration ' ' Velocity Linear Rectification G Integration Position 2D or 3D Pattern Synthesis Fig.3 Gravity Vector and Sensing Directions of Accelerometers. From equation (4), the coordinates b becomes as following equation. b = AA b 2 (5) Coordinates c n of vector c on coordinate system S n rotated n times from the coordinate system S is expressed as = ALA n b n. (6) c From equation (6), an acceleration vector on rotated coordinate system can be transformed to the vector on the normal coordinate system. 3 Reproduction Algorithm for Handwritten Patterns 3. Pen Acceleration of Handwriting While the pen is slanted in writing, the direction of the pen moving and sensing axis of the accelerometer are different each other, and the signal of the accelerometer includes gravity effect (Fig.3). It is important to eliminate gravity effect and to extract the acceleration of handwriting movement. An acceleration vector a s (t) measured by accelerometers at time t becomes as a ( t) = G( t) + a( t) + a ( t) s c (7) Handwritten Pattern Fig.4 Reproduction Process of Handwritten Patterns Using Pen Acceleration and Angular Velocity. where a(t) is an acceleration vector of handwriting movement, G(t) is gravity vector and a c (t) is a centrifugal force vector of circular motion of the pen. For simplicity, the centrifugal force vector a c (t) is assumed 0 in this paper. The pen acceleration vector is measured on the coordinate system of the pen itself, and it has to be transformed on the normal coordinate system with considering the pen rotation. Angular velocity of the pen rotation is measured by gyroscopes attached to the pen. Pen rotation angle is calculated from the angular velocity by the integration. Information of the pen rotation is used for coordinates transformation of the handwriting signals. 3.2 Reproduction Process for Handwritten Patterns Fig.4 shows the reproduction algorithm of handwritten patterns. The reproduction algorithm is composed by the software for the computer. First, signals from accelerometers and gyroscopes are input to the computer. Second, for eliminating the unnecessary frequency element, the acceleration signals are passed though the IIR(Infinite Impulse Response) type LPF(Low Pass Filter) whose cutoff frequency is 50[Hz]. Third, 3D pen acceleration

4 Gyroscopes Personal Computer PCI Bus Digital I/O Board Counter Board A/D Board a a 05[mm] Accelerometer Outputs Gyro Sensor Outputs a Pen 65[mm] Fig.5 Accelerometers and Gyroscope Attached Pen. Fig.6 Pen-Based System for Handwritten Pattern Reproduction. vector for the handwritten pattern reproduction is calculated using the signals of accelerations and angular velocities. Fourth, the pen velocity vector is obtained by the integration of the acceleration vector. Fifth, the pen position vector is obtained by the integration of the velocity vector. The pen position vector d(t) is calculated from the handwriting acceleration a(t) as follows. d( t ) = a( t) dtdt (8) Finally, the handwritten pattern is shown by the 2D or 3D synthesis using the pen position vector. 4 Pen-Based System and Results of Handwritten Pattern Reproduction 4. Pen-Based Input System According to the previous experimental results, while we are writing a pattern of approximately 5[cm], the handwriting acceleration usually ranges within from to +[G], and angular velocity ranges from within 300 to +300[deg/s]. The maximum acceleration is expected to range from 2 to +2[G] including gravity effect. The accelerometers used in this study should be small enough to be attached to the pen. The IC type two-axis accelerometer(analog Devices ADL202) is used in the system. The full-scale input range of this accelerometer is from 2 to +2[G]. The package of this accelerometer is small enough to be attached to the pen. For the measuring the angular velocity of the pen, piezoelectric vibrating type gyroscope(murata Manufacturing ENC-03J) is used. The full-scale input range of the gyroscope is from 300 to +300[deg/s]. The gyroscope can sense the one-axis angular velocity. The accelerometers and gyroscopes are attached to the pen for measuring the 3D acceleration and angular velocity of the pen. The sensor-attached pen is shown in Fig.5. Two accelerometers and three gyroscopes are attached to the pen. For writing on paper using this pen, a ball-point pen is attached at the pen point. Fig.6 shows the outline of the system. The outputs of accelerometers and gyroscopes are converted to the 2bits digital signals. The converted data are input into the PC(Personal Computer) through the digital I/O board. The input data are calculated for the reproduction of the handwritten patterns by the PC.

5 The sampling interval is set to 2[ms]. For the transmission of the acceleration signals from the pen to digital I/O board, wires are used in our system. But it isn't difficult to make the wireless system using available technologies. The pattern reproduction from the acceleration data is executed after data sampling has finished. 4.2 Reproduction Results of Handwritten Patterns The sensor-attached pen was held by a hand, and output signals of accelerometers and gyroscopes are measured while writing patterns on the paper. In handwriting, we paid attention only to writing clearly. The sensor- attached pen is held normally. The written patterns are reproduced one after another. The reproduction results of alphabet character of b and star shape are shown in Fig.7 and Fig.8. Fig.7(a) and Fig.8(a) show the trace of the handwritten patterns written on the paper. Fig.7(b) and Fig.8(b) show the 2D reproduced patterns of and axes. Fig.7(c) and Fig.8(c) show the 2D patterns of and axes, and Fig.7(d) and Fig.8(d) show the 2D patterns of and axes. The square box sizes in the figure are 5x 5[cm 2 ]. The arrows show the axes of 2D plane and the writing direction at the beginning of each pattern. Each reproduced pattern is similar to the handwritten pattern looked from slanted direction. The normal coordinate system for reproduction is set to the coordinate system of the pen at the beginning of writing, and the initial pen slant causes the direction of normal coordinate system. Some differences of shape and size are shown between handwritten patterns and reproduced patterns, but we can recognize these characters clearly. For more exact pattern reproduction, we consider that the following further improvements are necessary; elimination of the sensor noise, more exact setting of the sensor axes and consideration of centrifugal force effect of the pen circular motion. 4 Conclusion In this paper, we showed the 3D measurement method for acceleration and angular velocity of handwriting movement and reproduction method of the handwritten patterns using measured signals. The accelerometers and gyroscopes are attached to the Fig.7 Reproduction Results of Handwritten Alphabet b. Fig.8 Reproduction Results of Handwritten Star Shape. pen for measuring the writing movement. Then the pen acceleration vector on the normal coordinate system is calculated from the sensor outputs. In the pattern reproduction process, the pen velocity vector and the pen position vector are obtained. Using the

6 pen position signals, handwritten patterns could be reproduced. The experimental results show the effectiveness of proposed system and reproduction algorithms for handwritten patterns. We have improved our system for interface of human-computer interaction. It is considered that our pen-based device can be applied to handwritten character recognition or a signature verification system. In the reproduction process, many kinds of information, such as the pen acceleration, angular velocity, slant angles, velocities and patterns are obtained. If a special recognition algorithm or a signature verification algorithm for proposed system is available, the proposed pen-based device can be a more practical interface for handwriting. Acknowledgments We would like to thank Mr. Kaneko and Dr. Arai in Shinshu University for their helpful comments. We would also like to thank HIOKI E.E. Corporation for their support of the study. IEICE D-I (in Japanese), Vol.J76-D-I, No.0, 993, pp [7] H. Ishikawa,. onezawa, K. Itoh and M. Hashimoto, Reproduction of character pattern from pen acceleration in handwriting, Proc. of 24th Conference on Image Engineering (in Japanese), Vol.-4, 993, pp [8] J. R. W. Morris, Accelerometry-A Technique for The Measurement of Human Body Movements, J. Biomechanics, Vol.6, 973, pp [9] R. Plamondon and G. Lorette, Automatic Signature Verification And Writer Identification The State Of The Art, Pattern Recognition, Vol.22, No.2, 989, pp [0] T. Miyagawa,. onezawa, K. Itoh and M. Hashimoto, A New Computer Input Device Using Accelerometers for Handwritten Character, Proc. of Second Magneto-Electronics International Symposium, 999, pp References: [] A. R. Schuler, A. Grammatikos and K. A. Fegley, Measuring Rotational Motion with Linear Accelerometers, IEEE Trans. on Aero. Elect., Vol.AES-3, No.3, 967, pp [2] C. C. Tappert, C.. Suen and T. Wakahara, The State of the Art in On-Line Handwriting Recognition, IEEE Trans. on Pattern Anal. Machine Intell., Vol.2, No.8, 990, pp [3] E. Foxlin, Inertial Head-Tracker Sensor Fusion by a Complementary Separate-Bias Kalman Filter, Proc. of VRAIS'96, 996, pp [4] F. Leclerc and R. Plamondon, Automatic Signature Verification; The State Of The Art , International Journal of Pattern Recognition and Artificial Intelligence, Vol.8, No.3, 994, pp [5] G. A. L. Meijer, K. R. Westerterp, F. M. H. Verhoeven, H. B. M. Koper and F. Hoor, Methods to Assess Physical Activity with Special Reference to Motion Sensors and Accelerometers, IEEE Trans. on Biomedical Eng., Vol.38, No.3, 99. [6] H. Ishikawa,. onezawa, K. Itoh and M. Hashimoto, Extraction of Character Information from Acceleration of Handwriting Pen, Trans. of