Real-time Video Based Finger Spelling Recognition System Using Low Computational Complexity Artificial Neural Networks

Transcription

1 Real-time Video Based Finger Spelling Recognition System Using Low Computational Complexity Artificial Neural Networks T. A. C. Bragatto G. I. S. Ruas M. V. Lamar Department of Electrical Engineering Federal University of Parana Curitiba, PR, Abstract The automatic sign language translation still is the most complex and challenging task for video recognition and processing. This work presents the Brazilian Sign Language Automatic Translation project and specifically focuses on low complexity Artificial Neural Networks dedicated to real-time video processing. A new approach for reducing the computational complexity of the activation function of the Multi-Layer Perceptron is proposed in this work, allowing complex processing of video signals be done in real-time. The low complexity neural networks are used in two stages of the system. In the color detection and hand posture classification blocks. The obtained results indicate an increase of the frame rate from 8.6 fps to 28.1 fps using a personal microcomputer with a USB webcam, without reduction of the correct recognition rate. Index Terms Gesture Recognition, Computational Complexity Reduction, Low Complexity Neural Networks. I. INTRODUCTION The communication between men and machines is done mainly by the use of instruments as keyboard and mouse, which are not natural ways of human communication. Therefore, researches that allow computers to see and hear are very important for the creation of a system capable of interact in a more natural way with the human beings. Voice recognition is a very popular study field, and many works have been published. Nowadays, with the advance of the computational power of personal computers, gesture recognition is starting to attract the attention of many researches. The development of real-time video processing systems is still a very challenging task. The most complex and grammatically structured set of human gestures are the gestures associated with the sign languages, used by hearing impaired people. The TALIS (from Portuguese Tradução Automática da Linguagem dos Sinais) project aims to develop a Brazilian Sign Language (LIBRAS) automatic translation system. The system has to be able to translate a video of a sentence gesture performed using the Brazilian Sign Language into voice, at real-time using a portable computer. In telecommunications, the coding of the video signal of signers, very common now-a-days in broadcast television, can be largely improved by using a better modeling of the hand movement, using this information to increase the performance of MPEG4 based codecs. In this work, we propose the use of low computational complexity neural networks for real-time video processing, and present the results obtained from their application in a pixel color classifier and in an user dependent hand posture recognition system. Comparisons of speed in real-time and recognition rate with classical classification methods, and published similar works are presented also. The paper is organized as follows: First, the proposed system is briefly described; next the methodology of the implementation of Artificial Neural Networks (ANN) with low computational cost is presented. The obtained results in an experiment of real-time hand location and posture recognition are presented and analyzed, and finally, the conclusions of this work are given. II. HAND GESTURE RECOGNITION SYSTEM The proposed block diagram of the hand gesture recognition system is shown in Fig.1. This system is still being developed and it aims to analyze and classify two hand gestures, that is to locate and tracking the position of the hands in a limited 2-D space jointly with the analysis of the variations of the hands postures over the time[1]. Fig. 1. Block diagram of the proposed hand gesture recognition system In this work, we are interested on the study of an efficient method to locate one hand over a video signal and classify its static posture at real-time. In order to reduce the computational effort of the tracking stage, an inexpensive color glove is used, where each finger has one different color, shown in Fig. 2. Such approach has been tested and approved by many researches [2][3][4]. The use of a color glove allows a quicker detection and modeling of the fingers postures than systems operating with free hand, which need a complex skin color detection stage[5], besides the modeling of a free hand becomes much more elaborated and computationally complex[5]. Each stage of Fig. 1 is briefly explained next. SBrT 955

2 Fig. 2. The color glove used in this work Fig. 3. Color classifier output A. Capture Stage We are working under Windows R environment, using Microsoft Visual C 6.0 R compiler with OpenCV, an opensource computer vision library, from Intel[6]. The OpenCV uses DirectX s Direct Show for capture, allowing acquisition of video signals at high frame rates from any popular webcam using a USB interface. When creating a new capture session, a callback function is defined in OpenCV. This function is called for each new grabbed frame. Inside this function the whole system is developed. All functions have been written in ANSI C, so it is straightforward to migrate the system either to Linux or MacOS, two platforms also supported by OpenCV. The possibility of using Linux Operating System leads this project to the encounter of the Brazilian government software politics that encourage the use of free software. Figure 2 shows an example of image grabbed by the system. In this work the background subtraction technique is used [7] in order to improve more the speed of the system, reducing the amount of pixels that need be analyzed by the next stages. B. Image Pre-Processing Once a video frame is grabbed, the image pre-processing step is performed in order to locate the hand. The preprocessing is composed by a color classifier based on ANN followed by two morphological filters. 1) Color Classification using ANN: The first step is to transform each pixel of the acquired image from the RGB color space to a set of 7 previous defined colors sub-space - one color for each finger, one for the palm of the hand, and one color representing the pixels that do not belong to the glove. A 3 layer Multi-Layer Perceptron (MLP) ANN has been designed for classifying each pixel in one of the 6 possible classes. 2) Morphological Filters: After color classification stage, an image matrix where each element is an integer number ranging from 0 to 6 is obtained. This matrix corresponds to a very noisy image with large amount of small regions that do not belong to the hand. This noise is mainly created by the own user s body and wears, non-uniform background, illumination and digitalization problems, Fig 3. To improve the quality of the image, two morphological filters are used. First an erosion filter is applied, which erases the isolated pixels or the ones in small connected regions. Next, a dilation filter is applied, acting in the opposite way. This improves the definition of the borders and close small gaps inside colored areas. The result of these filtering is shown in Fig. 4. Despite the use of both morphological filters, a great amount of no interest regions is left in the image. To seek the hand, all remaining connected regions are checked by their area and centroid. Fig. 4. Image obtained from the morphological filtering C. The Hand Modelling With the centroids and areas, the remaining regions are analyzed by the hand model proposed in [8]. The final selected regions are show in Fig 5. From each region a 4-D characteristic vector is extracted using the methodology proposed by Lamar et al. [8]. Using only the fingers information a time dependent 20-D vector is generated which represents the hand posture in a time moment. So, a hand motion is characterized by a temporal sequence of the 22-D vectors, where two components are the centroid of the hand palm, to locate the position of the hand in the image, and remaining 20 dimensions are used to model its posture. Fig. 5. Final selected regions supposed to belong to the color glove D. Hand Posture Classification Using ANN Given that this work aims to analyze only static hand postures, the information of the hand location is not used SBrT 956

3 and the system analyzes continually each captured frame. A MLP ANN was trained from the Brazilian Sign Language hand alphabet, shown on Fig. 6. The MLP ANN architecture has 20 input neuron and 26 output neurons. The number of hidden neurons and hidden layers was experimentally defined and is described on section IV. the sigmoidal activation function in real-time neural network systems. The sigmoidal function requires the computation of one division, one sum and one exponential. To reduce the computational complexity a piecewise linear approximated activation function is used. For a three lines approximation, the sigmoidal function is approximated by 0 x< 2.2, Φ(x) = 1 x>2.2, (2) x x 2.2. resulting a function composed by one multiplication, one sum and two comparisons. Figure 7 shows the sigmoidal and the 3 lines piecewise linear activation functions. Fig. 6. Brazilian Sign Language hand alphabet The most computationally expensive stage of the proposed system is the image pre-processing. Every pixel must be analyzed, classified and the connected regions also must be processed in order to generate a characteristic vector of each hand posture in the video sequence. The output classifier is also an important point, because the hand posture will be classified for every input frame. III. LOW COMPUTATIONAL COMPLEXITY ARTIFICIAL NEURAL NETWORK In order to allow the implementation of highly computationally demanding systems, such as video processing systems, based on neural networks in real-time, a study of the computational complexity of the implementation of such neural systems has been done by many researchers[9][10]. Most of them aim to design computationally efficient network structures, considering the number of neurons units and layers, as well the type of activation function. For programming implementation of systems using MLP ANN, the most computationally expensive is the activation function, which involves the evaluation of a sigmoidal non-linear function, defined by 1 Φ(x) = 1+e x (1) where Φ(x) is the sigmoidal activation function and x is the pondered sum of the neuron inputs. To reduce this complexity Bragatto et al. [11] proposed the use a piecewise linear approximation for the implementation of Fig. 7. Activation Functions A Computational Complexity Index (ICC) is defined to compare the speed gain of using the approximation. It is a index based on the assignment of relative weights for every mathematical operation, computer action and command, for analyzing a section of program code[11]. The ICC is used in this work to compare the obtained results. IV. RESULTS The results presented in this section were obtained using a Pentium IV 2.4 GHz, 512 MBytes RAM microcomputer and a simple webcam that provides 352x288 pixel images, 24 bits/pixel at a maximum frame rate of 30fps. The experiment environment is a typical computer research laboratory, with complex background and no rigorous luminosity control, as shown in Fig. 2. A. Color Classification A 3 layer MLP ANN has been designed for classifying each pixel in one of the 6 possible classes. The ANN has 3 input neurons, one for each RGB component normalized between -1 and 1. The outputs consist of 6 neuron units, which indicate the classified color of the input pixel. The training set has been built with 50 examples of pixels of each color, sampled from different angles and luminosity SBrT 957

4 conditions. The test set is composed by 30 pixels for each color, sampled from other acquired images. To minimize the computational complexity, ANNs with 5, 4, 3 and 2 neurons on the hidden layer have been tested. The results obtained using the networks with 5 and 4 neurons were equivalent to the one with 3 hidden neurons. The network with only 2 neurons did not achieve satisfactory results. Based on this analysis, the network with 3 neurons on the hidden layer was chosen. The network structure is shown by Fig. 8, where the connections with bias are not shown. Fig. 9. ANN error in the test set Fig. 8. The proposed ANN structure for color classification TABLE I COLOR CLASSIFICATION RESULTS Method ICC Correct Recognition Rate TM Euclidean % TM City Block % K-NN Euclidean 2, % ANN Sigmoidal % ANN Piecewise % K-NN City Block 1, % Each output neuron corresponds to one color belonging to the glove. The winner-takes-all approach validates the color of the largest output neuron level. As it does not have an output neuron indicating the colors that do not belong to the glove, a threshold is defined in agreement with the confidence of the ANN in the classification. If the winner neuron output level is less than 0.65, value empirically determined, the pixel is considered as not belonging to the glove, so it has the black color attributed. Once the weights of the ANN were trained using the classical Error Backpropagation algorithm, it was implemented in a classical way, using sigmoidal activation function, and also using the proposed approach of using a piecewise linear activation function. The ANNs had their performances compared with classical classifiers based on Template Matching (TM)[12] and K- Nearest Neighbor (K-NN). The metrics tested are the Euclidean and City-Block distances. The same training set used to train the ANNs were used to determine the TM templates. One template for each color is assigned as the average of the samples points of that color. In the K-NN classifier, all training set is used as templates and the best result was obtained using K =1. Figure 9 presents the error on the test set - formed by 180 samples, 30 samples for each color of the glove. This error is calculated by the difference of the correct and the obtained outputs of the ANN. Once the confidence threshold defined is 0.65, the outputs larger than 0.35(dashed line) are considered errors. Table I shows the ICC and correct recognition rate obtained in the color classification stage using K-Nearest Neighbor (K- NN), Template Matching (TM), classic ANN (Sigmoidal) and low computational complexity ANN (Piecewise), in a test set. As expected, the fastest classifier is the Template Matching using City Block distance, however its correct color identi- fication rate is very poor. On the other hand, the best color classifier is obtained applying K-NN with City Block distance metric, achieving 93.8% of correct classification. However, it uses all training set as template vectors increasing considerably the computational effort, achieving an ICC of 1,824, near to 50 times slower than TM City Block. ANN classifiers give better classification rate, comparable with K-NN, having only a very low computational complexity. The use of 3 lines approach piecewise linear activation function reduces the ICC from 364 to 81, allowing the implementation of systems having color classification stage almost 4,5 times faster and without any reduction in the recognition rate, 93.3%. Figure 10 shows the importance of a efficient color classification scheme for the next stages and hand location. We can see that the Template Matching produces a very poor hand modeling, losing important information, making more difficult the posterior hand posture classification. On the other hand, ANN achieves very good visual results for both detection and hand modeling. In this work, the system analyzes every acquired frame, so a reduction of approximately 80% in the computational complexity of the ANN color classifier, allows an increment of the frame rate from 8.7 fps to 27.1 fps. It is because each pixel of the image is processed by this ANN causing a massive computational saving. B. Hand Posture Classification Table II presents a comparison of the use of ANN and classic classifiers in the hand posture recognition stage. In these experiments two networks with different architectures gave equivalent results. ANN1 is a MLP ANN with 1 hidden layer and 100 hidden neuron units. ANN2 is a MLP ANN with 2 hidden layers and 30 neurons on each hidden layer. SBrT 958

5 Fig. 10. (a) (b) (c) (d) (e) (f) Color. (a)(d) Original (b)(e) TM-Euclidean (c)(f) ANN - Sigmoidal TABLE II HAND POSTURE CLASSIFICATION RESULTS Method ICC Correct Recognition Rate TM Euclidian 1, % TM City Block 1, % K-NN Euclidian 31, % K-NN City Block 21, % ANN1 Sigmoidal 13, % ANN1 Piecewise 9, % ANN2 Sigmoidal 7, % ANN2 Piecewise 4, % In this classification stage, ANN overcame the quality of all classic methods. Due to the noisy input data, even K- NN, which is well known, is not an appropriate classification method for noisy signals. ANN are able to model better the problem due it inherent noise robustness. TM is the lightest classification method, however its recognition performance is not good. Comparing ANN1 and ANN2, we can conclude that the use of 2 hidden layers is able to solve a problem with less computational effort, because has a reduced number of neurons and weights to train and more inter connectivity between neurons, when compared with a large 1 hidden layer ANN. Since this stage in executed only once per frame, the speed gain of using a low computational complexity ANN is not so significant, improving the frame rate from 27.1 to 28.1 fps. Table III shows a comparison with related published work using color gloves, different approaches and backgrounds homogenous and complex as well as their recognition rate. TABLE III COMPARISON WITH RELATED WORKS Approach Method Background Recognition Rate Iwai[13] Decision Tree Homogeneous 92.3% Lamar[8] Neural Network Complex 94.5% Proposed Method Neural Network Complex 99.2% Both first and second authors are using finger spelling from American Sign Language(ASL) which has also 26 different hand postures. So the proposed system outperform both approaches. V. CONCLUSIONS This paper presents the TALIS project, which aims to develop a Brazilian Sign Language Automatic Translation System. This work specifically proposes the use of a piecewise linear activation function on two neural networks, for color classification and hand posture recognition, aiming to produce video processing in real-time. Comparisons using different classification methods and development environments are also shown, demonstrating how the same system can be implemented, obtaining different speeds of processing. The use of low complexity neural networks allows a considerable decrement of the computational complexity of the whole system, achieving 28.1 fps for a continuous hand posture classification from Brazilian Sign Language hand alphabet set. New technologies of classification are being tested, such as Support Vector Machines and a second prototype is being developed, using not only static hand postures. ACKNOWLEDGMENTS The authors would like to give special thanks to CNPq The Brazilian National Council for Scientific and Technological Development for the Scholarship; to IEP Parana s Engineering Institute for supporting author s registration. REFERENCES [1] M.V.Lamar, Md.S.Bhuiyan, and A. Iwata, Hand gesture recognition using t-combnet: A new neural network model, IEICE Trans. on Information and Systems, vol. E83-D, no. 11, pp , [2] K. Abe, H. Saito, and S. Ozawa, 3-d drawing system via hand motion recognition from two cameras, Systems, Man, and Cybernetics, 2000 IEEE International Conference on, vol. 2, no , [3] M. Bray, H. Sidenbladh, and J.-O. Eklundh, Recognition of gestures in the context of speech, Pattern Recognition, Proceedings. 16th International Conference on, vol. 1, no , [4] T. Starner and A. Pentland, Visual recognition of american sign language using hidden markov models, International Workshop on Automatic Face and Gesture Recognition, [5] S.G.Wysoski, M. Lamar, S. Kuroyanagi, and A. Iwata, A rotation invariant approach on static-gesture recognition using boundary histograms and neural networks, 9th International Conference on Neural Information Processing, no , [6] G. Sudra, Seminar medizinische simulationssysteme intel R opencv, [7] Y. Ivanov, A. Bobick, and J. Liu, Fast lighting independent background subtraction, International Journal of Computer Vision, vol. 37, no. 2, pp , [8] M. V. Lamar, M. S. Bhuiyan, and A. Iwata, Hand gesture recognition using morphological principal component analysis and an improved CombNET-II, Proc. of IEEE International Conference on System, Man, and Cybernetics, vol. IV, pp , [9] J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the theory of neural computation. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., [10] J. A. Anderson, A simple neural network generating an interactive memory, pp , [11] T. A. C. Bragatto, J. Y. Sugawara, V. A. P. Benso, and M. V. Lamar, Reconhecimento de gestos em tempo-real utilizando uma rede neural artificial de baixa complexidade computacional para detecção de cores, Brazilian Symposium on Computer Graphics and Image Processing, [12] K. Fukunaga, Introduction to Statistical Pattern Recognition. Academic Press, [13] Y. Iwai, K. Watanabe, Y. Yagi, and M. Yachida, Gesture recognition using colored gloves, IEEE International Conference on Pattern Recognition, vol. A, pp , SBrT 959