Amit Choudhary* Deptt. of Comp. Sc. Maharaja Surajmal Institute, New Delhi, India. Rahul Rishi

Transcription

1 Amt Choudhary et al/(ijcse) Internatonal Journal on Computer Scence and Engneerng, Optmal Feed Forward MLPArchtecture for Off-Lne Cursve Numeral Recognton Amt Choudhary* Deptt. of Comp. Sc. Maharaja Surajmal Insttute, New Delh, Inda. Rahul Rsh Deptt. of Comp. Sc.and Engg. TITS, Bhwan, Haryana, Inda. Savta Ahlawat Deptt. of Comp. Sc.and Engg. Maharaja Surajmal Insttute of Technology, New Delh, Inda. Vaypal Sngh Dhaa Deptt. of Comp. Sc. IMS, Noda, UP, Inda. Abstract The purpose of ths wor s to analyze the performance of bac-propagaton feed-forward algorthm usng varous dfferent actvaton functons for the neurons of hdden and output layer and varyng the number of neurons n the hdden layer. For sample creaton, 250 numerals were gathered form 35 people of dfferent ages ncludng male and female. After bnarzaton, these numerals were clubbed together to form tranng patterns for the neural networ. Networ was traned to learn ts behavor by adjustng the connecton strengths at every teraton. The conjugate gradent descent of each presented tranng pattern was calculated to dentfy the mnma on the error surface for each tranng pattern. Experments were performed by selectng dfferent combnatons of two actvaton functons out of the three actvaton functons logsg, tansg and pureln for the neurons of the hdden and output layers and the results revealed that as the number of neurons n the hdden layer s ncreased, the networ gets traned n small number of epochs and the percentage recognton accuracy of the neural networ was observed to ncrease up to certan level and then t starts decreasng when number of hdden neurons exceeds a certan level. Keywords- Numeral Recognton, MLP, Hdden Layers, Bacpropagaton, Conjugate Gradent Descent, Actvaton Functons. I. INTRODUCTION The process of offlne numerals recognton nvolves the automatc converson of numeral s gray scale/ bnary mages obtaned from paper documents, photographs etc. nto dgt codes that are nterpreted by the computer and other textprocessng applcatons. The OCR technology has mportance n varous felds le Banng System, Postal Department etc to recognze courtesy amount on ban checs [2] and to recognze postal pn codes wrtten on post cards / letters. The automated processng of handwrtten materal optmzes the processng speed as compared to human processng and shows hgh recognton accuracy and acheves hgh benefts n monetary terms. Recognton of handwrtten numerals s a very complex problem. The dgts could be wrtten n dfferent dmenson, thcness or slant [3]. These wll gve unlmted varatons. The capablty of neural networ to generalze and ts robust nature would be very benefcal n recognzng handwrtten dgts. The artfcal neural networ structure nvolved n our experment of handwrtten numeral recognton s a mult layered perceptron (MLP) wth one hdden layer whch s a supervsed learnng networ n nature. A MLP s a feedforward neural networ that uses error bac-propagaton algorthm for ts tranng [5]. It s a modfcaton of the standard lnear perceptron n that t uses one or more hdden layers of neurons wth non-lnear actvaton functons and s more powerful than the perceptron n that t can classfy data that s not lnearly separable or separable by a hyper plane. The experments conducted n ths paper have shown the effect of the number of neurons n the hdden layer and the transfer functon used n the hdden layer and output layer on the learnng and recognton accuracy of the neural networ. The crteron to judge the performance of the networ wll be the number of tranng epochs, the MSE (Mean Square Error) value and the percentage recognton accuracy. All other expermental condtons such Learnng Rate, Momentum Constant ( ), Maxmum Epochs Allowed, Acceptable Error Level and Termnaton Condton were ept same for all the experments. The rest of the paper s organzed as follows: Secton II deals wth the overall system desgn and the varous steps nvolved n the OCR System [4]. Neural Networ Archtecture s presented n detal n secton III. Secton IV provdes varous expermental condtons for all the experments conducted under ths wor. Functonng of the proposed system s explaned n Secton V. Dscusson of Results and * Correspondng Author

2 Amt Choudhary et al/(ijcse) Internatonal Journal on Computer Scence and Engneerng, nterpretatons are descrbed n secton VI. Secton VII presents the more threshold value. Ths parameter s decded as shown the concluson and also gves the future path for contnual wor n Table : n ths feld. II. OVERALL SYSTEM DESIGN A typcal handwrtten numeral recognton system s characterzed by a number of steps, whch nclude () Dgtzaton / Image Acquston, (2) Preprocessng, (3) Image Bnarzaton (4) Networ Tranng (5) Classfcaton / Recognton and (6) Testng. Fg. llustrates varous steps nvolved n a handwrtten dgt recognton system. Fgure. Typcal Off-Lne Handwrtten Numeral Recognton System A. Preprocessng Preprocessng ams at elmnatng the varablty that s nherent n cursve and hand-prnted numerals. Scalng, Nose Elmnaton, Slant Estmaton and Correcton and Contour Smoothng are some preprocessng technques that have been employed n an attempt to ncrease the performance of the recognton process. Scalng sometmes may be necessary to produce numeral mages of equal sze. Nose (small dots or blobs) may be ntroduced easly nto an mage durng mage acquston; therefore, these small foreground components are usually removed. S. Srhar, V. Govndaraju and R. Srhar also analyzed the sze and shape of connected components n a numeral mage and compared them to a threshold to remove the nose []. Slant estmaton and correcton s an ntegral part of any numeral mage preprocessng. The slope can be estmated through analyss of the slanted vertcal projectons at varous angles. Contour Smoothng [7] s a technque to remove contour nose that s ntroduced n the form of bumps and holes due to the process of slant correcton. TABLE I. INTENSITY / THRESHOLD COMPARISON TABLE Gray-Scale Intensty of the Threshold Text (I) The frst column of Table- represents the ntensty of handwrtten dgt present n the document. Ths ntensty s a gray-scale value when the document s saved after scannng n gray scale mage format and the second column of ths table represents the correspondng value of threshold level for bnarzaton process [6]. The threshold parameter along wth the grayscale mage s made an nput to the bnarzaton program desgned n MATLAB. The output s a bnary mage shown n Fg. 2(a). C. Reshape and Reszng of Numerals for Sample Creaton Each dgt s frst converted nto a bnary form and then reszed to 0x0 matrx M usng nearest neghborhood nterpolaton and reshaped to a 00x logcal vector so that t can be presented as nput to the networ for learnng. The matrx s reshaped [8] usng followng algorthm: Algorthm: ReshapeCharacter(M,00,). Fnd the transpose of Character s bnary matrx M. 2. Read every element vertcally.e. column wse. 3. Save t n the new matrx accordng to ts dmensons n row frst manner. 4. Repeat step 2-3 untl every element s read. B. Bnarzaton of Numeral Images All hand prnted numerals are scanned nto gray scale mages. Each numeral mage s traced vertcally after convertng the gray scale mage nto bnary matrx. Ths bnarzaton s done usng logcal operaton on gray ntensty level as followes: If (I >= Threshold) Else I = 0 I = where 0 < Threshold <= Ths threshold parameter s based on the gray-scale ntensty of the text n the document. More ntensty leads to Fgure 2. (a) Bnary Representaton of Dgt 0 ; (b) Bnary Matrx representaton and (c) Reshaped sample of Dgt 0. The output of ths algorthm s a bnary matrx of sze 00x whch s made as an nput to the neural networ for learnng and testng. Bnary matrx representaton of dgt 0 can be defned as n Fg 2(b). The Reszed dgts were clubbed together n a matrx of sze (00, 0) to form a SAMPLE as shown n Fg. 3. 2

3 Amt Choudhary et al/(ijcse) Internatonal Journal on Computer Scence and Engneerng, So the fnal samples for networ tranng/testng were made n as follows: y f ( Z W ) (3.) Where f s the output functon, Z s the output of hdden layer and W s the weght between hdden and output layer. And, also for hdden layer s processng unt output ; Z n f ( V X ) (3.2) j j j Fgure 3. Input tranng / testng sample preparaton For sample creaton, 250 numerals were gathered form 35 people of dfferent ages ncludng male and female. After the preprocessng s over, 5 samples were consdered for tranng such that each sample was consstng of 0 dgts (0-9) and 3 samples were consdered for testng the recognton accuracy of the networ. III. NEURAL NETWORK S ARCHITECTURE USED IN THE RECOGNITION PROCESS The archtecture selected for the neural networ s a Multlayer Perceptron (MLP) wth one hdden layer. The networ has 00 nput neurons that are equvalent to the nput numeral s sze as we have reszed every numeral nto a bnary matrx of sze 0 X 0. The 0 output neurons correspond to 0 dgts. The number of hdden neurons s drectly proportonal to the system resources. The bgger the number more the resources are requred. The processng nodes of nput layer used the lner actvaton functon and the nodes of hdden and output layers used the non-lner dfferentable actvaton functon as shown n Fg 4. Where X j s the output of nput layer and weght between nput and hdden layer. V s the The MSE between the desred and actual output of the networ s gven by: as; E [ t y ] (3.3) Where t s desred output The error mnmzaton can be shown as; E ' [ t y ] f ( y z W ) (3.4) Weghts modfcatons on the hdden layer can be defned E y z (3.5) ' V *( )[ t y ] f ( y V ) v ' [Let, = t y ] f ( ) ] and we have [ y ' V * W f ( z ) (3.6) Thus the weght updates for output unt can be represented as; W ( t ) W ( t) W ( t) W ( t ) (3.7) Where W (t) s the state of weght matrx at teraton t j W ( t ) s the state of weght matrx at next teraton Fgure 4. Archtecture of the Neural Networ used n the Recognton System. The output of the networ can be determned as: ( t ) W s the state of weght matrx at prevous teraton. and (t) s current change/ modfcaton n weght matrx W 3

4 s standard classcal momentum varable to accelerate learnng process and depends on the learnng rate of the networ. The neural networ was exposed to 5 dfferent samples as acheved n secton III, each sample beng presented 00 tmes. Actual output of the networ was obtaned by COMPET functon. Ths s a compettve transfer functon whch wors as follows: Functon Compet (net N) Start: [S, Q] = Sze (N) (N beng networ smulated vector) [Maxn, ndn] = max (N, [ ], ) Result Matrx A = Sparse (ndn, : Q, Ones (, Q), S, Q) End Functon. Ths COMPET functon puts at the output neuron n whch the maxmum trust s shown and rest neuron s result nto 0 status. The output matrx s a bnary matrx of sze (0, 0). The output s of the sze (0 0) because each dgt has 0 output vector. Frst 0 column stores the frst dgt's recognton output, the followng column wll be for next dgt and so on for 0 dgts. For each dgt the 0 vector wll contan value at only one place. For example dgt 0 f correctly recognzed, wll result n [, 0, 0, 0 all 0]. The dfference between the desred and actual output s calculated for each cycle and the weghts are adjusted by the equaton (3.7). Ths process contnues tll the networ converges to the allowable error or tll the maxmum number of tranng epochs s reached. IV. EXPERIMENTAL CONDITIONS The varous parameters and ther respectve values used n the learnng process of all the experments wth varous actvaton functons of the neurons n hdden and output layers and havng dfferent number of neurons n the hdden layer are shown n Table 2. TABLE II. EXPERIMENTAL CONDITIONS OF THE NEURAL NETWORK Parameters Value Input Layer No. of Input neurons 00 Transfer / Actvaton Functon Lnear Hdden Layer No. of neurons Between 5 and 95 Transfer / Actvaton Functon LogSg or Tansg or Pureln Learnng Rule Momentum Output Layer No. of Output neurons 0 Transfer / Actvaton Functon LogSg or Tansg or Pureln Learnng Rule Momentum Learnng Constant 0.0 Acceptable Error (MSE) 0.00 Momentum Term ( ) 0.90 Amt Choudhary et al/(ijcse) Internatonal Journal on Computer Scence and Engneerng, Maxmum Epochs 000 Termnaton Condtons Based on mnmum Mean Square Error or maxmum number of epochs allowed Intal Weghts and based term values Randomly generated values between 0 and Number of Hdden Layers V. FUNCTIONAL DETAILS The neural networ structure s determned by the number of neurons n the nput, hdden and output layers. The networ performance s also nfluenced by the actvaton functons of the hdden and output layer neurons. The number of neurons n the nput and output layer of the networ depends on the type of the problem. In the current stuaton, the number of neurons n the nput and output layers are fxed at 00 and 0 respectvely. The number of neurons n the hdden layer and the actvaton functons of the neurons n the hdden and output layers are to be decded. It s very dffcult to determne the optmal number of hdden neurons. Too few hdden neurons wll result n underfttng and there wll be hgh tranng error and statstcal error due to the lac of enough adjustable parameters to map the nput-output relatonshp. Too many hdden neurons wll result n overfttng and hgh varance. The networ wll tend to memorse the nput-output relatons and normally fal to generalze. Testng data or unseen data could not be mapped properly. Each neuron n the neural networ has a transformaton functon. To produce an output, the neuron performs the transformaton functon on the weghted sum of ts nputs. Varous transfer functons used n neural networs are compet, hardlm, logsg, posln, pureln, radbas, satln, softmax, tansg and trbas etc. The three transfer functons normally used n MLP are logsg, tansg and pureln. logsg transfer functon s also nown as Logstc Sgmod: log sg( x), x e tansg transfer functon s also nown as hyperbolc tangent: x x e e tan sg( x) x x e e The hyperbolc tangent and logstc sgmod are related by: tan sg( x) 2 e We get: 2x 2 tan sg ( x) 2x e pureln transfer functon s also nown as smple lnear transfer functon and produces the same output as ts nput. pureln ( x) x 4

5 These transfer functons are commonly used as they are easy to use mathematcally and whle saturatng, they are close to lnear near the orgn. There s not any rule that gves the calculaton for the deal parameter settng for a neural networ. In our problem, the structure analyss has to be carred out to fnd the optmum number of neurons n the hdden layer and the best actvaton functons for the neurons n the hdden and output layers. For ths analyss, the actvaton functon of the neurons n the hdden layer was selected as logsg and the actvaton functon of the neurons n the output layer was changed and the process s repeated by selectng dfferent number of neurons n the hdden layer as shown n Table. Smlarly, as shown n Table 2 and Table 3, the actvaton functon of the hdden layer neurons was selected as tansg and pureln respectvely and the actvaton functon s changed for output layer neurons wth varous dfferent numbers of neurons n the hdden layer. Mean Square Error (MSE) s an accepted measure of the performance ndex often used n MLP networs. The lower value of MSE ndcates that the networ s capable of mappng the nput and output accurately. The accepted error level s set to 0.00 and the tranng wll stop when the fnal value of MSE reaches at 0.00 or below ths level. The number of epochs requred to tran a networ also ndcates the networ performance. The adjustable parameters of the networ wll not converge properly f the number of tranng epochs s nsuffcent and the networ wll not be well traned. On the other hand, the networ wll tae unnecessary long tranng tme f there are excessve tranng epochs. The number of tranng epochs should be suffcent enough so as to meet the am of tranng. If there s a saturated or horzontal straght lne at the end of the MSE plot, the further tranng epochs are no longer benefcal and the tranng should be stopped by ntroducng the stoppng crteron such as maxmum number of tranng epochs allowed n addton to the stoppng crteron of maxmum acceptable error as specfed n the tranng algorthm. Ths type of MSE plot s observed when there s a very complcated problem or nsuffcent tranng algorthm or the networ have very lmted resources. Amt Choudhary et al/(ijcse) Internatonal Journal on Computer Scence and Engneerng, networ decreases. Insuffcent number of hdden neurons results n underfttng and the MSE does not fall below the acceptable error level and the tranng s stopped by the stoppng crteron of maxmum allowed number of tranng epochs. Overfttng s observed when the number of hdden neurons exceeds a certan count and the percentage recognton accuracy gradually falls wth the ncrease n the number of hdden neurons. The networ wll tend to memorse the nputoutput relatons and normally fal to generalze. However the tranng of the networ becomes faster as ndcated by the decrease n number of tranng epochs as the number of hdden neurons ncreases. The optmal number of hdden neurons and the best combnaton of actvaton functons for the hdden and output neurons can be decded by consderng the numeral recognton accuracy of the networ. By detaled analyss of Table3, Table 4 and Table 5, t s clear that the good recognton accuracy s observed when the selected combnaton of actvaton functon for hdden and output neurons were logsg-tansg, logsgpurelne, tansg-tansg and tansg-purelne. By tang nto consderaton all the performance measure parameters such as MSE, overfttng and underfttng, tranng tme and the avalable system resources, t s evdent from Table 2 that the neural networ wth 25 neurons n the hdden layer and wth tansg-tansg actvaton functon combnaton for the hdden and output neurons provdes the best sample recognton accuracy of 99.% wth least MSE of and suffcent number of tranng epochs. For ths networ, the profle of MSE plot for the tranng epochs s drawn n Fg 5 VI. RESULTS AND DISCUSSION Three types of transfer functons: logsg, tansg and pureln are used to smulate the neural networ used n our problem of numeral recognton. As seen n Table 3, the actvaton functon for the output layer was changed to logsg, tansg and pureln one by one by fxng the actvaton functon of hdden neurons to logsg. The number of tranng epochs and MSE values of the networ are ndcated correspondng to the number of neurons n the hdden layer. Smlarly n Table 4 and Table 5, the actvaton functon of the hdden neurons was fxed as tansg and pureln respectvely. If we gnore the type of actvaton functon used n hdden layer and output layer, t s clear from Table 3, Table 4 and Table 5, that, as the number of neurons n the hdden layer s ncreased, the number of epochs requred for the tranng of the Fgure 5. The varaton of MSE wth the tranng epochs VII. CONCLUSION AND FUTURE SCOPE The MLP used n the proposed method for the handwrtten numeral recognton employng the bac propagaton algorthm performed exceptonally well wth 25 neurons n the hdden layer and tansg as the actvaton functon for both hdden and output layer neurons. In ths optmal MLP structure, the number of neurons n the nput and output layers were already fxed to 00 and 0 respectvely. 5

6 Thus t can be concluded that f the accuracy of the results s a crtcal factor for an applcaton, then the networ havng the optmal structure should be used but f tranng tme s a crtcal factor then the networ havng a large number of hdden neurons should be used. Tranng speed can be acheved at the cost of system resources because the number of tranng epochs decreases wth the ncrease n the number of hdden neurons. More wor need to be done especally on the test for more complex handwrtten numerals. The proposed wor can be carred out to recognze numeral strngs after proper segmentaton of the dgts nto solated dgt mages. REFERENCES [] S Srhar, V Govndraju, R Srhar. Handwrtten text recognton. Proc 4 th Internatonal Worshop on fronters n handwrtng recognton, Tawan, , 994. [2] Neves, R.F.P., et al A New Technque to Threshold the Courtesy Amount of Brazlan Ban Checs. In Proceedngs of the 5th IWSSIP, (Bratslava, Slova Republc, June 2008), IEEE Press, n press. [3] S. Ouchtat, B. Mould, A. Lachour, Segmentaton and Recognton of Handwrtten Numerc Chans. In Journal of Computer Scence 3 (4) ISSN , , 997. Amt Choudhary et al/(ijcse) Internatonal Journal on Computer Scence and Engneerng, [4] N. Arca and F. Yarman-Vural, "An Overvew of Character Recognton Focused on Offlne Handwrtng", IEEE Transactons on Systems, Man, and Cybernetcs, Part C: Applcatons and Revews, Vol. 3, No.2, pp , 200. [5] B. K. Verma, "Handwrtten Hnd Character Recognton Usng Multlayer Perceptron and Radal Bass Functon Neural Networ", IEEE Internatonal Conference on Neural Networ, vol. 4, pp , 995. [6] R. Iln; R. Kozma; P. J. Werbos. Beyond feedforward models traned by bacpropagaton: A practcal tranng tool for a more effcent unversal approxmator. IEEE Transactons on Neural Networs, 9(6): , June [7] An OCR System for Telugu, Proceedngs of the Sxth Internatonal Conference on Document Analyss and Recognton, p.0, September 0-3, 200 [8] N. P. Banashree, R. Vasanta, OCR for Scrpt Identfcaton of Hnd (Devnagar) Numerals usng Feature Sub Selecton by Means of End- Pont wth Neuro-Memetc Model. In Proceedngs of World Academy of Scence, Engneerng and Technology, vol. 22 ISSN , 78 82, [9] B. Burton, R.G. Harley, G. Dana, and J.R. Rodgerson, "Reducng the Computatonal Demands of Contnually Onlne- Traned Artfcal Neural Networs for System Identfcaton and Control of Fast Processes" IEEE transacton on Industry Applcatons,Vol 34. no.3, May/June 998, pp TABLE III. BEHAVIOR OF MLP WITH LOGSIG FUNCTION IN HIDDEN LAYER Number of Logsg- Logsg Logsg -Tansg Logsg -Purelne Hdden Unts % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 6

7 TABLE IV. Amt Choudhary et al/(ijcse) Internatonal Journal on Computer Scence and Engneerng, BEHAVIOR OF MLP WITH TANSIG FUNCTION IN HIDDEN LAYER Number of Tansg - Logsg Tansg Tansg Tansg -Purelne Hdden Unts % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % TABLE V. BEHAVIOR OF MLP WITH PURELIN FUNCTION IN HIDDEN LAYER Number of Purelne - Logsg Purelne Tansg Purelne Purelne Hdden Unts % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 7