BRNO UNIVERSITY OF TECHNOLOGY

Transcription

1 BRNO UNIVERSITY OF TECHNOLOGY FACULTY OF INFORMATION TECHNOLOGY DEPARTMENT OF INTELLIGENT SYSTEMS ALGORITHMIC AND MATHEMATICAL PRINCIPLES OF AUTOMATIC NUMBER PLATE RECOGNITION SYSTEMS B.SC. THESIS AUTHOR ONDREJ MARTINSKY BRNO 2007

2 Copyrght 2007 Ondrej Martnsky The author s ndebted to the supervsor of ths thess, doc. Ing. Frantšek Zbořl, CSc. for hs great help. THIS WORK IS A PART OF THE RESEARCH PLAN "SECURITY-ORIENTED RESEARCH IN INFORMATION TECHNOLOGY, MSM " AT BRNO UNIVERSITY OF TECHNOLOGY Lcensed under the terms of Creatve Commons Lcense, Attrbuton-NonCommercal-NoDervs 2.5. You are free to copy, dstrbute and transmt ths work under the followng condtons. You must attrbute the work n the manner specfed by the author or lcensor (but not n any way that suggests that they endorse you or your use of the work). You may not use ths work for commercal purposes. For further nformaton, please read the full legal code at

3 Abstract Ths work deals wth problematc from feld of artfcal ntellgence, machne vson and neural networks n constructon of an automatc number plate recognton system (ANPR). Ths problematc ncludes mathematcal prncples and algorthms, whch ensure a process of number plate detecton, processes of proper characters segmentaton, normalzaton and recognton. Work comparatvely deals wth methods achevng nvarance of systems towards mage skew, translatons and varous lght condtons durng the capture. Work also contans an mplementaton of a demonstraton model, whch s able to proceed these functons over a set of snapshots. Key Words: Machne vson, artfcal ntellgence, neural networks, optcal character recognton, ANPR

4 Contents 1 Introducton ANPR systems as a practcal applcaton of artfcal ntellgence Mathematcal aspects of number plate recognton systems Physcal aspects of number plate recognton systems Notatons and mathematcal symbols 3 2 Prncples of number plate area detecton Edge detecton and rank flterng Convoluton matrces Horzontal and vertcal mage projecton Double-phase statstcal mage analyss Vertcal detecton - band clppng Horzontal detecton - plate clppng Heurstc analyss and prorty selecton of number plate canddates Prorty selecton and basc heurstc analyss of bands Deeper analyss Deskewng mechansm Detecton of skew Correcton of skew 18 3 Prncples of plate segmentaton Segmentaton of plate usng a horzontal projecton Etracton of characters from horzontal segments Pece etracton Heurstc analyss of peces 23 4 Feature etracton and normalzaton of characters Normalzaton of brghtness and contrast Hstogram normalzaton Global thresholdng Adaptve thresholdng Normalzaton of dmensons and resamplng Nearest-neghbor downsamplng Weghted-average downsamplng Feature etracton Pel matr Detecton of character edges Skeletonzaton and structural analyss 35 5 Recognton of characters General classfcaton problem Bologcal neuron and ts mathematcal models 43 v

5 5.2.1 McCulloch-Ptts bnary threshold neuron Percepton Feed-forward neural network Adaptaton mechansm of feed-forward neural network Actve phase Partal dervatves and gradent of error functon Adaptaton phase Heurstc analyss of characters 53 6 Syntactcal analyss of a recognzed plate Prncple and algorthms Recognzed character and ts cost Syntactcal patterns Choosng the rght pattern 57 7 Tests and fnal consderatons Choosng the representatve set of snapshots Evaluaton of a plate number correctness Bnary score Weghted score Results 61 Summary 62 Append A: Case study 63 Append B: Demo recognton software user s manual 73 Bblography 76 v

6 Lst of Fgures 1.1.a Illumnated number plate b Snapshot degraded by the sgnfcant moton blur Convoluton matr Varous rank and edge detecton flters Vertcal projecton of an mage nto a y as Double-phase plate clppng Vertcal projecton of the snapshot after convoluton wth a rank vector Band detected by an analyss of a vertcal projecton Horzontal projecton of the band and ts dervatve Wder area of the number plate after deskewng Prncple of a number plate postvty determnaton usng the color hstogram Dfference between the rotated and sheared number plate Illustraton of Hough transformaton Eample of the Hough transformaton Eample of a number plate before and after deskewng Eample of a number plate after applcaton of the adaptve thresholdng Pece etracton algorthm a Segmentaton phase nput b Segmentaton phase output Hstogram normalzaton by the Lagrange nterpolatng polynomal Partally shadowed number plate Chow and Kaneko approach of adaptve thresholdng Prncple of the downsamplng Nearest-neghbor and weghted-average downsamplng The pel matr feature etracton method Regon layouts n character btmap Possble types of 22 edges n character btmap The four-pel and eght-pel neghborhood Skeletonzaton algorthm Types of structural elements n character btmap Dfferent types of junctons n two nstances of the same character ab Combnaton of structural constrants and neural networks c Eample of the 913 upper-case alphabet Rectangular and polar coordnate systems n the character btmap General classfcaton problem Bologcal neuron a Parts of the bologcal neuron b Synaptc connectons between dendrtes and termnal buttons a Summaton and gan functon of the percepton b Sgmod saturaton functon Three layer feed-forward neural network Dependency of an error functon on a number of neurons Fndng a global mnmum n the error landscape Character segments after applcaton of the pece etracton algorthm Dfferent types of car snapshots 60 v

7 Lst of Tables 4.1 Structural constrants of characters Table of segment propertes related to the fgure Recognton rates of the ANPR system 61 v

8 Chapter 1 Introducton 1.1 ANPR systems as a practcal applcaton of artfcal ntellgence Massve ntegraton of nformaton technologes nto all aspects of modern lfe caused demand for processng vehcles as conceptual resources n nformaton systems. Because a standalone nformaton system wthout any data has no sense, there was also a need to transform nformaton about vehcles between the realty and nformaton systems. Ths can be acheved by a human agent, or by specal ntellgent equpment whch s be able to recognze vehcles by ther number plates n a real envronment and reflect t nto conceptual resources. Because of ths, varous recognton technques have been developed and number plate recognton systems are today used n varous traffc and securty applcatons, such as parkng, access and border control, or trackng of stolen cars. In parkng, number plates are used to calculate duraton of the parkng. When a vehcle enters an nput gate, number plate s automatcally recognzed and stored n database. When a vehcle later ets the parkng area through an output gate, number plate s recognzed agan and pared wth the frst-one stored n the database. The dfference n tme s used to calculate the parkng fee. Automatc number plate recognton systems can be used n access control. For eample, ths technology s used n many companes to grant access only to vehcles of authorzed personnel. In some countres, ANPR systems nstalled on country borders automatcally detect and montor border crossngs. Each vehcle can be regstered n a central database and compared to a black lst of stolen vehcles. In traffc control, vehcles can be drected to dfferent lanes for a better congeston control n busy urban communcatons durng the rush hours. 1.2 Mathematcal aspects of number plate recognton systems In most cases, vehcles are dentfed by ther number plates, whch are easly readable for humans, but not for machnes. For machne, a number plate s only a grey pcture defned as a two-dmensonal functon f (, y), where and y are spatal coordnates, and f s a lght ntensty at that pont. Because of ths, t s necessary to desgn robust mathematcal machnery, whch wll be able to etract semantcs from spatal doman of the captured mage. These functons are mplemented n so-called ANPR systems, where the acronym ANPR stands for an Automatc Number Plate Recognton. ANPR system means transformaton of data between the real envronment and nformaton systems. The desgn of ANPR systems s a feld of research n artfcal ntellgence, machne vson, pattern recognton and neural networks. Because of ths, the man goal of ths thess s to study algorthmc and mathematcal prncples of automatc number plate recognton systems. Chapter two deals wth problematc of number plate area detecton. Ths problematc ncludes algorthms, whch are able to detect a rectangular area of the number plate n orgnal mage. Humans defne the number plate n a natural language as a small plastc or metal plate attached to a vehcle for offcal dentfcaton purposes, but machnes do not understand ths defnton. Because of ths, there s a need to fnd an alternatve defnton of the number plate based on descrptors, whch wll be comprehensble for machnes. Ths s a fundamental problem of machne vson and of ths chapter. Chapter three descrbes prncples of the character segmentaton. In most cases, characters are segmented usng the horzontal projecton of a pre-processed number plate, but sometmes 1

9 these prncples can fal, especally f detected number plates are too warped or skewed. Then, more sophstcated segmentaton algorthms must be used. Chapter four deals wth varous methods normalzaton and detecton of characters. At frst, character dmensons and brghtness must be normalzed to ensure nvarance towards a sze and lght condtons. Then, a feature etracton algorthm must be appled on a character to flter rrelevant data. It s necessary to etract features, whch wll be nvarant towards character deformatons, used font style etc. Chapter fve studes pattern classfers and neural networks and deals wth ther usage n recognton of characters. Characters can be classfed and recognzed by the smple nearest neghbor algorthm (1NN) appled to a vector of etracted features, or there s also possblty to use one of the more sophstcated classfcaton methods, such as feed-forward or Hopfeld neural networks. Ths chapter also presents addtonal heurstc analyses, whch are used for elmnaton of non-character elements from the plate. Sometmes, the recognton process may fal and the detected plate can contan errors. Some of these errors can be detected by a syntactcal analyss of the recognzed plate. If we have a regular epresson, or a rule how to evaluate a country-specfc lcense plate, we can reconstruct defectve plates usng ths rule. For eample, a number zero 0 can be automatcally repared to a character O on postons, where numbers are not allowed. Chapter s deals wth ths problematc. 1.3 Physcal aspects of number plate recognton systems Automatc number plate recognton system s a specal set of hardware and software components that proceeds an nput graphcal sgnal lke statc pctures or vdeo sequences, and recognzes lcense plate characters from t. A hardware part of the ANPR system typcally conssts of a camera, mage processor, camera trgger, communcaton and storage unt. The hardware trgger physcally controls a sensor drectly nstalled n a lane. Whenever the sensor detects a vehcle n a proper dstance of camera, t actvates a recognton mechansm. Alternatve to ths soluton s a software detecton of an ncomng vehcle, or contnual processng of the sampled vdeo sgnal. Software detecton, or contnual vdeo processng may consume more system resources, but t does not need addtonal hardware equpment, lke the hardware trgger. Image processor recognzes statc snapshots captured by the camera, and returns a tet representaton of the detected lcense plate. ANPR unts can have own dedcated mage processors (all-n-one soluton), or they can send captured data to a central processng unt for further processng (generc ANPR). The mage processor s runnng on specal recognton software, whch s a key part of whole ANPR system. Because one of the felds of applcaton s a usage on road lanes, t s necessary to use a specal camera wth the etremely short shutter. Otherwse, qualty of captured snapshots wll be degraded by an undesred moton blur effect caused by a movement of the vehcle. For eample, usage of the standard camera wth shutter of 1/100 sec to capture a vehcle wth speed of 80 km/h wll cause a moton skew n amount of 0.22 m. Ths skew means the sgnfcant degradaton of recognton abltes. There s also a need to ensure system nvarance towards the lght condtons. Normal camera should not be used for capturng snapshots n darkness or nght, because t operates n a vsble lght spectrum. Automatc number plate recognton systems are often based on cameras operatng n an nfrared band of the lght spectrum. Usage of the nfrared camera n combnaton wth an nfrared llumnaton s better to acheve ths goal. Under the llumnaton, plates that are made from refleve materal are much more hghlghted than rest of the mage. Ths fact makes detecton of lcense plates much easer. 2

10 Fgure 1.1: (a) Illumnaton makes detecton of refleve mage plates easer. (b) Long camera shutter and a movement of the vehcle can cause an undesred moton blur effect. 1.4 Notatons and mathematcal symbols Logc symbols p q Eclusve logcal dsjuncton ( p or q ) p q Logcal conjuncton ( p and q ) p q Logcal dsjuncton ( p or q ) p Ecluson ( not p ) Mathematcal defnton of mage f (, y ) and y are spatal coordnates of an mage, and f s an ntensty of lght at that pont. Ths functon s always dscrete on dgtal computers. N 0 y N 0, where N 0 denotes the set of natural numbers ncludng zero. f ( p ) The ntensty of lght at pont p. f ( p) = f (, y), where p = [, y] Pel neghborhoods p1n ɺɺ 4 p2 Pel p 1 s n a four-pel neghborhood of pel p 2 (and vce versa) p1n ɺɺ 8 p2 Pel p 1 s n an eght-pel neghborhood of pel p 2 (and vce versa) Convolutons a( ) b( ) Dscrete convoluton of sgnals a( ) and b( ) ɶ b Dscrete perodcal convoluton of sgnals a( ) and b( ) a( ) ( ) 3

11 Vectors and sets [, y] m The element n th column and y th row of matr m. ma A mn A mean A medan A A The mamum value contaned n the set A. The scope of elements can be specfed by addtonal condtons The mnmum value contaned n the set A The mean value of the elements contaned n the set A The medan value of the elements contaned n the set A The cardnalty of the set A. (Number of elements contaned n the set) Vectors or any other ordered sequences of numbers are prnted bold. The elements of vectors are denoted as, where s a sequence number (startng wth zero), such as 0 n 1, where n = s a cardnalty of the vector (number of elements) The element a of the vector. For eample, the vector can contan [ a] ( ) elements a, b, c, d, such as = ( a, b, c, d ) If there s more than one vector denoted as, they are dstngushed by ther ndees. The upper nde ( ) does not mean the th element of vector. Intervals a < < b les n the nterval between a and b. Ths notaton s used when s the spatal coordnate n mage (dscrete as well as contnuous) a b Quantfcators Ths notaton has the same meanng as the above one, but t s used when s a dscrete sequence number. There ests at least one! There ests eactly one n There ests eactly n There does not est For every Roundng Number rounded down to the nearest nteger Number rounded up to the nearest nteger 4

12 Chapter 2 Prncples of number plate area detecton The frst step n a process of automatc number plate recognton s a detecton of a number plate area. Ths problematc ncludes algorthms that are able to detect a rectangular area of the number plate n an orgnal mage. Humans defne a number plate n a natural language as a small plastc or metal plate attached to a vehcle for offcal dentfcaton purposes, but machnes do not understand ths defnton as well as they do not understand what vehcle, road, or whatever else s. Because of ths, there s a need to fnd an alternatve defnton of a number plate based on descrptors that wll be comprehensble for machnes. Let us defne the number plate as a rectangular area wth ncreased occurrence of horzontal and vertcal edges. The hgh densty of horzontal and vertcal edges on a small area s n many cases caused by contrast characters of a number plate, but not n every case. Ths process can sometmes detect a wrong area that does not correspond to a number plate. Because of ths, we often detect several canddates for the plate by ths algorthm, and then we choose the best one by a further heurstc analyss. f, y, where and y are spatal Let an nput snapshot be defned by a functon ( ) coordnates, and f s an ntensty of lght at that pont. Ths functon s always dscrete on dgtal computers, such as N 0 y N 0, where N 0 denotes the set of natural numbers ncludng zero. We defne operatons such as edge detecton or rank flterng as mathematcal transformatons of functon f. The detecton of a number plate area conssts of a seres of convolve operatons. Modfed snapshot s then projected nto aes and y. These projectons are used to determne an area of a number plate. 2.1 Edge detecton and rank flterng We can use a perodcal convoluton of the functon f wth specfc types of matrces m to detect varous types of edges n an mage: w 1 h 1 (, ) = (, ) ɶ m[, ] = (, ) m mod w ( ),mod h ( ) f y f y y f y y j = 0 j= 0 where w and h are dmensons of the mage represented by the functon f Note: The epresson [, y] m represents the element n th column and y th row of matr m Convoluton matrces Each mage operaton (or flter) s defned by a convoluton matr. The convoluton matr defnes how the specfc pel s affected by neghborng pels n the process of convoluton. 5

13 Indvdual cells n the matr represent the neghbors related to the pel stuated n the centre of the matr. The pel represented by the cell y n the destnaton mage (fg. 2.1) s affected by the pels 0 8 accordng to the formula: y = 0 m0 + 1 m1 + 2 m2 + 3 m3 + 4 m4 + 5 m5 + 6 m6 + 7 m7 + 8 m8 Fgure 2.1: The pel s affected by ts neghbors accordng to the convoluton matr. Horzontal and vertcal edge detecton To detect horzontal and vertcal edges, we convolve source mage wth matrces m he and m ve. The convoluton matrces are usually much smaller than the actual mage. Also, we can use bgger matrces to detect rougher edges m he = ; m ve = Sobel edge detector The Sobel edge detector uses a par of 33 convoluton matrces. The frst s dedcated for evaluaton of vertcal edges, and the second for evaluaton of horzontal edges G = ; G y = The magntude of the affected pel s then calculated usng the formula pras, t s faster to calculate only an appromate magntude as G = G + G y. 2 2 y G = G + G. In Horzontal and vertcal rank flterng Horzontally and vertcally orented rank flters are often used to detect clusters of hgh densty of brght edges n the area of the number plate. The wdth of the horzontally orented rank flter matr s much larger than the heght of the matr ( w h ), and vce versa for the vertcal rank flter ( w h ). To preserve the global ntensty of an mage, t s necessary to each pel be replaced wth an average pel ntensty n the area covered by the rank flter matr. In general, the convoluton matr should meet the followng condton: 6

14 w 1 h 1 m = 0 j= 0 hr [ j], = 1.0 where w and h are dmensons of the matr. The followng pctures show the results of applcaton of the rank and edge detecton flters. Fgure 2.2: (a) Orgnal mage (b) Horzontal rank flter (c) Vertcal rank flter (d) Sobel edge detecton (e) Horzontal edge detecton (f) Vertcal edge detecton 2.2 Horzontal and vertcal mage projecton After the seres of convoluton operatons, we can detect an area of the number plate accordng to a statstcs of the snapshot. There are varous methods of statstcal analyss. One of them s a horzontal and vertcal projecton of an mage nto the aes and y. The vertcal projecton of the mage s a graph, whch represents an overall magntude of the mage accordng to the as y (see fgure 2.3). If we compute the vertcal projecton of the mage after the applcaton of the vertcal edge detecton flter, the magntude of certan pont represents the occurrence of vertcal edges at that pont. Then, the vertcal projecton of so transformed mage can be used for a vertcal localzaton of the number plate. The horzontal projecton represents an overall magntude of the mage mapped to the as. 7

15 Fgure 2.3: Vertcal projecton of mage to a y as Let an nput mage be defned by a dscrete functon f (, y ). Then, a vertcal projecton p y of the functon f at a pont y s a summary of all pel magntudes n the y th row of the nput mage. Smlarly, a horzontal projecton at a pont of that functon s a summary of all magntudes n the th column. We can mathematcally defne the horzontal and vertcal projecton as: h 1 w 1 ( ) = (, ) ; py ( y) = f (, y) p f j j= 0 where w and h are dmensons of the mage. = Double-phase statstcal mage analyss The statstcal mage analyss conssts of two phases. The frst phase covers the detecton of a wder area of the number plate. Ths area s then deskewed, and processed n the second phase of analyss. The output of double-phase analyss s an eact area of the number plate. These two phases are based on the same prncple, but there are dfferences n coeffcents, whch are used to determne boundares of clpped areas. The detecton of the number plate area conssts of a band clppng and a plate clppng. The band clppng s an operaton, whch s used to detect and clp the vertcal area of the number plate (so-called band) by analyss of the vertcal projecton of the snapshot. The plate clppng s a consequent operaton, whch s used to detect and clp the plate from the band (not from the whole snapshot) by a horzontal analyss of such band. Snapshot Assume the snapshot s represented by a functon (, ) The [, y ] represents the upper left corner of the snapshot, and [, ] 0 0 f y, where 0 1 and y0 y y1. y represents the bottom rght corner. If w and h are dmensons of the snapshot, then 0 = 0, y 0 = 0, 1 = w 1 and y =. 1 h

16 Band =, such as: The band b n the snapshot f s an arbtrary rectangle b (, y,, y ) Plate b0 b0 b1 b1 ( = ) ( = ) ( y y < y y ) b0 mn b1 ma mn b0 b1 ma =, such as: Smlarly, the plate p n the band b s an arbtrary rectangle p ( p0, y p0, p 1, y p1) ( b0 p0 p 1 b 1) ( y p0 = yb0 ) ( y p0 = yb0 ) The band can be also defned as a vertcal selecton of the snapshot, and the plate as a horzontal selecton of the band. The fgure 2.4 schematcally demonstrates ths concept: y b0 p0 p 1 y b1 Fgure 2.4: The double-phase plate clppng. Black color represents the frst phase of plate clppng, and red color represents the second one. Bands are represented by dashed lnes, and plates by sold lnes Vertcal detecton band clppng The frst and second phase of band clppng s based on the same prncple. The band clppng s a vertcal selecton of the snapshot accordng to the analyss of a graph of vertcal projecton. If r p y contans h h s the heght of the analyzed mage, the correspondng vertcal projecton ( ) values, such as y 0; h 1. The graph of projecton may be sometmes too ragged for analyss due to a bg statstcal r p y. There are two approaches how to solve ths problem. We can blur dsperson of values ( ) y the source snapshot (costly soluton), or we can decrease the statstcal dsperson of the ragged r projecton p y by convolvng ts projecton wth a rank vector: r ( ) = ( ) m ɶ [ ] p y p y y y y hr where m hr s the rank vector (analogous to the horzontal rank matr n secton 2.1.1). The wdth of the vector m hr s nne n default confguraton. After convoluton wth the rank vector, the vertcal projecton of the snapshot n fgure 2.3 can look lke ths: y 9

17 py 100% ( y) yb0 ybm y b 1 0% y 0 y 1 y Fgure 2.5: The vertcal projecton of the snapshot 2.3 after convoluton wth a rank vector. The fgure contans three detected canddates. Each hghlghted area corresponds to one detected band. The fundamental problem of analyss s to compute peaks n the graph of vertcal projecton. The peaks correspond to the bands wth possble canddates for number plates. The mamum p y correspondng to the ale of band can be computed as: value of ( ) y y0 y y1 { y ( )} y = arg ma p y The y b0 and y b1 are coordnates of band, whch can be detected as: bm { ( ) ( )} { ( ) ( )} y = ma y p y c p y b0 y y y bm y0 y ybm y = mn y p y c p y b1 y y y bm ybm y y1 c y s a constant, whch s used to determne the foot of peak y bm. In pras, the constant s calbrated to c 1 = 0.55 for the frst phase of detecton, and c 2 = 0.42 for the second phase. Fgure 2.6: The band detected by the analyss of vertcal projecton Ths prncple s appled teratvely to detect several possble bands. The y b0 and y b1 coordnates are computed n each step of teratve process. After the detecton, values of projecton p y n nterval yb0, y b1 are zerozed. Ths dea s llustrated by the followng pseudo-code: let L to be a lst of detected canddates for :=0 to number_of_bands_to_be_detected do begn detect y b0 and y b1 by analyss of projecton save y b0 and y b1 to a lst L zeroze nterval yb0, y b1 end p y The lst L of coordnates y b0 and y b1 wll be sorted accordng to value of peak ( y bm ). The band clppng s followed by an operaton, whch detects plates n a band. 10

18 2.3.2 Horzontal detecton plate clppng In contrast wth the band clppng, there s a dfference between the frst and second phase of plate clppng. Frst phase There s a strong analogy n a prncple between the band and plate clppng. The plate clppng s based on a horzontal projecton of band. At frst, the band must be processed by a vertcal detecton flter. If w s a wdth of the band (or a wdth of the analyzed mage), the r p contans w values: correspondng horzontal projecton ( ) Please notce that p ( ) y b1 ( ) (, ) p = f j j= yb 0 s a projecton of the band, not of the whole mage. Ths can be acheved by a summaton n nterval yb0, y b1, whch represents the vertcal boundares of the r p may have a bg statstcal dsperson, we decrease band. Snce the horzontal projecton ( ) t by convolvng wth a rank vector ( p ( ) = p r ( ) ɶ m [ ] vr usually equal to a half of an estmated wdth of the number plate. Then, the mamum value correspondng to the plate can be computed as: 0 y 1 { ( )} = arg ma p bm The b0 and b1 are coordnates of the plate, whch can be then detected as: { ( ) ( )} { ( ) ( )} = ma p c p b0 bm 0 bm = mn p c p b1 bm bm 1 where c s a constant, whch s used to determne the foot of peak calbrated to c = 0.86 for the frst phase of detecton. Second phase ). The wdth of the rank vector s bm. The constant s In the second phase of detecton, the horzontal poston of a number plate s detected n another way. Due to the skew correcton between the frst and second phase of analyss, the wder plate f, y be a correspondng functon of such area must be duplcated nto a new btmap. Let ( ) ( ) btmap. Ths pcture has a new coordnate system, such as [0,0] represents the upper left corner and [ w 1, h 1] the bottom rght, where w and h are dmensons of the area. The wder area of the number plate after deskewng s llustrated n fgure 2.8. In contrast wth the frst phase of detecton, the source plate has not been processed by the vertcal detecton flter. If we assume that plate s whte wth black borders, we can detect that borders as black-to-whte and whte-to-black transtons n the plate. The horzontal projecton p of the mage s llustrated n the fgure 2.7.a. To detect the black-to-whte and whte-toblack transtons, there s a need to compute a dervatve p ( ) of the projecton p ( ) n. Snce the projecton s not contnuous, the dervaton step cannot be an nfntely small number 11

19 ( h lm ). If we derve a dscrete functon, the dervaton step h must be an ntegral number 0 (for eample 4 h = ). Let the dervatve of p ( ) be defned as: ( ) p = ( ) ( ) p p h h Where h = 4. p 100% ( ) 0% p ( ) w 1 0 w 1 Fgure 2.7: (a) The horzontal projecton p ( ) of p ( ) of the plate n fgure 2.8. (b) The dervatve. Arrows denote the BW and WB transtons, whch are used to determne the boundares of the plate. Fgure 2.8: The wder area of the number plate after deskewng. The left and rght boundary of the plate can be determned by an analyss of the projecton p ( ). The left corner p0 s represented by the black-to-whte transton (postve peak n fgure 2.7.b), and rght corner p1 by the whte-to-black transton (negatve peak n fgure 2.7.b): { ( ) ma { ( )}} = mn p c p p0 w d 0 < 0 < w 2 { ( ) mn { ( )}} = ma p c p p1 w d < w 0 < w 2 where c d s a constant used to determne the most left negatve and the most rght postve peak. The left and rght corners must le on the opposte halves of the detected plate accordng to the w w constrants 0 < for p0, and < w for p

20 In ths phase of the recognton process, t s not possble to select a best canddate for a number plate. Ths can be done by a heurstc analyss of characters after the segmentaton. 2.4 Heurstc analyss and prorty selecton of number plate canddates In general, the captured snapshot can contan several number plate canddates. Because of ths, the detecton algorthm always clps several bands, and several plates from each band. There s a predefned value of mamum number of canddates, whch are detected by analyss of projectons. By default, ths value s equals to nne. There are several heurstcs, whch are used to determne the cost of selected canddates accordng to ther propertes. These heurstcs have been chosen ad hoc durng the practcal epermentatons. The recognton logc sorts canddates accordng to ther cost from the most sutable to the least sutable. Then, the most sutable canddate s eamned by a deeper heurstc analyss. The deeper analyss defntely accepts, or rejects the canddate. As there s a need to analyze ndvdual characters, ths type of analyss consumes bg amount of processor tme. The basc concept of analyss can be llustrated by the followng steps: 1. Detect possble number plate canddates. 2. Sort them accordng to ther cost (determned by a basc heurstcs). 3. Cut the frst plate from the lst wth the best cost. 4. Segment and analyze t by a deeper analyss (tme consumng). 5. If the deeper analyss refuses the plate, return to the step Prorty selecton and basc heurstc analyss of bands The basc analyss s used to evaluate the cost of canddates, and to sort them accordng to ths cost. There are several ndependent heurstcs, whch can be used to evaluate the cost α. The heurstcs can be used separately, or they can be combned together to compute an overall cost of canddate by a weghted sum: α = 0.15 α α α α Heurstcs Illustraton Descrpton α 1 = yb0 y The heght of band n pels. Bands b1 wth a lower heght wll be preferred. α 2 = 1 p p ( y ) ( ) y y The ( ) bm y bm y bm p y s a mamum value of peak of vertcal projecton of snapshot, whch corresponds to the processed band. Bands wth a hgher amount of vertcal edges wll be preferred. α = 3 y b1 py ( y) b0 y= yb0 1 y y b 1 Ths heurstcs s smlar to the prevous one, but t consders not only the value of the greatest peak, but a value of area under the graph between ponts y b0 and y b1. These ponts defne a vertcal poston of the evaluated band. 13

21 p0 p 1 α4 = 5 y y b0 b1 The proportons of the one-row number plates are smlar n the most countres. If we assume that wdth/heght rato of the plate s about fve, we can compare the measured rato wth the estmated one to evaluate the cost of the number plate Deeper analyss The deeper analyss determnes the valdty of a canddate for the number plate. Number plate canddates must be segmented nto the ndvdual characters to etract substantal features. The lst of canddates s teratvely processed untl the frst vald number plate s found. The canddate s consdered as a vald number plate, f t meets the requrements for valdty. Assume that plate p s segmented nto several characters p0 pn 1, where n s a number of characters. Let w be a wdth of th character (see fgure 2.9.a). Snce all segmented characters have roughly unform wdth, we can use a standard devaton of these values as a heurstcs: 1 1 n = 0 ( w w) β1 = n where w s an arthmetc average of character wdths n w n = 0 w =. If we assume that the number plate conssts of dark characters on a lght background, we can use a brghtness hstogram to determne f the canddate meets ths condton. Because some country-specfc plates are negatve, we can use the hstogram to deal wth ths type of plates (see fgure 2.9.b). Let ( ) b be a value of a darkest and lghtest pont. Then, ( ) ma H b be a brghtness hstogram, where b s a certan brghtness value. Let b mn and are equal to b. The plate s negatve when the heurstcs β 2 s negatve: bma bmd ( ) H ( b) β2 = H b b= bmd b= bmn bma bmn where b md s a mddle pont n the hstogram, such as bmd =. 2 H b s a count of pels, whose values 14

22 p0 p1 p2 p3 p4 p5 p6 p7 p8 p9 ( ) H b b mn w( p 2 ) bmd bma Pel numbers b Fgure 2.9: (a) The number plate must be segmented nto ndvdual characters for deeper heurstc analyss. (b) Brghtness hstogram of the number plate s used to determne the postvty of the number plate. 2.5 Deskewng mechansm The captured rectangular plate can be rotated and skewed n many ways due to the postonng of vehcle towards the camera. Snce the skew sgnfcantly degrades the recognton abltes, t s mportant to mplement addtonal mechansms, whch are able to detect and correct skewed plates. The fundamental problem of ths mechansm s to determne an angle, under whch the plate s skewed. Then, deskewng of so evaluated plate can be realzed by a trval affne transformaton. It s mportant to understand the dfference between the sheared and rotated rectangular plate. The number plate s an object n three-dmensonal space, whch s projected nto the twodmensonal snapshot durng the capture. The postonng of the object can sometmes cause the skew of angles and proportons. If the vertcal lne of plate v p s not dentcal to the vertcal lne of camera objectve v c, the plate may be sheared. If the vertcal lnes v p and v c are dentcal, but the as a p of plate s not parallel to the as of camera a c, the plate may be rotated. (see fgure 2.10) 15

23 a p a p a p v = v c p ac ap ac vp = vc a c v c = v p a a v = v p c p c v c a c v p a a v v p c p c Fgure 2.10: (a) Number plate captured under the rght angle (b) rotated plate (c) Sheared plate Detecton of skew Hough transform s a specal operaton, whch s used to etract features of a specfc shape wthn a pcture. The classcal Hough transform s used for the detecton of lnes. The Hough transform s wdely used for mscellaneous purposes n the problematc of machne vson, but I have used t to detect the skew of captured plate, and also to compute an angle of skew. It s mportant to know, that Hough transform does not dstngush between the concepts such as rotaton and shear. The Hough transform can be used only to compute an appromate angle of mage n a two-dmensonal doman. The mathematcal representaton of lne n the orthogonal coordnate system s an equaton y = a + b, where a s a slope and b s a y-as secton of so defned lne. Then, the lne s a set of all ponts [, y ], for whch ths equaton s vald. We know that the lne contans an nfnte number of ponts as well as there are an nfnte number of dfferent lnes, whch can cross a certan pont. The relaton between these two assertons s a basc dea of the Hough transform. The equaton y = a + b can be also wrtten as b = a + y, where and y are parameters. Then, the equaton defnes a set of all lnes ( a, b ), whch can cross the pont [, y ]. For each pont n the XY coordnate system, there s a lne n an AB coordnate system (so called Hough space ) y b [, y ] 0 0 m k l m l k b = 0 a + y0 a y n the Fgure 2.11: The XY and AB ( Hough space ) coordnate systems. Each pont [, ] XY coordnate system corresponds to one lne n the Hough space (red color). The are several ponts (marked as k, l, m ) n the Hough space, that correspond to the lnes n the XY coordnate system, whch can cross the pont.[, y ]

24 Let f (, y ) be a contnuous functon. For each pont [ a, b ] n Hough space, there s a lne n the XY coordnate system. We compute a magntude of pont [ a, b ] as a summary of all ponts n the XY space, whch le on the lne a + b. f, y s a dscrete functon, whch represents the snapshot wth defnte Assume that ( ) dmensons ( w h). To compute the Hough transform of the functon lke ths, t s necessary to normalze t nto a unfed coordnate system n the followng way: 2 = 1 ; w 2 y y = 1 h Although the space defned by a unfed coordnate system s always dscrete (floatng pont) on dgtal computers, we wll assume that t s contnuous. Generally, we can defne the Hough h a, b f, y n the unfed coordnate system as: transform ( ) of a contnuous functon ( ) 1 (, ) = (, + ) h a b f a b d 1 π 2 0 θ π 0 2 a y b b Fgure 2.12: (a) Number plate n the unfed XY coordnate system after applcaton of the horzontal edge detecton flter (b) Hough transform of the number plate n the θ B coordnate system (c) Colored Hough transform n the AB coordnate system. We use the Hough transform of certan mage to evaluate ts skew angle. You can see the colored Hough transform on the fgure 2.12.c. The pels wth a relatvely hgh value are marked by a red color. Each such pel corresponds to a long whte lne n the fgure 13.a. If we assume that the angle of such lnes determnes the overall angle, we can fnd the longest lne as: ( a, b ) = arg ma h ( a, b ) m m { } 0 1 a 0 b 1 To compute the angle of such a lne, there s a need to transform t back to the orgnal coordnate system: am 1 bm 1 am, bm = w, h 2 2 [ ] where w and h are dmensons of the evaluated mage. Then, the overall angle θ of mage can be computed as: θ = arctan ( a ) m 17

25 The more sophstcated soluton s to determne the angle from a horzontal projecton of the Hough transform h. Ths approach s much better because t covers all parallel lnes together, not only the longest one: where p ( a ) a ˆ aˆ 1 θ arctan w m = 2 ; aˆ = arg ma p ( a ) m { } a 1 1 a s a horzontal projecton of the Hough space, such as: a 1 = ( ) (, ) p a f a b db Correcton of skew The second step of a deskewng mechansm s a geometrc operaton over an mage f (, ) y. As the skew detecton based on Hough transform does not dstngush between the shear and rotaton, t s mportant to choose the proper deskewng operaton. In pras, plates are sheared n more cases than rotated. To correct the plate sheared by the angle θ, we use the affne transformaton to shear t by the negatve angle θ. For ths transformaton, we defne a transformaton matr A : where S and drecton of the Y-as. S y are shear factors. The ( θ ) 1 S y 0 1 tan 0 A = S 1 0 = S s always zero, because we shear the plate only n a Let P be a vector representng the certan pont, such as = [, y,1] coordnates of that pont. The new coordnates = [, y,1] be computed as: P s = s s s P where and y are P of that pont after the shearng can P A where A s a correspondng transformaton matr. Let the deskewed number plate be defned by a functon computed n the followng way: s T ( ) (, ) = [,,1] A [ 1,0,0 ] T,[,,1] A [ 0,1,0 ] f y f y y f s. The functon f s can be After the substtuton of the transformaton matr A : ( θ ) ( θ ) 1 tan tan 0 0 fs (, y) = f [, y,1] , [, y,1]

26 Fgure 2.13: (a) Orgnal number plate. (b) Number plate after deskewng. 19

27 Chapter 3 Prncples of plate segmentaton The net step after the detecton of the number plate area s a segmentaton of the plate. The segmentaton s one of the most mportant processes n the automatc number plate recognton, because all further steps rely on t. If the segmentaton fals, a character can be mproperly dvded nto two peces, or two characters can be mproperly merged together. We can use a horzontal projecton of a number plate for the segmentaton, or one of the more sophstcated methods, such as segmentaton usng the neural networks. If we assume only one-row plates, the segmentaton s a process of fndng horzontal boundares between characters. Secton 3.2 deals wth ths problematc. The second phase of the segmentaton s an enhancement of segments. The segment of a plate contans besdes the character also undesrable elements such as dots and stretches as well as redundant space on the sdes of character. There s a need to elmnate these elements and etract only the character. Secton 3.3 deals wth these problems. 3.1 Segmentaton of plate usng a horzontal projecton Snce the segmented plate s deskewed, we can segment t by detectng spaces n ts horzontal projecton. We often apply the adaptve thresholdng flter to enhance an area of the plate before segmentaton. The adaptve thresholdng s used to separate dark foreground from lght background wth non-unform llumnaton. You can see the number plate area after the thresholdng n fgure 3.1.a. After the thresholdng, we compute a horzontal projecton ( ) p of the plate f (, ) y. We use ths projecton to determne horzontal boundares between segmented characters. These boundares correspond to peaks n the graph of the horzontal projecton (fgure 3.1.b). y p ( ) v m v a Fgure 3.1: (a) Number plate after applcaton of the adaptve thresholdng (b) Horzontal projecton of plate wth detected peaks. Detected peaks are denoted by dotted vertcal lnes. v b 20

28 The goal of the segmentaton algorthm s to fnd peaks, whch correspond to the spaces between characters. At frst, there s a need to defne several mportant values n a graph of the p : horzontal projecton ( ) m v - The mamum value contaned n the horzontal projecton p ( ) v = ma p ( ), where w s a wdth of the plate n pels. m a b 0 < w { } v - The average value of horzontal projecton p ( ), such as v p ( ) a 1 1 w = w = 0, such as v - Ths value s used as a base for evaluaton of peak heght. The base value s always calculated as vb = 2 va vm. The v a must le on vertcal as between the values v b and v m. The algorthm of segmentaton teratvely fnds the mamum peak n the graph of vertcal projecton. The peak s treated as a space between characters, f t meets some addtonal condtons, such as heght of peak. The algorthm then zerozes the peak and teratvely repeats ths process untl no further space s found. Ths prncple can be llustrated by the followng steps: 1. Determne the nde of the mamum value of horzontal projecton: = arg ma p m 0 < w { ( )} 2. Detect the left and rght foot of the peak as: { ( ) ( )} { ( ) ( )} = ma p c p l m 0 m = mn p c p r < w m m 3. Zeroze the horzontal projecton ( ) 4. If p ( ) < c v, go to step 7. m w m 5. Dvde the plate horzontally n the pont m. 6. Go to step End. p on nterval, Two dfferent constants have been used n the algorthm above. The constant l r c s used to determne foots of peak m. The optmal value of c s 0.7. The constant c w determnes the mnmum heght of the peak related to the mamum value of the projecton ( v m ). If the heght of the peak s below ths mnmum, the peak wll not be consdered as a space between characters. It s mportant to choose a value of constant carefully. An nadequate small value causes that too many peaks wll be treated as spaces, and characters wll be mproperly dvded. A bg value of c w causes that not all regular peaks wll be treated as spaces, and characters wll be mproperly merged together. The optmal value of c w s To ensure a proper behavor of the algorthm, constants c and c w should meet the followng condton: (,, ) P : c v p ( ) p ( ) > l m r w m l r where P s a set of all detected peaks m wth correspondng foots l and r. c w 21

29 3.2 Etracton of characters from horzontal segments The segment of plate contans besdes the character also redundant space and other undesrable elements. We understand under the term segment the part of a number plate determned by a horzontal segmentaton algorthm. Snce the segment has been processed by an adaptve thresholdng flter, t contans only black and whte pels. The neghborng pels are grouped together nto larger peces, and one of them s a character. Our goal s to dvde the segment nto the several peces, and keep only one pece representng the regular character. Ths concept s llustrated n fgure 3.2. Horzontal segment Pece 1 Pece 2 Pece 3 Pece 4 Fgure 3.2: Horzontal segment of the number plate contans several groups (peces) of neghborng pels Pece etracton Let the segment be defned by a dscrete functon f (, ) such as [ 0,0 ] s an upper left corner of the segment, and [ w 1, h 1] where w and h are dmensons of the segment. The value of f (, ) pels, and 0 for the whte space. The pece Ρ s a set of all neghborng pels [, ] The pel [, y ] belongs to the pece Ρ f there s at least one pel [, y ] [, y ] and [, y ] are neghbors: [, y] Ρ [, y ] Ρ :[, y] N ɺɺ [, y ] y n the relatve coordnate system, s a bottom rght corner, y s 1 for the black y, whch represents a contnuous element. 4 from the Ρ, such as The notaton anɺɺ 4b means a bnary relaton a s a neghbor of b n a four-pel neghborhood : Algorthm [ ] ɺɺ [ ], y N, y = 1 y y = 1 4 The goal of the pece etracton algorthm s to fnd and etract peces from a segment of the plate. Ths algorthm s based on a smlar prncple as a commonly known seed-fll algorthm. 22

30 Let pece Ρ be a set of (neghborng) pels [, y ] Let S be a set of all peces Ρ from a processed segment defned by the functon f, y. ( ) {,, 1} Let X be a set of all black pels: X [ y] f ( y) Let A be an aulary set of pels = = Prncple of the algorthm s llustrated by the followng pseudo-code: let set S = 0/ { } let set X = [, y] f (, y) = 1 [ 0,0 ] [, y] < [ w, h] whle set X s not empty do begn let set Ρ = 0/ let set A = 0/ pull one pel from set X and nsert t nto set A whle set A s not empty do begn let [, y ] be a certan pel from A pull pel [, y ] from a set A f f (, y) 1 [, y] A [ 0,0 ] [, y] [ w, h] begn end end add Ρ to set S end = < then pull pel [, ] nsert pels [ 1, y],[ + 1, y],[, y 1],[, 1] y from set A and nsert t nto set Ρ y + nto set A Note 1: The operaton pull one pel from a set s non-determnstc, because a set s an unordered group of elements. In real mplementaton, a set wll be mplemented as an ordered lst, and the operaton pull one pel from a set wll be mplemented as pull the frst pel from a lst Note 2: The mathematcal concluson [ mn, ymn ] < [, y] < [ ma, yma ] means The pel [, y ] les n a rectangle defned by pels [, y ] and [, y ]. More formally: mn mn ma ma [, y]r[, y ] R yry where R s a one of the bnary relatons: <, >,, and = Heurstc analyss of peces The pece s a set of pels n the local coordnate system of the segment. The segment usually contans several peces. One of them represents the character and others represent redundant elements, whch should be elmnated. The goal of the heurstc analyss s to fnd a pece, whch represents character. Let us place the pece Ρ nto an magnary rectangle ( 0, y0, 1, y 1), where [ 0, y 0] s an upper left corner, and [ 1, y 1] s a bottom rght corner of the pece: 23

31 { } { } { } { } = mn [, y] Ρ y = mn y [, y] Ρ 0 0 = ma [, y] Ρ y = ma y [, y] Ρ 1 1 The dmensons and area of the magnary rectangle are defned as w = 0 1, h = y0 y1 and S = w h. Cardnalty of the set Ρ represents the number of black pels n b. The number of whte pels n w can be then computed as n w = S n b = w h Ρ. The overall magntude M of a pece s a rato between the number of black pels n b and the area S of an magnary rectangle M = n / S. b In pras, we use the number of whte pels n w as a heurstcs. Peces wth a hgher value of n w wll be preferred. The pece chosen by the heurstcs s then converted to a monochrome btmap mage. Each such mage corresponds to one horzontal segment. These mages are consdered as an output of the segmentaton phase of the ANPR process (see fgure 3.3) Fgure 3.3: The nput (a) and output (b) eample of the segmentaton phase of the ANPR recognton process. 24

32 Chapter 4 Feature etracton and normalzaton of characters To recognze a character from a btmap representaton, there s a need to etract feature descrptors of such btmap. As an etracton method sgnfcantly affects the qualty of whole OCR process, t s very mportant to etract features, whch wll be nvarant towards the varous lght condtons, used font type and deformatons of characters caused by a skew of the mage. The frst step s a normalzaton of a brghtness and contrast of processed mage segments. The characters contaned n the mage segments must be then reszed to unform dmensons (second step). After that, the feature etracton algorthm etracts approprate descrptors from the normalzed characters (thrd step). Ths chapter deals wth varous methods used n the process of normalzaton. 4.1 Normalzaton of brghtness and contrast The brghtness and contrast characterstcs of segmented characters are varyng due to dfferent lght condtons durng the capture. Because of ths, t s necessary to normalze them. There are many dfferent ways, but ths secton descrbes the three most used: hstogram normalzaton, global and adaptve thresholdng. Through the hstogram normalzaton, the ntenstes of character segments are redstrbuted on the hstogram to obtan the normalzed statstcs. Technques of the global and adaptve thresholdng are used to obtan monochrome representatons of processed character segments. The monochrome (or black & whte) representaton of mage s more approprate for analyss, because t defnes clear boundares of contaned characters Hstogram normalzaton The hstogram normalzaton s a method used to re-dstrbute ntenstes on the hstogram of the character segments. The areas of lower contrast wll gan a hgher contrast wthout affectng the global characterstc of mage. f, y. Let I be a total number Consder a grayscale mage defned by a dscrete functon ( ) of gray levels n the mage (for eample I = 256 ). We use a hstogram to determne the number of occurrences of each gray level, 0 I 1 : { } ( ) [, ] 0 0 (, ) H = y < w y < h f y = The mnmum, mamum and average value contaned n the hstogram s defned as: mn = mn { (, )} ; Hma = ma { f (, y) } ; Havg = f (, y) H f y 0 < w 0 y< h 0 < w 0 y< h 1 w h w 1 h 1 = 0 y= 0 25

33 where the values H mn, H ma and H avg are n the followng relaton: 0 H H H I 1 mn avg ma The goal of the hstogram normalzaton s to obtan an mage wth normalzed statstcal I characterstcs, such as H mn = 0, Hma = I 1, H avg =. To meet ths goal, we construct a 2 g as a Lagrange polynomal wth nterpolaton ponts transformaton functon ( ) [, y ] = [ H,0], [ ] 1 1 mn I 2, y2 = Havg, 2 and [ ] [ ] 3, y3 Hma, I 1 ( ) g = : 3 3 k = y j j= 1 k = 1 j k k j Ths transformaton functon can be eplctly wrtten as: g = y + y + y ( ) After substtuton of concrete ponts, and concrete number of gray levels I = 256 : H H H Havg H H H H H H H H mn ma ( ) g mn = I 1 avg mn avg ma ma mn ma avg g ( ) I 2 0 H mn Havg Hma brghtness before transformaton Fgure 4.1: We use the Lagrange nterpolatng polynomal as a transformaton functon to normalze the brghtness and contrast of characters. The Lagrange nterpolatng polynomal as a transformaton functon s a costly soluton. It s lke harvestng one potato by a tractor. In pras, there s more useful to construct the transformaton usng a smple lnear functon that spreads the nterval Hmn, H ma nto the unfed nterval 0, I 1 : 26

34 H H H mn ( ) = ( I 1) g ma The normalzaton of mage s proceeded by the transformaton functon n the followng way: n mn ( ) (, ) = (, ) f y g f y Global Thresholdng The global thresholdng s an operaton, when a contnuous gray scale of an mage s reduced nto monochrome black & whte colors accordng to the global threshold value. Let 0,1 be a gray scale of such mage. If a value of a certan pel s above the threshold t, the new value of the pel wll be zero. Otherwse, the new value wll be one for pels wth values above the threshold t. Let v be an orgnal value of the pel, such as v 0,1. The new value v s computed as: 0 f v 0, t v = 1 f v t,1 The threshold value t can be obtaned by usng a heurstc approach, based on a vsual nspecton of the hstogram. We use the followng algorthm to determne the value of t automatcally: 1. Select an ntal estmate for threshold t (for eample t = 0.5) 2. The threshold t dvdes the pels nto the two dfferent sets: [, ] (, ) { } S = y f y t. and [, ] (, ) b ) a { } S = y f y < t, 3. Compute the average gray level values µ a and µ b for the pels n sets S a and S b as: 1 1 µ a = f (, y) ; µ b S = f (, y) a [, y] S S a b [, y] Sb 1 4. Compute a new threshold value t = ( µ a + µ b ) 2 5. Repeat steps 2, 3, 4 untl the dfference t n successve teratons s smaller than predefned precson t p Snce the threshold t s global for a whole mage, the global thresholdng can sometmes fal. Fgure 4.2.a shows a partally shadowed number plate. If we compute the threshold t usng the algorthm above, all pels n a shadowed part wll be below ths threshold and all other pels wll be above ths threshold. Ths causes an undesred result llustrated n fgure 4.2.b. 27

35 H ( b) t Pel numbers b A B C Fgure 4.2: (a) The partally shadowed number plate. (b) The number plate after thresholdng. (c) The threshold value t determned by an analyss of the hstogram Adaptve thresholdng The number plate can be sometmes partally shadowed or nonunformly llumnated. Ths s most frequent reason why the global thresholdng fal. The adaptve thresholdng solves several dsadvantages of the global thresholdng, because t computes threshold value for each pel separately usng ts local neghborhood. Chow and Kaneko approach There are two approaches to fndng the threshold. The frst s the Chow and Kaneko approach, and the second s a local thresholdng. The both methods assumes that smaller rectangular regons are more lkely to have appromately unform llumnaton, more sutable for thresholdng. The mage s dvded nto unform rectangular areas wth sze of m n pels. The local hstogram s computed for each such area and a local threshold s determned. The threshold of concrete pont s then computed by nterpolatng the results of the submages. 1? Fgure 4.3: The number plate (from fgure 4.2) processed by the Chow and Kaneko approach of the adaptve thresholdng. The number plate s dvded nto the several areas, each wth own hstogram and threshold value. The threshold value of a concrete pel (denoted by ) s computed by nterpolatng the results of the submages (represented by pels 1-6). Local thresholdng The second way of fndng the local threshold of pel s a statstcal eamnaton of, y be a pel, for whch we compute the local threshold t. For neghborng pels. Let [ ] 28

36 smplcty we condder a square neghborhood wth wdth 2 r 1 [ r, y + r], [ + r, y r] and [ r, y r] approaches of computng the value of threshold: Mean of the neghborhood : t (, y) = mean f (, j) +, where [ r, y r], + + are corners of such square. There are severals r + r y r j y+ r { } { } Medan of the neghborhood : t (, y) = medan f (, j) r + r y r j y+ r Mean of the mnmum and mamum value of the heghborhood: 1 t (, y) = mn { f (, j) } + ma { f (, j) } 2 r + r r + r y r j y+ r y r j y+ r The new value f (, y) of pel [, ] y s then computes as: ( ) ( )) ( ) ( ) 0 f f, y 0, t, y f (, y) = 1 f f, y 0, t, y 4.2 Normalzaton of dmensons and resamplng Before etractng feature descrptors from a btmap representaton of a character, t s necessary to normalze t nto unfed dmensons. We understand under the term resamplng the process of changng dmensons of the character. As orgnal dmensons of unnormalzed characters are usually hgher than the normalzed ones, the characters are n most cases downsampled. When we downsample, we reduce nformaton contaned n the processed mage. There are several methods of resamplng, such as the pel-resze, blnear nterpolaton or the weghted-average resamplng. We cannot determne whch method s the best n general, because the successfulness of partcular method depends on many factors. For eample, usage of the weghed-average downsamplng n combnaton wth a detecton of character edges s not a good soluton, because ths type of downsamplng does not preserve sharp edges (dscussed later). Because of ths, the problematc of character resamplng s closely assocated wth the problematc of feature etracton. We wll assume that m n are dmensons of the orgnal mage, and m n are dmensons of the mage after resamplng. The horzontal and vertcal aspect rato s defned as r = m / m and r = n / n, respectvely. y Nearest-neghbor downsamplng The prncple of the nearest-neghbor downsampng s a pckng the nearest pel n the orgnal f, y be mage that corresponds to a processed pel n the mage after resamplng. Let ( ) a dscrete functon defnng the orgnal mage, such as 0 < m and 0 y < n. Then, the functon f (, y ) of the mage after resamplng s defned as: y f (, y ) = f, r r y 29

37 where 0 < m and 0 y < n. If the aspect rato s lower than one, then each pel n the resampled (destnaton) mage corresponds to a group of pels n the orgnal mage, but only one value from the group of source pels affects the value of the pel n the resampled mage. Ths fact causes a sgnfcant reducton of nformaton contaned n orgnal mage (see fgure 4.5). Fgure 4.4: One pel n the resampled mage corresponds to a group of pels n the orgnal mage Although the nearest neghbor downsampng sgnfcantly reduces nformaton contaned n the orgnal mage by gnorng a bg amount of pels, t preserves sharp edges and the strong bpolarty of black and whte pels. Because of ths, the nearest neghbor downsampng s sutable n combnaton wth the edge detecton feature etracton method descrbed n secton Weghed-average downsamplng In contrast wth the nearest-neghbor method, the weghted-average downsampng consders all pels from a correspondng group of pels n the orgnal mage. Let r and r y be a horzontal and vertcal aspect rato of the resampled mage. The value of n the destnaton mage s computed as a mean of source pels n the range the pel [, y ] [, y ] to [, ] mn where: mn y : ma ma 1 f, y = f, j ( ) ma ma mn j ymn ( ) ( ) ma mn yma y mn = = y ( ) mn = r y ; mn y = ry ; ma + 1 = r y ; ma y + 1 = ry 30

38 The weghted-average method of downsamplng does not preserve sharp edges of the mage (n contrast wth the prevous method). You can see the vsual comparson of these two methods n Fgure 4.5. n n n n m m Fgure 4.5: (a) Nearest-neghbor resamplng sgnfcantly reduces nformaton contaned n the orgnal mage, but t preserves sharp edges. (b) Weghted average resamplng gves a better vsual result, but the edges of the result are not sharp. m m 4.3 Feature etracton Informaton contaned n a btmap representaton of an mage s not sutable for processng by computers. Because of ths, there s need to descrbe a character n another way. The descrpton of the character should be nvarant towards the used font type, or deformatons caused by a skew. In addton, all nstances of the same character should have a smlar descrpton. A descrpton of the character s a vector of numeral values, so-called descrptors, or patterns : = ( ) 0,, n 1 Generally, the descrpton of an mage regon s based on ts nternal and eternal representaton. The nternal representaton of an mage s based on ts regonal propertes, such as color or teture. The eternal representaton s chosen when the prmary focus s on shape characterstcs. The descrpton of normalzed characters s based on ts eternal characterstcs because we deal only wth propertes such as character shape. Then, the vector of descrptors ncludes characterstcs such as number of lnes, bays, lakes, the amount of horzontal, vertcal and dagonal or dagonal edges, and etc. The feature etracton s a process of transformaton of data from a btmap representaton nto a form of descrptors, whch are more sutable for computers. If we assocate smlar nstances of the same character nto the classes, then the descrptors of characters from the same class should be geometrcally closed to each other n the vector space. Ths s a basc assumpton for successfulness of the pattern recognton process. Ths secton deals wth varous methods of feature etracton, and eplans whch method s the most sutable for a specfc type of character btmap. For eample, the edge detecton method should not be used n combnaton wth a blurred btmap Pel matr The smplest way to etract descrptors from a btmap mage s to assgn a brghtness of each pel wth a correspondng value n the vector of descrptors. Then, the length of such vector s equal to a square ( w h ) of the transformed btmap: 31

39 = f,modw ( ) w where 0,, w h 1. Bgger btmaps produce etremely long vector of descrptors, whch s not sutable for recognton. Because of ths, sze of such processed btmap s very lmted. In addton, ths method does not consder geometrcal closeness of pels, as well as ts neghborng relatons. Two slghtly based nstances of the same character n many cases produce very dfferent descrpton vectors. Even though, ths method s sutable f the character btmaps are too blurry or too small for edge detecton. h w 251, 181, 068, 041, 032, 071, 197, 196, 014, 132, 213, 187, 043, 041, 174, 011, 200, 254, 254, 232, 164, 202, 014, 012, 128, 242, 255, 255, = 253, 212, 089, 005, 064, 196, 253, 255, 255, 251, 196, 030, 009, 165, 127, 162, 251, 254, 197, 009, 105, 062, 005, 100, 144, 097, 006, 170, 207, 083, 032, 051, 053, 134, 250 Fgure 4.6: The pel matr feature etracton method Detecton of character edges In contrast wth the prevous method, the detecton of character edges does not consder absolute postonng of each pel, but only a number of occurrences of ndvdual edge types n a specfc regon of the character btmap. Because of ths, the resultng vector s nvarant towards the ntra-regonal dsplacement of the edges, and towards small deformatons of characters. Btmap regons Let the btmap be descrbed by a dscrete functon f (, ) y, where w and h are dmensons, such as 0 < w and 0 y < h. We dvde t nto s equal regons organzed to three rows and two columns n the followng way: ( ) ( ) Let mn, y mn and ( ) ( ) ma, y ma be an upper left and bottom rght pont of a rectangle, whch determnates the regon r, such as: (0) (0) (0) w (0) h Regon r 0 : mn = 0, y mn = 0, ma = 1 2, yma = 1 3 (1) w Regon r 1 : mn = 2, (1) (1) (1) h y mn = 0, ma = w 1, yma = 1 3 (2) (2) h Regon r 2 : mn = 0, ymn = 3, (2) w (2) 2 h ma = 1 2, yma = 1 3 (3) w Regon r 3 : mn = 2, (3) h ymn = 3, (3) (3) 2 h ma = w 1, yma =

40 (4) (4) 2 h Regon r 4 : mn = 0, ymn = 3, (4) w ma = 2, (4) yma = h 1 (5) w Regon r 5 : mn = 2, (5) 2 h ymn = 3, (5) (5) ma = w 1, yma = h 1 There are several ways how to dstrbute regons n the character btmap. The regons can be dsjunctve as well as they can overlap each other. The fgure 4.7 shows the several possble layouts of regons. Fgure 4.7: Layouts of regons n the character btmap. The regons can be dsjunctve as well as they can overlap each other. Edge types n regon Let us defne an edge of the character as a 22 whte-to-black transton n a btmap. Accordng to ths defnton, the btmap mage can contan fourteen dfferent edge types llustrated n fgure 4.8. Fgure 4.8: The processed btmap can contan dfferent types of 22 edges. The statstcs of occurrence of each edge type causes uselessly long vector of descrptors. Because of ths, the smlar types of edges are consdered as the same. The followng lsts shows how the edges can be grouped together: (vertcal edges) (horzontal edges) ( / -type dagonal edges) ( \ -type dagonal edges) (bottom rght corner) (bottom left corner) (top rght corner) (top left corner) For smplcty, assume that edge types are not grouped together. Let η be a number of dfferent edge types, where h s a 22 matr that corresponds to the specfc type of edge: h 0 = 1 0, h 1 = 0 1, h 2 = 0 0, h 3 = 1 1, h 4 = 0 1, h 5 = 1 0, h 6 = h 7 = 0 0, h 8 = 1 0, h 9 = 0 1, h 10 = 1 1, h 11 = 1 1, h 12 = 0 1, h 13 =

41 Let ρ be a number of rectangular regons n the character btmap, where ( ) ma ( ) mn, ( ) mn y, ( ) ma and y are boundares of the regon r ( 0 ρ 1 ). If the statstcs consder η dfferent edge types for each of ρ regons, the length of the resultng vector s computed as η ρ : = (,,, η ρ ) Feature etracton algorthm At frst, we have to embed the character btmap f (, ) y nto a bgger btmap wth whte paddng to ensure a proper behavor of the feature etracton algorthm. Let the paddng be one pel wde. Then, dmensons of the embeddng btmap wll be w + 2 and h + 2. The f, y s then defned as: embeddng btmap ( ) 1 f = 0 y = 0 = w + 1 y = h + 1 f (, y) = f y f y w y h ( 1, 1) ( = 0 = 0 = + 1 = + 1) where w and h are dmensons of character btmap before embeddng. Color of the paddng s whte (value of 1). The coordnates of pels are shfted one pel towards the orgnal poston. The structure of vector of output descrptors s llustrated by the pattern below. The notaton h r means number occurrences of an edge represented by the matr h j n the regon r. ( r0, r0,, η r0, r1, r1,, η r1, rρ 1, rρ 1,, η rρ 1 ) = h h h h h h h h h regon r regon 0 1 regon rρ 1 We compute the poston k of the r r n the vector as k = η + j, where η s the j number of dfferent edge types (and also the number of correspondng matrces). The followng algorthm demonstrates the computaton of the vector of descrptors : zeroze vector for each regon begn for each pel [, ] begn for each matr begn f begn r, where 0,, ρ 1 do y n regon j r,where ( ) ( ) h, where j 0,, η 1 (, ) ( + 1, ) (, + 1) ( + 1, + 1) f y f y f y f y mn do h j = then let k = η + j let k = k + 1 end end end end and y ( ) y y ( ) do ma mn ma 34

42 4.3.3 Skeletonzaton and structural analyss The feature etracton technques dscussed n the prevous two chapters are based on the statstcal mage processng. These methods do not consder structural aspects of analyzed mages. The small dfference n btmaps sometmes means a bg dfference n the structure of contaned characters. For eample, dgts 6 and 8 have very smlar btmaps, but there s a substantal dfference n ther structures. The structural analyss s based on hgher concepts than the edge detecton method. It does not deal wth terms such as pels or edges, but t consders more comple structures (lke junctons, lne ends or loops). To analyze these structures, we must nvolve the thnnng algorthm to get a skeleton of the character. Ths chapter deals wth the prncple of skeletonzaton as well as wth the prncple of structural analyss of skeletonzed mage. The concept of skeletonzaton The skeletonzaton s a reducton of the structural shape nto a graph. Ths reducton s accomplshed by obtanng a skeleton of the regon va the skeletonzaton algorthm. The skeleton of a shape s mathematcally defned as a medal as transformaton. To defne the medal as transformaton and skeletonzaton algorthm, we must ntroduce some elementary prerequste terms.,, y, such as anɺɺ b means a Let N ɺɺ be a bnary relaton between two pels [ y ] and [ ] s a neghbor of b. Ths relaton s defned as: [ ] ɺɺ 8 [ ] [ ] ɺɺ [ ], y N, y = 1 y y = 1 for eght-pel neghbourhood, y N, y = 1 y y = 1 for four-pel neghbourhood 4 The border B of character s a set of boundary pels. The pel [, y ] s a boundary pel, f t s black and f t has at least one whte neghbor n the eght-pel neghborhood: [, y] B f (, y) = 0 [, y ]: f (, y) = 1 [, y] N ɺɺ [, y ] The nner regon I of character s a set of black pels, whch are not boundary pels: [, ] (, ) 0 [, ] y I f y = y B 8 Fgure 4.9: (a) Illustraton of the four-pel and eght-pel neghborhood. (b) The set of boundary and nner pels of character. The pece Ρ s then a unon of all boundary and nner pels ( B I Ρ = ). Snce there s only one contnuous group of black pels, all black pels belong to the pece Ρ. The prncple and the related termnology of the skeletonzaton are smlar to the pece etracton algorthm dscussed n secton

43 Medal as transformaton The medal as transformaton of the pece Ρ defned as follows. For each nner pel p I, we fnd the closest boundary pel pb B. If a pel p has more than one such neghbor, t s sad to belong to the medal as (or skeleton) of the Ρ. The concept of the closest boundary pel depends on the defnton of the Eucldean dstance between two pels n the orthogonal coordnate system. Mathematcally, the medal as (or skeleton) S s a subset of the Ρ defned as: { } ( ) ( ) ( ) p S p p : p B p B d p, p = d p, p = mn d p, p The pel p belongs to the medal as S f there ests at least two pels p 1 and p 2, such as Eucldean dstance between pels p and p 1 s equal to the dstance between pels p and p 2, and these pels are closest boundary pels to pel p. p B The Eucldean dstance between two pels p = [, y ] and p [, y ] Skeletonzaton algorthm (, ) = 2 ( ) ( ) d p p = s defned as: Drect mplementaton of the mathematcal defnton of the medal as transformaton s computatonally epensve, because t nvolves calculatng the dstance from every nner pel from the set I to every pel on the boundary B. The medal as transformaton s ntutvely defned by a so-called fre front concept. Consder that a fre s lt along the border. All fre fronts wll advance nto the nner of character at the same speed. The skeleton of a character s then a set of pels reached by more than one fre front at the same tme. The skeletonzaton (or thnnng) algorthm s based on the fre front concept. The thnnng s a morphologcal operaton, whch preserves end-pels and does not break connectvty. Assume that pels of the pece are black (value of zero), and background pels are whte (value of one). The thnnng s an teratve process of two successve steps appled to boundary pels of a pece. Wth reference to the eght-pel neghborhood notaton n fgure 4.9, the frst step flags a boundary pel p for deleton f each of the followng condtons s satsfed: At least one of the top, rght and bottom neghbor of the pel p must be whte (the pel p s whte just when t does not belong to the pece Ρ ). t r b p Ρ p Ρ p Ρ At least one of the left, rght and bottom neghbor of pel p must be whte. l r b p Ρ p Ρ p Ρ The pel p must have at least two, and at most s black neghbors from the pece Ρ. Ths condton prevents the algorthm from erasng end-ponts and from breakng the connectvty. 36

44 { p pɺɺ 8 p p } 2 N Ρ 6 The number of whte-to-black transtons n the ordered sequence t tr r br b bl l tl t p, p, p, p, p, p, p, p, p must be equal to one. ( t Ρ tr Ρ ) + ( tr Ρ r Ρ ) + ( r Ρ br Ρ ) + ( br Ρ b Ρ) ( b Ρ bl Ρ ) + ( bl Ρ l Ρ ) + ( l Ρ tl Ρ ) + ( tl Ρ t Ρ ) = 1 v p p v p p v p p v p p v p p v p p v p p v p p ( ) v 0 = 1 f f The frst step flags pel p for deleton, f ts neghborhood meets the condtons above. However, the pel s not deleted untl all other pels have been processed. If at least one of the condtons s not satsfed, the value of pel p s not changed. After step one has been appled to all boundary pels, the flagged pels are defntely deleted n the second step. Every teraton of these two steps thns the processed character. Ths teratve process s appled untl no further pels are marked for deleton. The result of thnnng algorthm s a skeleton (or medal as) of the processed character. Let the pece Ρ be a set of all black pels contaned n skeletonzed character. Let B be a set of all boundary pels. The followng pseudo-code demonstrates the thnnng algorthm more formally. Ths algorthm proceeds the medal as transformaton over a pece Ρ. do // teratve thnnng process let contnue = false let B = 0/ for each pel p n pece Ρ do // create a set of boundary pels p : p Ρ pn ɺɺ p then // f the pel p has at least one whte neghbor nsert pel p nto set B // but keep t also n Ρ f 8 for each pel p n set B do // 1.step of the teraton begn // f at least one condton s volated, skp ths pel f t ( p r p b p ) f l ( p r p b p ) f { p p 8 p p } f Ρ Ρ Ρ then contnue Ρ Ρ Ρ then contnue ( 2 N 6 ) ɺɺ then contnue Ρ ( t Ρ tr Ρ ) + ( tr Ρ r Ρ ) + ( r Ρ br Ρ ) + ( br Ρ b Ρ) ( b Ρ bl Ρ ) + ( bl Ρ l Ρ ) + ( l Ρ tl Ρ ) + ( tl Ρ t Ρ) 1 v p p v p p v p p v p p v p p v p p v p p v p p then begn contnue end // all tests passed flag pont p for deleton let contnue = true 37

45 end for each pel p n set B do // 2.step of the teraton f p s flagged then pull pont p from pece Ρ whle contnue = true Note: The pel p belongs to the pece Ρ when t s black: p Ρ f ( p) = 0 Fgure 4.10: (a) The character btmap before skeletonzaton. (b) The thnnng algorthm teratvely deletes boundary pels. Pels deleted n the frst teraton are marked by a lght gray color. Pels deleted n the second and thrd teraton are marked by dark gray. (c) The result of the thnnng algorthm s a skeleton (or a medal as). Structural analyss of skeletonzed character The structural analyss s a feature etracton method that consders more comple structures than pels. The basc dea s that the substantal dfference between two compared characters cannot be evaluated by the statstcal analyss. Because of ths, the structural analyss etracts features, whch descrbe not pels or edges, but the more comple structures, such as junctons, lne ends and loops. Juncton The juncton s a pont, whch has at least three black neghbors n the eght-pel neghborhood. We consder only two types of junctons: the juncton of three and four lnes. The number of junctons n the skeletonzed pece Ρ s mathematcally defned as: Lne end { :{, } Nɺɺ 8 } { :{, } Nɺɺ 8 } 3 3 j n = p p p p Ρ p p 3 4 j n = p p p p Ρ p p The lne end s a pont, whch has eactly one neghbor n the eght-pel neghborhood. The number of lne-ends n a skeletonzed pece Ρ s defned as: e {! 1 : {, 1} N 8 1} n = p p p p Ρ p ɺɺ p The followng algorthm can be used to detect the number of junctons and number lne-ends n a skeletonzed pece Ρ : 38

46 let n j = 0 let n e = 0 for each pel p n pece Ρ do begn let neghbors = 0 end for each pel p n neghborhood { t, tr, r, br, b, bl, l, tl } f p Ρ then let neghbors = neghbors+1 f neghbors = 1 then let ne = ne +1 else f neghbors 3 then let n = n +1 j j p p p p p p p p do Fgure 4.11: (a, b) The juncton s a pel, whch as at least three neghbors n eghtpel neghborhood. (c) The lne end s a pel, whch has only one neghbor n eghtpel neghborhood (d) The loop s a group of pels, whch encloses the contnuous whte space. Loops It s not easy to determne the number of loops n l n the skeletonzed character. The algorthm s based on the followng prncple. At frst, we must negate the btmap of the skeletonzed character. Black pels wll be consdered as background and whte pels as foreground. The number of loops n the mage s equal to a number of lakes, whch are surrounded by these loops. Snce the lake s a contnuous group of whte pels n the postve mage, we apply the pece etracton algorthm on the negatve mage to determne the number of black peces. Then, the number of loops s equal to the number of black peces mnus one, because one pece represents the background of the orgnal mage (negated to the foreground). Another way s to use a seres of morphologcal erosons. Fgure 4.12: (a) We determne the number of lakes n skeleton by applyng the pece-etracton algorthm on negatve mage. The negatve mage (b) contans three peces. Snce the pece 3 s a background, only two peces are consdered as lakes. (c)(d) The smlar skeletons of the same character can dffer n the number of junctons 39

47 Snce we do not know the number of edges of the skeleton, we cannot use the standard cyclomatc equaton know from the graph theory. In addton, two smlar skeletons of the same character can sometmes dffer n a number of junctons (see fgure 4.12). Because of ths, t s not recommended to use constrants based on the number of junctons. Structural constrants To mprove the recognton process, we can assume structural constrants n the table 4.1. The syntactcal analyss can be combned by other methods descrbed n prevous chapters, such as edge detecton method or pel matr. The smplest way s to use one global neural network that returns several canddates and then select the best canddate that meets the structural constrants (fgure 4.13.a). More sophstcated soluton s to use the structural constrants for adaptve selecton of local neural networks (fgure 4.13.b). Lne ends Loops Junctons 0 BDO08 CEFGHIJKLMNSTUVWXYZ CDGIJLMNOSUVWZ PQ69 ADOPQR09 EFKPQTXY469 2 ACGIJLMNRSUVWZ B8 ABHR8 3 EFTY 4 HKX Table 4.1: Structural constrants of characters. A B C Fgure 4.13: (a, b) Structural constrants can be appled before and after the recognton by the neural network. (c) Eample of the skeletonzed alphabet. Feature etracton In case we know the poston of structural elements, we can form a vector of descrptors drectly from ths nformaton. Assume that there are several lne-ends, loops, and junctons n the 40

48 mage. The poston of loop s defned by ts centre. To form the vector, we must convert r, θ (see fgure 4.14): rectangular coordnates of the element nto polar coordnates [ ] 2 2 y r = + y ; θ = atg ; 2 w = ; w 2 y h y = h where and y are normalzed rectangular coordnates. The length and the structure of resultng vector vary accordng to a number and type of structural elements contaned n the character. Snce the structural constrants dvde characters nto the several classes, there are several possble types of descrpton vector. Each type of vector corresponds to one class of character. For eample, consder character wth two lne ends and one juncton. Ths constrant determnes the followng class of possble characters: (G, I, J, L, M, N, S, U, V, W, Z, 1, 2, 3, 5, 7). We defne a vector of descrptors to dstngush between these characters as follows: = ( r, θ, r, θ, r, θ ) lne end lne end juncton 1 2 w [, y ] [ r, θ ] h r θ 0 Fgure 4.14: (a) The skeleton of the character contans several structural elements, such as junctons, loops and lne ends. (b, c) Each element can be postoned n the rectangular or polar coordnate system. 1 41

49 Chapter 5 Recognton of characters The prevous chapter deals wth varous methods of feature etracton. The goal of these methods s to obtan a vector of descrptors (so-called pattern), whch comprehensvely descrbes the character contaned n a processed btmap. The goal of ths chapter s to ntroduce pattern recognton technques, such as neural networks, whch are able to classfy the patterns nto the approprate classes. 5.1 General classfcaton problem The general classfcaton problem s formulated usng the mappng between elements n two sets. Let A be a set of all possble combnatons of descrptors, and B be a set of all classes. The classfcaton means the projecton of group of smlar element from the set A nto a common class represented by one element n the set B. Thus, one element n the set B corresponds to one class. Usually the group of dstngushable nstances of the same character corresponds to the one class, but sometmes one class represents two mutually ndstngushable characters, such as 0 and O. Let F be a hypothetc functon that assgn each element from the set A to an element from the set B : F : A B ˆ = F( ) where A s a descrpton vector (pattern) whch descrbes the structure of classfed character and ˆ B s a classfer, whch represents the semantcs of such character. The functon F s the probably best theoretcal classfcator, but ts constructon s mpossble snce we cannot deal wth each combnaton of descrptors. In pras, we construct pattern classfer by usng only a lmted subset of the A B mappngs. Ths subset s known as a tranng set, such as At A and Bt B. Our goal s to construct an appromaton F ɶ, w of the hypothetc functon F, where w s a parameter that affects the qualty of the ( ) appromaton: ( w) Fɶ : A B ( w) ˆ = Fɶ, where At A, ˆ Bt B ˆF w s a restrcton of the projecton F over a set At A. We assume that for each At we know the desred value ˆ B t :. Formally we can say that ( ) ˆ, ˆ, ˆ,, n ˆ n t t 42

50 F A Fɶ ( w) ˆ B At Bt Fgure 5.1: The projecton between sets A and B. The F s a hypothetc functon that maps every possble combnaton of nput pattern A to a correspondng class ˆ B. Ths projecton s appromated by a functon Fɶ ( w), whch maps nput patterns from tranng set A t nto the correspondng classes from the set B t The problem s to fnd an optmal value (or values) of a parameter w. The w s typcally a vector (or matr) of syntactcal weghts n a neural network. Accordng to ths parameter, the F, w F for nput values of the functon ɶ ( ) should be as closest as possble to the values of ( ) patterns from the tranng set parameter w : A t. We defne an error functon to evaluate worthness of the m 1 ( ˆ ) 1 E ( w) = Fɶ (, w) 2 where m s a number of patterns 0 m 1 n the tranng set = 0 { } 2 A t. Let w + to be an optmal ɶ, w of value of the parameter w, such as w = E ( w ). Then, the appromaton ( ) + arg mn W the functon F ( ) s consdered as adapted. The adapted appromaton F (, w+ ) orgnal functon F ( ) for patterns from the tranng set w F + ɶ smulates A t. In addton, ths appromaton s able to predct the output classfer ˆ for unknown pattern from the test set ( A A At = ). The functon wth such predcton ablty partally substtutes the hypothetc F, w classfcator F ( ). Snce the functon ( ) network for ts mplementaton. A 5.2 Bologcal neuron and ts mathematcal models For a better understandng of artfcal neural network archtecture, there s a need to eplan the structure and functonalty of a bologcal neuron. The human bran s a neural network of about ten bllons nterconnected neurons. Each neuron s a cell that uses a bochemcal reacton to process and transmt nformaton. The neural cell has a body of sze about several mcrometers and thousands of nput connectons called dendrtes. It also has one output connecton called aon, whch can be several meters long. The data flow n the bologcal neural network s represented by electrcal sgnal, whch propagates along the aon. When the sgnal reaches a synaptc connecton between the aon and a consecutve dendrte, t releves molecules of chemcal agent (called medators or neuro-transmtters) nto such dendrte. Ths acton causes a local change of polarty of a dendrte transmsson membrane. The dfference n the polarty of the transmsson membrane actvates a dendrte-somatc potental wave, whch advances n a system of branched dendrtes nto the body of neuron. 43

51 Fgure 5.2: The bologcal neuron The bologcal neural network contans two types of synaptc connectons. The frst s an ectve connecton, whch amplfes the passng sgnal. The second (nhbtve) connecton suppresses the sgnal. The behavor of the connecton s represented by ts weght. The neural network contans mechansm whch s able to alter the weghts of connectons. Because of ths, the system of synaptc weghts s a realzaton of human memory. As the weghts are contnually altered, the old nformaton s beng forgotten lttle by lttle. dendrte aon termnal button of aon neuro-transmtters Cell body termnal buttons dendrte Fgure 5.3: (a) Schematc llustraton of the neural cell (b) The synaptc connecton between a dendrte and termnal button of the aon Snce the problematc of the bologcal neuron s very dffcult, the scentsts proposed several mathematcal models, such as McCulloch-Ptts bnary threshold neuron, or the percepton McCulloch-Ptts bnary threshold neuron The McCulloch-Ptts bnary threshold neuron was the frst model proposed by McCulloch and Ptts n The neuron has only two possble output values (0 or 1) and only two types of the synaptc weghts: the fully ectatve and the fully nhbtve. The ectatve weght (1) does not affect the nput, but the nhbtve one negates t (-1). 44

52 The weghted nputs are counted together and processed by a neuron as follows: J 1 y = g w, j j ϑ j= 0 0 f ξ < 0 g ( ξ ) = 1 f ξ 0 Ths type of neuron can perform logcal functons such as AND, OR, or NOT. In addton, McCulloch and Ptts proved that synchronous array of such neurons s able to realze arbtrary computatonal functon, smlarly as a Turng machne. Snce the bologcal neurons have not bnary response (but contnuous), ths model of neuron s not sutable for ts appromaton Percepton Another model of neuron s a percepton. It has been proved that McCulloch-Ptts networks wth modfed synaptc connectons can be traned for the recognton and classfcaton. The tranng s based on a modfcaton of a neuron weghts, accordng to the reacton of such neuron as follows. If the neuron s not actve and t should be, we ncrease the weghts. If the neuron s actve and t should not be, we decrease them. Ths prncple was been used n a frst model of the neural classfer called ADALINE (adaptve lnear neuron). The major problem of such networks s that they are not able to solve lnearly nonseparable problems. Ths problem has been solved when the scentsts Rumelhart, Hlton and Wllams proposed the error back-propagaton method of learnng for multlayered percepton networks. The smple McCulloch-Ptts bnary threshold neurons have been replaced by neurons wth contnuous saturaton nput/output functon. Percepton has multple analogous nputs and one analogous output. Let o j 1 be nputs wth correspondng weghts w,0 w, j 1. The weghted nputs are counted, thresholded and saturated together n the followng way: J 1 y = g w, j j ϑ j= 0 1 g ( ξ ) = 1 + where g ( ξ ) s a sgmod saturaton functon (see fgure 5.4.b) and ϑ s a threshold value. Sometmes, the threshold s mplemented as a dedcated nput wth a constant weght of -1 (see fgure 5.4.a). Then, the functon of a neuron can be smplfed to y = g ( ) e ξ w, where s a vector of nputs (ncludng the threshold value), and w s a vector of weghts (ncludng the constant weght -1). 45

53 1 1+ e ξ ϑ g y J 1 ξ Fgure 5.4: (a) The summaton Σ and gan (saturaton) g functon of the percepton wth a threshold mplemented as a dedcated nput. (b) The sgmod saturaton functon. 5.3 Feed-forward neural network Formally, the neural network s defned as an orented graph G ( N, E) =, where N s a nonempty set of neurons, and E s a set of orented connectons between neurons. The connecton e n, n E s a bnary relaton between two neurons n and n. The set of all neurons N s ( ) composed of dsjunctve sets N 0, N 1, N 2, where N = N0 N1 N2 The j th weght of a th neuron n a k th layer s denoted as ( k ) N s a set of all neurons from the th layer. ( k), j w and the threshold of th neuron n a k th layer s denoted as ϑ. Numbers of neurons for the nput (0), hdden (1) and output (2) layer are denoted as m, n, o, such as m = N0, n = N1 and o = N2. The number of neurons n the nput layer ( m ) s equal to a length of an nput pattern n order that each value of the pattern s dedcated to one neuron. Neurons n the nput layer do not perform any computaton functon, but they only dstrbute values of an nput pattern to neurons n the hdden layer. Because of ths, the nput layer neuron has only one nput drectly mapped (0) nto multple outputs. Because of ths, the threshold value ϑ of the nput layer neuron s equal (0) to zero, and the weghts of nputs w,0 are equal to one. The number of neurons n the hdden layer ( n ) s scalable, but t affects the recognton abltes of a neural network at a whole. Too few neurons n the hdden layer causes that the neural network would not be able to learn new patterns. Too many neurons cause network to be overlearned, so t wll not be able to generalze unknown patterns as well. The nformaton n a feed-forward neural network s propagated from lower layers to upper layers by one-way connectons. There are connectons only between adjacent layers, thus feedforward neural network does not contan feedback connectons, or connectons between arbtrary two layers. In addton, there est no connectons between neurons from the same layer. 46

54 y y 0 1 yo 1 (2) (2) ϑ 0 ϑ 1 (2) ϑo 1 Data flow z0 z1 zn 1 (1) (1) (1) ϑ 0 ϑ 1 ϑn 1 w w (2), j (1), j (0) (0) ϑ 0 ϑ 1 (0) ϑm m 1 Fgure 5.5: Archtecture of the three layer feed-forward neural network. 5.4 Adaptaton mechansm of feed-forward neural network There has been proven that a multlayered neural network composed of perceptons wth a sgmod saturaton functon can solve an arbtrary non-lnear problem. Mathematcally, for each m o functon F : R R there ests a multlayered feed-forward neural network that s able to realze ths functon. The proof s based on the Kolmogorov s theorem, whch tells that every contnuously growng functon f defned on nterval 0,1 m can be wrtten as: 2 m m 1 f ( 0 m 1 ) = α φ, j ( j ) = 0 j= 0 where α are properly chosen contnuous functons wth one parameter. The problem s how to construct the neural network correspondng to a gven non-lnear functon. At frst, we choose a proper topology of the network. The number of neurons n the nput and output layer s gven by lengths of the nput and output patterns, whle the number of neurons n the hdden layer s scalable. An adaptaton of the neural network means fndng the optmal parameter w + of the F ɶ, w dscussed n secton 5.1. Let us defne two error functons to appromaton functon ( ) evaluate a worthness of the parameter w : A t ( Fɶ (, ) ɶ ) 2 1 ; E Fɶ (, ) 1 E = 2 w t A ( ɶ ) 2 = 2 w where subscrpt t means tran, and means test. The E t s an error functon defned for patterns from the tranng set, and E for patterns from the test set. The response of the neural y = F ɶ, w. network to an nput pattern s gven as ( ) The error functon t E goes down as a number of neurons n the hdden layer grows. Ths relaton s vald also between the functon E t and a number of teratve steps of the adaptaton process. These relatons can be mathematcally descrbed as follows: 47

55 lm E = 0 ; lm E = 0 n t where n s the number of neurons n the nput layer and k s the number of teraton steps of the adaptaton process. The error functon E does not have a lmt at zero as n and k goes to nfnty. Because of ths, there ests an optmal number of neurons and optmal number of teraton steps, n whch the functon E has a mnmum (see fgure 5.6). E k t E E t n or k Fgure 5.6: Dependency of error functons E t and E on the number of neurons n nput layer ( n ) and the number of teraton steps ( k ). For smplcty, we wll assume only a feed-forward neural network wth one layer of hdden neurons defned n secton 5.3. All neurons n adjacent layers are connected by orented connectons. There are no feedback connectons, or connectons between neurons wthn a sngle layer. The actvtes of hdden and output neurons are defned as: ; m 1 (1) (1) z = g w, j j ϑ j= 0 actvtes of neurons n the hdden layer n 1 (2) (2) y = g w, j z j ϑ j= 0 actvtes of neurons n the output layer where g ( ξ ) s a sgmod saturaton functon (see fgure 5.4.b) Actve phase Evaluaton of the actvtes of hdden and output neurons s performed n so-called actve phase. The actve phase conssts of two steps n three-layer neural networks. The frst step s an evaluaton of actvtes z n the hdden layer, and the second step s an evaluaton of actvtes y. Snce the evaluaton of actvtes s performed from bottom layers to top ones, the term feed-forward refers to ths prncple. The actve phase corresponds to an appromaton F, w F, and t s performed every tme when there s a need to classfy the ɶ ( ) of functon ( ) nput pattern. The followng pseudo-code demonstrates the actve phase of feed-forward neural network. The notaton s the same as n fgure

56 procedure actvephase (nput: w, // vector of thresholds and weghts ; // nput pattern to be classfed output: z, // vector of actvtes of neurons n hdden layer y // vector of actvtes of neurons n output layer (neural network response) ) begn // frst step: evaluate actvtes of neurons n the hdden layer for each neuron n hdden layer wth nde 0,, 1 begn end (1) let ξ = w ϑ for each nput wth nde j 0,, m 1 let (1) ξ = ξ + w w, j = g ξ let z ( ) j do n do // second step: evaluate actvtes of neurons n the output layer for each neuron n output layer wth nde 0,, o 1 do begn (2) let end end ξ = w ϑ for each nput wth nde j 0,, n 1 let (2) ξ = ξ + w w, j z = g ξ let y ( ) j do Partal dervatves and gradent of error functon The goal of the tranng phase s to fnd optmal values of thresholds and weghts to mnmze the error functon E t. Adaptaton phase s an teratve process n whch a response y to an nput pattern s compared wth the desred response ˆ. The dfference between the obtaned and desred response s used for a correcton of weghts. The weghts are teratvely altered untl the value of the error functon E s neglgble. Gradent of error functon related to a sngle pattern t We compute a gradent g of an error functon related to a sngle pattern wth desred and obtaned responses y and ˆ. The gradent g s computed n drecton from upper layers to lower layers as follows: At frst, we compute components of the gradent related to thresholds layer as (2) ϑ n the output E ϑ (2) ( ˆ ) ( 1 ) = y y y 49

57 (1) Then, we compute components of the gradent related to thresholds ϑ n the hdden layer. These components are computed usng the components E (2) ϑ from the prevous step as follows: ϑ o 1 E E (2) = z (1) ( 1 y ) w (2), j j= 0 ϑ Smlarly, we compute components of the gradent related to weghts followng way: (2), j w and (1), j w n the E E = z w (2) (2), j ϑ j E E = ; (1) (1) w, j ϑ j The gradent g s a vector of components s gven as follows: E E E E E E E E g =,,,,,,,,,,, (1) (1) (2) (2) (1) (1) (2) (2) ϑ0 ϑn 1 ϑ0 ϑo 1 w0,0 wn 1, m 1 w0,0 w o 1, n 1 Overall gradent The overall gradent s defned as a summary of gradents related to ndvdual patterns of the tranng set A. Let g be a gradent related to a tranng par / ˆ. The overall gradent s t / ˆ computed as A t / ˆ / ˆ g Adaptaton phase The adaptaton phase s an teratve process of fndng optmal values of weght and thresholds, for whch a value of the error functon E s n a local mnmum. The fgure 5.7 schematcally llustrates a graph of the functon t t E - so called error landscape. Generally, the error landscape s w + 1 dmensonal, where w s a cardnalty of the vector of thresholds and weghts, such as: ( ϑ (1) (1) (2) (2) (1) (1) (2) (2) 0,, ϑn 1, ϑ0,, ϑo 1, w0,0,, wn 1, m 1, w0,0,, wo 1, n 1 ) w = 50

58 E t w Fgure 5.7: The numerc approach of fndng the global mnmum n the error landscape. The vector of optmal thresholds and weghts w + s represented by a global mnmum n the error landscape. Snce we cannot compute ths mnmum analytcally, we have to use a numerc approach. There are varous numerc optmzaton algorthms, such as Newton s method, or the gradent descent. We use the gradent descent algorthm to fnd the global mnmum n the error landscape. The sngle step of the teratve algorthm can looks lke follows: where k ( l), j E ( ) k+ 1 ( l) k ( l) k ( l) k 1 ( l) ϑ = ϑ λ + µ ϑ k ( l) ϑ ϑ E ( ) k + 1 ( l) k ( l) k ( l) k 1 ( l) w, j = w, j λ + µ w k ( l), j w, j w, j w s a weght of the connecton between the th neuron n l th layer and j th neuron n l- 1 th layer computed n a k th step of teratve process. The speed of convergence s represented by a parameter λ. Too small value of the parameter λ causes ecessvely slow convergence. Too bg value of the λ breaks the monotony of convergence. The µ s a momentum value, whch prevents the algorthm of gettng stuck n local mnmums. E E Note: The notaton g ( l) means the component of the vector (or gradent). The ( l) ϑ ϑ E ( l) s a partal dervatve of error functon E by the threshold value ϑ ( l). Smlarly, the ϑ E w ( l), j s a partal dervatve of functon E by the value of weght The whole adaptaton algorthm of feed-forward neural network can be llustrated by the followng pseudo code. ( l), j w. 51

59 procedure adaptaton (nput: A, // tranng set of patterns / ˆ t λ, // speed of convergence µ, // momentum value k ma, // mamum number of teratons ε, // precson of adaptaton process output: w +, // vector of optmal weghts and thresholds ) begn ntalze weghts and thresholds n w to random values let w prev = w // we haven t a prevous value of w at the begnnng let k = 0, E = whle k kma E > ε do begn // compute overall gradent zeroze overall gradent g for each par / ˆ of A t do begn // compute gradent g / ˆ for tranng par / ˆ zeroze gradent g / ˆ actvephase( w,, z, y )// compute actvtes z, y for each threshold (2) for each threshold ϑ (1) do E ϑ do g ( ˆ ) ( 1 ) / ˆ (2) ϑ o 1 E E (2) / ˆ = z (1) ( 1 y ) / ˆ w (2), j ϑ j= 0 ϑ g g w = y y y for each weght (2) E E w, j do g/ ˆ (2) = g / ˆ z (2) w, j ϑ E E g/ (1) = g / (1) w, j ϑ for each weght w (1) do ˆ ˆ, j let g = g + g / ˆ end // alter values of thresholds and weghts accordng to the gradent g end let w + = w end for each threshold ϑ ( l) n w + do ( l) ( l) E ( l) ( l) let w net ϑ = w ϑ λ g µ ( l) + ( w ϑ w prev ϑ ) ϑ for each weght ( l w ), j n w + do w, j let w prev = w, w = w net ( l) ( l) E ( l) ( l) let w net w, j = w w, j λ g µ ( l) + ( w w, j w prev w, j ) j j 52

60 5.5 Heurstc analyss of characters The segmentaton algotthm descrbed n chapter three can sometmes detect redundant elements, whch do not correspond to proper characters. The shape of these elements after normalzaton s often smlar to the shape of characters. Because of ths, these elements are not relably separable by tradtonal OCR methods, although they vary n sze as well as n contrast, brghtness or hue. Snce the feature etracton methods descrbed n chapter four do not consder these propertes, there s a need to use addtonal heurstc analyses to flter non-character elements. The analyss epects all elements to have smlar propertes. Elements wth consderably dfferent propertes are treated as nvald and ecluded from the recognton process. The analyss conssts of two phases. The frst phase deals wth statstcs of brghness and contrast of segmented characters. Characters are then normalzed and processed by the pece etracton algorthm. Snce the pece etracton and normalzaton of brghtness dsturbs statstcal propertes of segmented characters, t s necessary to proceed the frst phase of analyss before the applcaton of the pece etracton algorthm. In addton, the heghts of detected segments are same for all characters. Because of ths, there s a need to proceed the analyss of dmensons after applcaton of the pece etracton algorthm. The pece etracton algorthm strps off whte paddng, whch surrounds the character. Respectng the constrants above, the sequence of steps can be assembled as follows: 1. Segment the plate (result s n fgure 5.8.a). 2. Analyse the brghtness and contrast of segments and eclude faulty ones. 3. Apply the pece etracton algorthm on segments (result s n fgure 5.8.b). 4. Analyse the dmensons of segments and eclude faulty ones. Fgure 5.8: Character segments before (a) and after (b) applcaton of the pece etracton algorthm. Ths algorthm dsturbs statstcal propertes of brghtness and contrast. If we assume that there are not bg dfferences n brghtness and contrast of segments, we can eclude the segments, whch consderably dffers from the mean. Let th segment of plate be defned by a dscrete functon f (, y ), where w and h are dmensons of the element. We defne the followng statstcal propertes of an element: The global brghtness of such segment s defned as a mean of brghtnesses of ndvdual pels: ( ) b w h = = 0 y= 0 (, ) p f y 53

61 The global contrast of the th segment s defned as a standard devaton of brghtnesses of ndvdual pels: p ( ) c = w h = 0 y= 0 ( ) ( p (, )) 2 b f y w h The functon f (, ) y represents only an ntensty of grayscale mages, but the addtonal heurstc analyss of colors can be nvolved to mprove the recognton process. Ths analyss separates character and non-character elements on color bass. If the captured snapshot s represented by a HSV color model, we can drectly compute the global hue and saturaton of the segments as a mean of hue and saturaton of ndvdual pels: where h(, y ) and (, ) p ( ) h w h w h ( ) = h(, y) ; p s(, y) = 0 y= 0 s = = 0 y= 0 s y s a hue and saturaton of the certan pel n the HSV color model. If the captured snapshot s represented by a RGB color model, there s need to transform t to the HSV model frst. To determne the valdty of the element, we compute an average value of a chosen property over all elements. For eample, the average brghtness s computed as p b n 1 = a number of elements. The element s consdered as vald, f ts global brghtness not dffer more than 16 % from the average brghtness propertes have been calbrated as follows: b = 0 p ( ) b, where n s ( ) b p does p. The threshold values of ndvdual brghtness (BRI) hue (HUE) p p ( ) b ( ) h p p h b p p h h Heght (HEI) < 0.2 h h b < 0.16 Contrast (CON) < Saturaton (SAT) wdth/heght rato (WHR) p p ( ) c ( ) s p p s c p p s c < 0.1 < 0.24 w 0.1< < 0.92 h If the segment volates at least one of the constrants above, t s consdered as nvald and ecluded from the recognton process. The table 5.1 contans propertes of elements from fgure 5.8. Accordng to ths table, elements 0 and 10 have been refused due to an uncommon wdth/heght rato, and elements 1 and 4 due to a small heght. 54

62 BRI CON HUE SAT HEI WHR Volated constrants BRI,HUE,WHR HUE,HEI HEI,WHR WHR Table 5.1: Propertes of segments n fgure 5.8. The meanng of abbrevatons s as follows: BRI=brghtness, CON=contrast, HUE=hue, SAT=saturaton, HEI=heght, WHR=wdth/heght rato. 55

63 Chapter 6 Syntactcal analyss of recognzed plate 6.1 Prncple and algorthms In some stuatons when the recognton mechansm fals, there s a possblty to detect a falure by a syntactcal analyss of the recognzed plate. If we have country-specfc rules for the plate, we can evaluate the valdty of that plate towards these rules. Automatc synta-based correcton of plate numbers can ncrease recognton abltes of the whole ANPR system. For eample, f the recognton software s confused between characters 8 and B, the fnal decson can be made accordng to the syntactcal pattern. If the pattern allows only dgts for that poston, the character 8 wll be used rather than the character B. Another good eample s a decson between the dgt 0 and the character O. The very small dfference between these characters makes ther recognton etremely dffcult, n many cases mpossble Recognzed character and ts cost In most cases, characters are recognzed by neural networks. Each neuron n an output layer of a y = y,, y, y,, y be neural network typcally represents one character. Let ( ) 0 9 a vector of output actvtes. If there are 36 characters n the alphabet, the vector y wll be also 36-dmensonal. Let y be an th component of the vector y. Then, y means how much does the nput character corresponds to the th character n the alphabet, whch s represented by ths component. The recognzed character χ s represented by the greatest component of the vector y : χ = chr ( ma{ y} ) 0 z where chr ( y ) s the character, whch s represented by the th component of vector y. ( S ) Let y be a vector y descendngly sorted accordng to the values of components. Then, the recognzed character s represented by the frst component of so sorted vector: χ = chr y ( S ) ( 0 ) When the recognton process fals, the frst component of ( ) S y can contan nvald character, whch does not match the synta pattern. Then, t s necessary to use the net vald character wth a worse cost. A Z 56

64 6.1.2 Syntactcal patterns In pras, ANPR systems must deal wth many dfferent types of plate numbers. Number plates are not unfed, so each country has own type. Because of ths, number plate recognton system should be able to recognze a type of the number plate, and automatcally assgn the correct syntactcal pattern to t. The assgnaton of the rght syntactcal pattern s a fundamental problem n syntactcal analyss. Syntactcal pattern s a set of rules defnng characters, whch can be used on a certan poston n a plate number. If the plate number P s a sequence of n alphanumercal characters (0) ( n 1) = p p (0) ( n 1) `P = `p `p, P ( ), then the syntactcal pattern `P s a n -tuple of sets ( ) ( ) and `p s a set of all allowed characters for the th poston n a plate. For eample, czech number plates can contan dgt on a frst poston followed by a character denotng the regon, where the plate has been regstered and fve other dgts for a regstraton number of a car. Formally, the syntactcal pattern `P for czech number plates can looks lke ths: { 0,1, 2,3, 4,5, 6, 7,8,9 },{ C,B,K,H,L,T,N,E,P,A,S,U,J,Z }, { } { } { } { 0,1,2,3,4,5,6,7,8,9 },{ 0,1,2,3,4,5,6,7,8,9 },{ 0,1,2,3,4,5,6,7,8,9 } `P = 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, Choosng the rght pattern (0) ( n 1) If there are n syntactcal patterns `P `P, we have to choose the most sutable one for the evaluated plate number P. For ths purpose, we defne a metrcs (or a cost) δ for a computaton of a smlarty between the evaluated plate number and the correspondng syntactcal pattern: ( ) ( ) ( ) where { p p `p } n 1 ( ) ( ) ( ) 1 2 δ ( `P ) = { p p `p } + 10, ( ) = 0 ma{ y j } 0 j z s a number of characters, whch do not match to correspondng postons n the syntactcal pattern `P. Let ( ) y be an output vector for the th -recognzed ( ) character n a plate. The greatest component of that vector ma{ y j } 0 j z ( ) successfully the plate has been recognzed. Then, the recprocal value of ma{ y j } then ndcates how 0 j z s a cost of the character. Another way of the cost evaluaton s a usage of the Smth-Waterman algorthm to compute the dfference between the recognzed plate number and the syntactcal pattern. For eample, assume that plate number 0B01234 has been recognzed as , and the recognton pattern does not allow dgt at the second poston of a plate. If the character 8 has been recognzed wth smlarty rato of 0.90, and other characters wth the rato of 0.95, the metrcs for ths pattern s determned as follows δ ( `P ) = ( 1) = 1,

65 If there s a pattern that eactly matches to the evaluated plate number, we can say that number has been correctly recognzed, and no further correctons are needed. In addton, t s not possble to detect a faulty number plate, f t does not break rules of a syntactcal pattern. Otherwse, t s necessary to correct detected plate usng the pattern wth lowest cost δ : { δ ( ` )} `P = arg mn P ( sel ) ( ) 0 < n The correcton of a plate means the replacement of each nvald character by another one. If the ( ) character p at the th ( sel ) poston of the plate P does not match the selected pattern `P, t wll be replaced by the frst vald one from ( s) y. ( s) y s a sorted vector of output actvtes denotng how much the recognzed character s smlar to an ndvdual character from the alphabet. Heurstc analyss of a segmented plate can sometmes ncorrectly evaluate non-character elements as characters. Acceptance of the non-character elements causes that the recognzed plate wll contan redundant characters. Redundant characters occur usually on sdes of the plate, but rarely n the mddle. If the recognzed plate number s longer than the longest synta pattern, we can select the nearest pattern, and drop the redundant characters accordng to t. 58

66 Chapter 7 Tests and fnal consderatons 7.1 Choosng the representatve set of snapshots I have captured many of statc snapshots of vehcles for the test purposes. Random movng and standng vehcles wth Slovak and Czech number plates have been ncluded. At frst, my objectve was to fnd a representatve set of number plates, whch are recognzable by humans. Of course, the set lke ths contans etremely wde spectrum of plates, such as clear and easy recognzable as well as plates degraded by the sgnfcant moton blur or skew. Then, a recognton ablty of a machne s represented by a rato between the number of plates, whch have been recognzed by the machne, and the number of plates recognzed by a human. Practcally, t s mpossble to buld a machne wth the same recognton abltes as a human has. Because of ths, the test lke ths s etremely dffcult and useless. In pras, t s more useful to fnd a representatve set of number plates, whch can be captured by an ANPR camera. The poston of the camera has a sgnfcantly affects the qualty of captured mages, and a successfulness of the whole recognton process. The sutable poston of the camera towards the lane can lead to a better set of all possble snapshots. In some stuatons, we can avod of gettng skewed snapshots by a sutable postonng of the camera. Sometmes, ths s cleverer than a development of the robust de-skewng mechansms. Let S be a representatve set of all snapshots, whch can be captured by a concrete nstance of the ANPR camera. Some of the snapshots n ths set can be blurred, some of them can be too small, too bg, too skewed or too deformed. Because of ths, I have dvded the whole set nto a followng subsets: where plates, S = S S c b S S S S c s a subset of clear plates, S b s a subset of blurred plates, S s a subset of plates, whch has a dffcult surroundng envronment, and e subset of plates wth lttle characters. s e l S s s a subset of skewed S l s a 59

67 A B C D Fgure 7.1: Typcal snapshot from the set of (a) clear plates (b) plates wth lttle, or blurred characters (c) skewed plates (d) plates wth dffcult surroundng envronment 7.2 Evaluaton of a plate number correctness Plate numbers recognzed by a machne can sometmes dffer from the correct ones. Because of ths, there s a need to defne formulas and rules, whch wll be used to evaluate a degree of plate correctness. (0) ( n-1) Let P be a plate number, and S = { P,, P } be a set of all tested plate numbers. Then, recognton rate R ( S ) of the ANPR system tested on set S s calculated as: 1 ( ) n 1 ( ) s( ) R S = n P, = 0 where n s a cardnalty of the set S, and s( P ) s a correctness score of the plate P. The correctness score s a value, whch epress how successfully the plate has been recognzed. Now the queston s how to defne the correctness score of ndvdual plates. There are two dfferent approaches, how to evaluate t. The frst s a bnary score, and the second s a weghted score Bnary score Let us say, that plate number P s a sequence of n alphanumercal characters (0),, ( n 1) P = p p ( r) ( c). If P s the plate number recognzed by a machne, and P s the ( ) correct one, then bnary score s b of plate ( ) r P s evaluated as follows: 60

68 s ( r) ( P ) 0 = 1 ( r) ( c) b ( r) ( c) f f P P P = P Two plate numbers are equal, f all characters on correspondng postons are equal: ( r) ( c) ( r) ( r) ( = ) ( j : p p j j) P P, where p ( r) s the th character of plate number ( r) P Weghted score ( r) If P s a plate number recognzed by a machne, and ( r) score s w of plate P s gven as: s w ( r ) ( P ) ( r ) ( r ) ( c) { = } p p p m = = n ( r ) { p } ( c) P s the correct one, then weghted where m s the number of correctly recognzed characters, and n s the number of all characters n plate. For eample f the plate KE123AB has been recognzed as KE128AB, the weghted correctness score for ths plate s 0.85, but the bnary score s Results The table 7.1 shows recognton rates, whch has been acheved whle testng on varous set of number plates. Accordng to the results, ths system gves good responses only to clear plates, because skewed plates and plates wth dffcult surroundng envronment causes sgnfcant degradaton of recognton abltes. Total number Total number Weghted score of plates of characters Clear plates Blurred plates Skewed plates Average plates Table 7.1: Recognton rates of the ANPR system. 61

69 Summary The objectve of ths thess was to study and resolve algorthmc and mathematcal aspects of the automatc number plate recognton systems, such as problematc of machne vson, pattern recognton, OCR and neural networks. The problematc has been dvded nto several chapters, accordng to a logcal sequence of the ndvdual recognton steps. Even though there s a strong successon of algorthms appled durng the recognton process, chapters can be studed ndependently. Ths work also contans demonstraton ANPR software, whch comparatvely demonstrates all descrbed algorthms. I had more choces of programmng envronment to choose from. Mathematcal prncples and algorthms should not be studed and developed n a compled programmng language. I have consdered usage of the Matlab and the Java. Fnally, I mplemented ANPR n Java rather than n Matlab, because Java s a compromse programmng envronment between the Matlab and compled programmng language, such as C++. Otherwse, I would have to develop algorthms n Matlab, and then rewrte them nto a compled language as a fnal platform for ther usage n the real envronment. ANPR soluton has been tested on statc snapshots of vehcles, whch has been dvded nto several sets accordng to dffcultness. Sets of blurry and skewed snapshots gve worse recognton rates than a set of snapshots, whch has been captured clearly. The objectve of the tests was not to fnd a one hundred percent recognzable set of snapshots, but to test the nvarance of the algorthms on random snapshots systematcally classfed to the sets accordng to ther propertes. 62

70 Append A: Case study 134.jpg wdth:488 p heght:366 p The plate has been successfully recognzed - no further comment needed. Detected band wdth: 488 p heght: 30 p Detected plate Skew detecton 1.44 Segmentaton Number of detected characters: 10 Recognzed plate RK959AF 63

71 149.jpg wdth:526 p heght:350 p The plate has been successfully recognzed - no further comment needed. Detected band wdth: 526 p heght: 28 p Detected plate Skew detecton 0.0 Segmentaton Number of detected characters: 10 Recognzed plate RK959AD 64

72 034.jpg wdth:547 p heght:410 p The plate has been successfully recognzed - no further comment needed. Detected band wdth: 547 p heght: 42 p Detected plate Skew detecton Segmentaton Number of detected characters: 11 Recognzed plate LM010BE 65

73 049.jpg wdth:410 p heght:360 p The plate has been successfully recognzed - no further comment needed. Detected band wdth: 410 p heght: 27 p Detected plate Skew detecton Segmentaton Number of detected characters: 10 Recognzed plate RK878AC 66

74 040.jpg wdth:576 p heght:432 p The plate has been successfully recognzed - no further comment needed. Detected band wdth: 576 p heght: 21 p Detected plate Skew detecton 0.0 Segmentaton Number of detected characters: 11 Recognzed plate BA738DE 67

75 098.jpg wdth: 425 p heght: 330 p The plate has been successfully recognzed - no further comment needed. Detected band wdth: 425 p heght: 28 p Detected plate Skew detecton 0.0 Segmentaton Number of detected characters: 9 Recognzed plate 1B

76 60.jpg wdth:424 p heght:336 p Class: blurred characters A sgnfcant blur caused mproper detecton of the band n a graph of vertcal projecton. In addton, further heurstc analyses dd not detect ths fault. Because of ths, the ncorrect canddate to number plate has been deskewed and then segmented even though t does not have any semantcs. Detected band wdth: 424 p heght: 141 p Pont of falure: vertcal projecton and heurstc analyss of band. Detected plate Skew detecton Segmentaton Number of detected characters: 6 Recognzed plate N/A 69

77 023.jpg wdth:354 p heght:308 p Class: dffcult envronment Ths s a typcal snapshot wth a dffcult surroundng envronment. The table n the background contans more characters n one lne than a number plate n the foreground. Ths fact causes a bgger amount of horzontal edges n an area of the table. The three detected peaks n the graph of the horzontal projecton correspond Detected band wdth: 354 p heght: 19 p to three rows n the table. Although the number plate canddate s wrong, the further analyss dd not refuse t, because the number of characters (10) s wthn the allowed range. Detected plate Skew detecton Segmentaton Number of detected characters: 10 Recognzed plate 0CNCKEP 70

78 044.jpg wdth:530 p heght:397 p Class: skewed plates The part of graph correspondng to the peak 2 (n vertcal graph) has a wder dstrbuton due to the mproper vertcal projecton caused by a skew of the plate. Because of ths, the bottom of the last character E has been cut off mproperly. Detected band wdth: 530 p heght: 30 p Pont of falure : vertcal projecton band clppng Detected plate Skew detecton Segmentaton Number of detected characters: 7 Recognzed plate RK892AF 71

79 067.jpg wdth:402 p heght:298 p Class: etremely small characters Ths snapshot contans etremely small characters, whch are not dstngushable by a machne. The number plate has been properly detected and characters have been segmented as well, but t s very hard to dstngush between the B and 8 on the frst poston and between the 6 and 8 on the thrd poston of plate. Detected band wdth: 402 p heght: 21 p Pont of falure : character recognton Detected plate Skew detecton 4.00 Segmentaton Number of detected characters: 10 Recognzed plate 8Y849A4 72

80 Append B: Demo recognton software - User s manual JavaANPR s an ANPR recognton software that demonstrates prncples descrbed n ths thess. It s wrtten n the Java programmng language. If you want to run t, you wll need the Java SE runtme envronment (or hgher). After downloadng the dstrbuton package, please unpack t nto a chosen drectory. The dstrbuton package contans compled program classes, jar archve, source codes and addtonal program resources such as btmaps, neural networks etc. buld dst lb nbproject resources src Compled classes Dstrbuton drectory, contans JAR fle and addtonal resources Comple-tme lbrares Project metadata and buld confguraton Resources, confguraton fle, btmaps, neural networks Source fles 1. Cleanng, complng and buldng the project (optonal) Normally, you do not have to comple the project, because dstrbuton package already contans precompled bnares. If you want to recomple t agan, you can do t usng the Apache Ant utlty. At frst, change a workng drectory to the javaanpr, and type the followng command to clean the prevous buld of the JavaANPR. Issung ths command wll delete whole content of the buld and dst drectores: javaanpr # ant clean Then, ssue the ant comple and ant jar commands. The comple target wll comple all source fles n the src drectory. The jar target wll create the dst drectory wth a jar archve and addtonal run-tme resources. javaanpr # ant comple javaanpr # ant jar 2. Runnng the vewer You can run the nteractve ANPR vewer usng the ant by typng the followng command: javaanpr # ant run If you do not have nstalled the ANT utlty, you can run vewer manually by the followng commands: javaanpr # cd./dst dst # java jar javaanpr.jar Another way to run the vewer s a double-clck to a javaanpr.jar archve (n the MS Eplorer) 73

81 Fgure B.1: Graphcal user nterface of the JavaANPR vewer Important: By default, the program epects the confguraton fle confg.ml and other resources n the workng drectory. Because of ths, please do not run the jar archve from other drectores. Otherwse, the program wll not be able to start. 3. Usng command-lne arguments Besdes the graphcal user nterface, program also contans addtonal functons, whch are accessble usng the command-lne arguments. For more nformaton about t, please run the jar fle wth a -help command: Automatc number plate recognton system Copyrght (c) Ondrej Martnsky, Lcensed under the Educatonal Communty Lcense Usage : java -jar javaanpr.jar [-optons] Where optons nclude: -help Dsplays ths help -gu Run GUI vewer (default choce) -recognze - <snapshot> Recognze sngle snapshot -recognze - <snapshot> -o <dstdr> Recognze sngle snapshot and save report html nto specfed drectory -newconfg -o <fle> Generate default confguraton fle -newnetwork -o <fle> Tran neural network accordng to specfed feature etracton method and learnng parameters (n confg. fle) and saves t nto output fle -newalphabet - <srcdr> -o <dstdr> Normalze all mages n <srcdr> and save t to <dstdr>. 3.1 Command-lne recognton If you do not want to use the GUI vewer, you can recognze snapshot by ssung the followng command. The recognzed plate wll be wrtten to standard output. 74

82 dst # java jar javaanpr.jar recognze <name of mage> 3.2 Recognton report Sometmes, t s good to see nsde the recognton process of concrete mage. Because of ths, JavaANPR supports a generaton of HTML reports. The recognton report contans mages and verbose debuggng nformaton about each step of the recognton process. HTML report can be used to determne a pont, n whch the recognton process faled. The followng command wll recognze the mage specfed by ts name, and save the report nto a specfed destnaton drectory: dst # java jar javaanpr.jar recognze <name of mage> -o <destnaton drectory> 3.3 Creatng the default confguraton fle Confguraton fle contans settngs and parameters, whch are needed durng the recognton process. If confguraton fle does not est, program wll not be able to start. Because of ths, JavaANPR s able to generate a default confguraton fle wth recommended confguraton settngs by the followng command: dst # java jar javaanpr.jar -newconfg -o <fle> 75

83 Bblography [1] Fajmon B.: Numerc Math and Probablty, scrpts, Faculty of Electrcal Engneerng and Communcaton, Brno, Czech Republc, 2005 [2] Fraser N.: Introducton to Neural Networks, [3] Fukunaga K.: Introducton to statstcal pattern recognton, Academc Press, San Dego, USA, 1990 [4] Gonzalez R., Woods R.: Dgtal Image Processng, Prentce Hall, Upper Saddle Rver, New Jersey, 2002 [5] Kovar M.: Dscreet Math, scrpts, Faculty of Electrcal Engneerng and Communcaton, Brno, Czech Republc, 2003 [6] Kuba M.: Neural Networks, scrpts, Faculty of Informatcs, Masaryk Unversty, Brno, Czech Republc [7] Kvasncka V., Benuskova L., Pospchal J., Farkas I., Tno P., Kral A.: Introducton to Neural Networks, Techncal Unversty, Kosce, Slovak Republc [8] Mnsky M., Papert S.: Perceptons. An Introducton to Computatonal Geometry, MIT Press:. Cambrdge, Massachusetts, 1969 [9] Shapro V., Dmov D., Bonchev S., Velchkov V., Gluhchev G.: Adaptve Lcense Plate Image Etracton, Internatonal Conference Computer Systems and Technologes, Rousse, Bulgara, 2004 [10] Smagt P.: Comparatve study of neural network algorthms appled to optcal character recognton, Internatonal conference on Industral and engneerng applcatons of artfcal ntellgence and epert systems, Charleston, South Carolna, USA, 1990 [11] Srvastava R: Transformatons and dstortons tolerant recognton of numerals usng neural networks, ACM Annual Computer Scence Conference, San Antono, Teas, USA, 1991 [12] Wang J., Jean J.: Segmentaton of merged characters by neural networks and shortest-path, Symposum on Appled Computng, Indanapols, Indana, USA, 1993 [13] Zborl F.: Neural Networks, Department of Intellgent Systems, Faculty of Informaton Technology, BUT Brno, Czech Republc [14] Zhang Y., Zhang C.: New Algorthm for Character Segmentaton of Lcense Plate, Intellgent Vehcles Symposum, IEEE, 2003 [15] ANPR-tutoral.com, Quercus technologes,