BRNO UNIVERSITY OF TECHNOLOGY

Transcription

1 BRNO UNIVERSITY OF TECHNOLOGY FACULTY OF INFORMATION TECHNOLOGY DEPARTMENT OF INTELLIGENT SYSTEMS ALGORITHMIC AND MATHEMATICAL PRINCIPLES OF AUTOMATIC NUMBER PLATE RECOGNITION SYSTEMS B.SC. THESIS AUTHOR ONDREJ MARTINSKY BRNO 2007

2 Copyrght 2007 Ondrej Martnsky The author s ndebted to the supervsor of ths thess, doc. Ing. Frantšek Zbořl, CSc. for hs great help. THIS WORK IS A PART OF THE RESEARCH PLAN "SECURITY-ORIENTED RESEARCH IN INFORMATION TECHNOLOGY, MSM " AT BRNO UNIVERSITY OF TECHNOLOGY Lcensed under the terms of Creatve Commons Lcense, Attrbuton-NonCommercal-NoDervs 2.5. You are free to copy, dstrbute and transmt ths work under the followng condtons. You must attrbute the work n the manner specfed by the author or lcensor (but not n any way that suggests that they endorse you or your use of the work). You may not use ths work for commercal purposes. For further nformaton, please read the full legal code at

3 Abstract Ths work deals wth problematc from feld of artfcal ntellgence, machne vson and neural networks n constructon of an automatc number plate recognton system (ANPR). Ths problematc ncludes mathematcal prncples and algorthms, whch ensure a process of number plate detecton, processes of proper characters segmentaton, normalzaton and recognton. Work comparatvely deals wth methods achevng nvarance of systems towards mage skew, translatons and varous lght condtons durng the capture. Work also contans an mplementaton of a demonstraton model, whch s able to proceed these functons over a set of snapshots. Key Words: Machne vson, artfcal ntellgence, neural networks, optcal character recognton, ANPR

4 Contents 1 Introducton ANPR systems as a practcal applcaton of artfcal ntellgence Mathematcal aspects of number plate recognton systems Physcal aspects of number plate recognton systems Notatons and mathematcal symbols 3 2 Prncples of number plate area detecton Edge detecton and rank flterng Convoluton matrces Horzontal and vertcal mage projecton Double-phase statstcal mage analyss Vertcal detecton - band clppng Horzontal detecton - plate clppng Heurstc analyss and prorty selecton of number plate canddates Prorty selecton and basc heurstc analyss of bands Deeper analyss Deskewng mechansm Detecton of skew Correcton of skew 18 3 Prncples of plate segmentaton Segmentaton of plate usng a horzontal projecton Etracton of characters from horzontal segments Pece etracton Heurstc analyss of peces 23 4 Feature etracton and normalzaton of characters Normalzaton of brghtness and contrast Hstogram normalzaton Global thresholdng Adaptve thresholdng Normalzaton of dmensons and resamplng Nearest-neghbor downsamplng Weghted-average downsamplng Feature etracton Pel matr Detecton of character edges Skeletonzaton and structural analyss 35 5 Recognton of characters General classfcaton problem Bologcal neuron and ts mathematcal models 43 v

5 5.2.1 McCulloch-Ptts bnary threshold neuron Percepton Feed-forward neural network Adaptaton mechansm of feed-forward neural network Actve phase Partal dervatves and gradent of error functon Adaptaton phase Heurstc analyss of characters 53 6 Syntactcal analyss of a recognzed plate Prncple and algorthms Recognzed character and ts cost Syntactcal patterns Choosng the rght pattern 57 7 Tests and fnal consderatons Choosng the representatve set of snapshots Evaluaton of a plate number correctness Bnary score Weghted score Results 61 Summary 62 Append A: Case study 63 Append B: Demo recognton software user s manual 73 Bblography 76 v

6 Lst of Fgures 1.1.a Illumnated number plate b Snapshot degraded by the sgnfcant moton blur Convoluton matr Varous rank and edge detecton flters Vertcal projecton of an mage nto a y as Double-phase plate clppng Vertcal projecton of the snapshot after convoluton wth a rank vector Band detected by an analyss of a vertcal projecton Horzontal projecton of the band and ts dervatve Wder area of the number plate after deskewng Prncple of a number plate postvty determnaton usng the color hstogram Dfference between the rotated and sheared number plate Illustraton of Hough transformaton Eample of the Hough transformaton Eample of a number plate before and after deskewng Eample of a number plate after applcaton of the adaptve thresholdng Pece etracton algorthm a Segmentaton phase nput b Segmentaton phase output Hstogram normalzaton by the Lagrange nterpolatng polynomal Partally shadowed number plate Chow and Kaneko approach of adaptve thresholdng Prncple of the downsamplng Nearest-neghbor and weghted-average downsamplng The pel matr feature etracton method Regon layouts n character btmap Possble types of 22 edges n character btmap The four-pel and eght-pel neghborhood Skeletonzaton algorthm Types of structural elements n character btmap Dfferent types of junctons n two nstances of the same character ab Combnaton of structural constrants and neural networks c Eample of the 913 upper-case alphabet Rectangular and polar coordnate systems n the character btmap General classfcaton problem Bologcal neuron a Parts of the bologcal neuron b Synaptc connectons between dendrtes and termnal buttons a Summaton and gan functon of the percepton b Sgmod saturaton functon Three layer feed-forward neural network Dependency of an error functon on a number of neurons Fndng a global mnmum n the error landscape Character segments after applcaton of the pece etracton algorthm Dfferent types of car snapshots 60 v

7 Lst of Tables 4.1 Structural constrants of characters Table of segment propertes related to the fgure Recognton rates of the ANPR system 61 v

8 Chapter 1 Introducton 1.1 ANPR systems as a practcal applcaton of artfcal ntellgence Massve ntegraton of nformaton technologes nto all aspects of modern lfe caused demand for processng vehcles as conceptual resources n nformaton systems. Because a standalone nformaton system wthout any data has no sense, there was also a need to transform nformaton about vehcles between the realty and nformaton systems. Ths can be acheved by a human agent, or by specal ntellgent equpment whch s be able to recognze vehcles by ther number plates n a real envronment and reflect t nto conceptual resources. Because of ths, varous recognton technques have been developed and number plate recognton systems are today used n varous traffc and securty applcatons, such as parkng, access and border control, or trackng of stolen cars. In parkng, number plates are used to calculate duraton of the parkng. When a vehcle enters an nput gate, number plate s automatcally recognzed and stored n database. When a vehcle later ets the parkng area through an output gate, number plate s recognzed agan and pared wth the frst-one stored n the database. The dfference n tme s used to calculate the parkng fee. Automatc number plate recognton systems can be used n access control. For eample, ths technology s used n many companes to grant access only to vehcles of authorzed personnel. In some countres, ANPR systems nstalled on country borders automatcally detect and montor border crossngs. Each vehcle can be regstered n a central database and compared to a black lst of stolen vehcles. In traffc control, vehcles can be drected to dfferent lanes for a better congeston control n busy urban communcatons durng the rush hours. 1.2 Mathematcal aspects of number plate recognton systems In most cases, vehcles are dentfed by ther number plates, whch are easly readable for humans, but not for machnes. For machne, a number plate s only a grey pcture defned as a two-dmensonal functon f (, y), where and y are spatal coordnates, and f s a lght ntensty at that pont. Because of ths, t s necessary to desgn robust mathematcal machnery, whch wll be able to etract semantcs from spatal doman of the captured mage. These functons are mplemented n so-called ANPR systems, where the acronym ANPR stands for an Automatc Number Plate Recognton. ANPR system means transformaton of data between the real envronment and nformaton systems. The desgn of ANPR systems s a feld of research n artfcal ntellgence, machne vson, pattern recognton and neural networks. Because of ths, the man goal of ths thess s to study algorthmc and mathematcal prncples of automatc number plate recognton systems. Chapter two deals wth problematc of number plate area detecton. Ths problematc ncludes algorthms, whch are able to detect a rectangular area of the number plate n orgnal mage. Humans defne the number plate n a natural language as a small plastc or metal plate attached to a vehcle for offcal dentfcaton purposes, but machnes do not understand ths defnton. Because of ths, there s a need to fnd an alternatve defnton of the number plate based on descrptors, whch wll be comprehensble for machnes. Ths s a fundamental problem of machne vson and of ths chapter. Chapter three descrbes prncples of the character segmentaton. In most cases, characters are segmented usng the horzontal projecton of a pre-processed number plate, but sometmes 1

9 these prncples can fal, especally f detected number plates are too warped or skewed. Then, more sophstcated segmentaton algorthms must be used. Chapter four deals wth varous methods normalzaton and detecton of characters. At frst, character dmensons and brghtness must be normalzed to ensure nvarance towards a sze and lght condtons. Then, a feature etracton algorthm must be appled on a character to flter rrelevant data. It s necessary to etract features, whch wll be nvarant towards character deformatons, used font style etc. Chapter fve studes pattern classfers and neural networks and deals wth ther usage n recognton of characters. Characters can be classfed and recognzed by the smple nearest neghbor algorthm (1NN) appled to a vector of etracted features, or there s also possblty to use one of the more sophstcated classfcaton methods, such as feed-forward or Hopfeld neural networks. Ths chapter also presents addtonal heurstc analyses, whch are used for elmnaton of non-character elements from the plate. Sometmes, the recognton process may fal and the detected plate can contan errors. Some of these errors can be detected by a syntactcal analyss of the recognzed plate. If we have a regular epresson, or a rule how to evaluate a country-specfc lcense plate, we can reconstruct defectve plates usng ths rule. For eample, a number zero 0 can be automatcally repared to a character O on postons, where numbers are not allowed. Chapter s deals wth ths problematc. 1.3 Physcal aspects of number plate recognton systems Automatc number plate recognton system s a specal set of hardware and software components that proceeds an nput graphcal sgnal lke statc pctures or vdeo sequences, and recognzes lcense plate characters from t. A hardware part of the ANPR system typcally conssts of a camera, mage processor, camera trgger, communcaton and storage unt. The hardware trgger physcally controls a sensor drectly nstalled n a lane. Whenever the sensor detects a vehcle n a proper dstance of camera, t actvates a recognton mechansm. Alternatve to ths soluton s a software detecton of an ncomng vehcle, or contnual processng of the sampled vdeo sgnal. Software detecton, or contnual vdeo processng may consume more system resources, but t does not need addtonal hardware equpment, lke the hardware trgger. Image processor recognzes statc snapshots captured by the camera, and returns a tet representaton of the detected lcense plate. ANPR unts can have own dedcated mage processors (all-n-one soluton), or they can send captured data to a central processng unt for further processng (generc ANPR). The mage processor s runnng on specal recognton software, whch s a key part of whole ANPR system. Because one of the felds of applcaton s a usage on road lanes, t s necessary to use a specal camera wth the etremely short shutter. Otherwse, qualty of captured snapshots wll be degraded by an undesred moton blur effect caused by a movement of the vehcle. For eample, usage of the standard camera wth shutter of 1/100 sec to capture a vehcle wth speed of 80 km/h wll cause a moton skew n amount of 0.22 m. Ths skew means the sgnfcant degradaton of recognton abltes. There s also a need to ensure system nvarance towards the lght condtons. Normal camera should not be used for capturng snapshots n darkness or nght, because t operates n a vsble lght spectrum. Automatc number plate recognton systems are often based on cameras operatng n an nfrared band of the lght spectrum. Usage of the nfrared camera n combnaton wth an nfrared llumnaton s better to acheve ths goal. Under the llumnaton, plates that are made from refleve materal are much more hghlghted than rest of the mage. Ths fact makes detecton of lcense plates much easer. 2

10 Fgure 1.1: (a) Illumnaton makes detecton of refleve mage plates easer. (b) Long camera shutter and a movement of the vehcle can cause an undesred moton blur effect. 1.4 Notatons and mathematcal symbols Logc symbols p q Eclusve logcal dsjuncton ( p or q ) p q Logcal conjuncton ( p and q ) p q Logcal dsjuncton ( p or q ) p Ecluson ( not p ) Mathematcal defnton of mage f (, y ) and y are spatal coordnates of an mage, and f s an ntensty of lght at that pont. Ths functon s always dscrete on dgtal computers. N 0 y N 0, where N 0 denotes the set of natural numbers ncludng zero. f ( p ) The ntensty of lght at pont p. f ( p) = f (, y), where p = [, y] Pel neghborhoods p1n ɺɺ 4 p2 Pel p 1 s n a four-pel neghborhood of pel p 2 (and vce versa) p1n ɺɺ 8 p2 Pel p 1 s n an eght-pel neghborhood of pel p 2 (and vce versa) Convolutons a( ) b( ) Dscrete convoluton of sgnals a( ) and b( ) ɶ b Dscrete perodcal convoluton of sgnals a( ) and b( ) a( ) ( ) 3

11 Vectors and sets [, y] m The element n th column and y th row of matr m. ma A mn A mean A medan A A The mamum value contaned n the set A. The scope of elements can be specfed by addtonal condtons The mnmum value contaned n the set A The mean value of the elements contaned n the set A The medan value of the elements contaned n the set A The cardnalty of the set A. (Number of elements contaned n the set) Vectors or any other ordered sequences of numbers are prnted bold. The elements of vectors are denoted as, where s a sequence number (startng wth zero), such as 0 n 1, where n = s a cardnalty of the vector (number of elements) The element a of the vector. For eample, the vector can contan [ a] ( ) elements a, b, c, d, such as = ( a, b, c, d ) If there s more than one vector denoted as, they are dstngushed by ther ndees. The upper nde ( ) does not mean the th element of vector. Intervals a < < b les n the nterval between a and b. Ths notaton s used when s the spatal coordnate n mage (dscrete as well as contnuous) a b Quantfcators Ths notaton has the same meanng as the above one, but t s used when s a dscrete sequence number. There ests at least one! There ests eactly one n There ests eactly n There does not est For every Roundng Number rounded down to the nearest nteger Number rounded up to the nearest nteger 4

12 Chapter 2 Prncples of number plate area detecton The frst step n a process of automatc number plate recognton s a detecton of a number plate area. Ths problematc ncludes algorthms that are able to detect a rectangular area of the number plate n an orgnal mage. Humans defne a number plate n a natural language as a small plastc or metal plate attached to a vehcle for offcal dentfcaton purposes, but machnes do not understand ths defnton as well as they do not understand what vehcle, road, or whatever else s. Because of ths, there s a need to fnd an alternatve defnton of a number plate based on descrptors that wll be comprehensble for machnes. Let us defne the number plate as a rectangular area wth ncreased occurrence of horzontal and vertcal edges. The hgh densty of horzontal and vertcal edges on a small area s n many cases caused by contrast characters of a number plate, but not n every case. Ths process can sometmes detect a wrong area that does not correspond to a number plate. Because of ths, we often detect several canddates for the plate by ths algorthm, and then we choose the best one by a further heurstc analyss. f, y, where and y are spatal Let an nput snapshot be defned by a functon ( ) coordnates, and f s an ntensty of lght at that pont. Ths functon s always dscrete on dgtal computers, such as N 0 y N 0, where N 0 denotes the set of natural numbers ncludng zero. We defne operatons such as edge detecton or rank flterng as mathematcal transformatons of functon f. The detecton of a number plate area conssts of a seres of convolve operatons. Modfed snapshot s then projected nto aes and y. These projectons are used to determne an area of a number plate. 2.1 Edge detecton and rank flterng We can use a perodcal convoluton of the functon f wth specfc types of matrces m to detect varous types of edges n an mage: w 1 h 1 (, ) = (, ) ɶ m[, ] = (, ) m mod w ( ),mod h ( ) f y f y y f y y j = 0 j= 0 where w and h are dmensons of the mage represented by the functon f Note: The epresson [, y] m represents the element n th column and y th row of matr m Convoluton matrces Each mage operaton (or flter) s defned by a convoluton matr. The convoluton matr defnes how the specfc pel s affected by neghborng pels n the process of convoluton. 5

13 Indvdual cells n the matr represent the neghbors related to the pel stuated n the centre of the matr. The pel represented by the cell y n the destnaton mage (fg. 2.1) s affected by the pels 0 8 accordng to the formula: y = 0 m0 + 1 m1 + 2 m2 + 3 m3 + 4 m4 + 5 m5 + 6 m6 + 7 m7 + 8 m8 Fgure 2.1: The pel s affected by ts neghbors accordng to the convoluton matr. Horzontal and vertcal edge detecton To detect horzontal and vertcal edges, we convolve source mage wth matrces m he and m ve. The convoluton matrces are usually much smaller than the actual mage. Also, we can use bgger matrces to detect rougher edges m he = ; m ve = Sobel edge detector The Sobel edge detector uses a par of 33 convoluton matrces. The frst s dedcated for evaluaton of vertcal edges, and the second for evaluaton of horzontal edges G = ; G y = The magntude of the affected pel s then calculated usng the formula pras, t s faster to calculate only an appromate magntude as G = G + G y. 2 2 y G = G + G. In Horzontal and vertcal rank flterng Horzontally and vertcally orented rank flters are often used to detect clusters of hgh densty of brght edges n the area of the number plate. The wdth of the horzontally orented rank flter matr s much larger than the heght of the matr ( w h ), and vce versa for the vertcal rank flter ( w h ). To preserve the global ntensty of an mage, t s necessary to each pel be replaced wth an average pel ntensty n the area covered by the rank flter matr. In general, the convoluton matr should meet the followng condton: 6

14 w 1 h 1 m = 0 j= 0 hr [ j], = 1.0 where w and h are dmensons of the matr. The followng pctures show the results of applcaton of the rank and edge detecton flters. Fgure 2.2: (a) Orgnal mage (b) Horzontal rank flter (c) Vertcal rank flter (d) Sobel edge detecton (e) Horzontal edge detecton (f) Vertcal edge detecton 2.2 Horzontal and vertcal mage projecton After the seres of convoluton operatons, we can detect an area of the number plate accordng to a statstcs of the snapshot. There are varous methods of statstcal analyss. One of them s a horzontal and vertcal projecton of an mage nto the aes and y. The vertcal projecton of the mage s a graph, whch represents an overall magntude of the mage accordng to the as y (see fgure 2.3). If we compute the vertcal projecton of the mage after the applcaton of the vertcal edge detecton flter, the magntude of certan pont represents the occurrence of vertcal edges at that pont. Then, the vertcal projecton of so transformed mage can be used for a vertcal localzaton of the number plate. The horzontal projecton represents an overall magntude of the mage mapped to the as. 7

15 Fgure 2.3: Vertcal projecton of mage to a y as Let an nput mage be defned by a dscrete functon f (, y ). Then, a vertcal projecton p y of the functon f at a pont y s a summary of all pel magntudes n the y th row of the nput mage. Smlarly, a horzontal projecton at a pont of that functon s a summary of all magntudes n the th column. We can mathematcally defne the horzontal and vertcal projecton as: h 1 w 1 ( ) = (, ) ; py ( y) = f (, y) p f j j= 0 where w and h are dmensons of the mage. = Double-phase statstcal mage analyss The statstcal mage analyss conssts of two phases. The frst phase covers the detecton of a wder area of the number plate. Ths area s then deskewed, and processed n the second phase of analyss. The output of double-phase analyss s an eact area of the number plate. These two phases are based on the same prncple, but there are dfferences n coeffcents, whch are used to determne boundares of clpped areas. The detecton of the number plate area conssts of a band clppng and a plate clppng. The band clppng s an operaton, whch s used to detect and clp the vertcal area of the number plate (so-called band) by analyss of the vertcal projecton of the snapshot. The plate clppng s a consequent operaton, whch s used to detect and clp the plate from the band (not from the whole snapshot) by a horzontal analyss of such band. Snapshot Assume the snapshot s represented by a functon (, ) The [, y ] represents the upper left corner of the snapshot, and [, ] 0 0 f y, where 0 1 and y0 y y1. y represents the bottom rght corner. If w and h are dmensons of the snapshot, then 0 = 0, y 0 = 0, 1 = w 1 and y =. 1 h

16 Band =, such as: The band b n the snapshot f s an arbtrary rectangle b (, y,, y ) Plate b0 b0 b1 b1 ( = ) ( = ) ( y y < y y ) b0 mn b1 ma mn b0 b1 ma =, such as: Smlarly, the plate p n the band b s an arbtrary rectangle p ( p0, y p0, p 1, y p1) ( b0 p0 p 1 b 1) ( y p0 = yb0 ) ( y p0 = yb0 ) The band can be also defned as a vertcal selecton of the snapshot, and the plate as a horzontal selecton of the band. The fgure 2.4 schematcally demonstrates ths concept: y b0 p0 p 1 y b1 Fgure 2.4: The double-phase plate clppng. Black color represents the frst phase of plate clppng, and red color represents the second one. Bands are represented by dashed lnes, and plates by sold lnes Vertcal detecton band clppng The frst and second phase of band clppng s based on the same prncple. The band clppng s a vertcal selecton of the snapshot accordng to the analyss of a graph of vertcal projecton. If r p y contans h h s the heght of the analyzed mage, the correspondng vertcal projecton ( ) values, such as y 0; h 1. The graph of projecton may be sometmes too ragged for analyss due to a bg statstcal r p y. There are two approaches how to solve ths problem. We can blur dsperson of values ( ) y the source snapshot (costly soluton), or we can decrease the statstcal dsperson of the ragged r projecton p y by convolvng ts projecton wth a rank vector: r ( ) = ( ) m ɶ [ ] p y p y y y y hr where m hr s the rank vector (analogous to the horzontal rank matr n secton 2.1.1). The wdth of the vector m hr s nne n default confguraton. After convoluton wth the rank vector, the vertcal projecton of the snapshot n fgure 2.3 can look lke ths: y 9

17 py 100% ( y) yb0 ybm y b 1 0% y 0 y 1 y Fgure 2.5: The vertcal projecton of the snapshot 2.3 after convoluton wth a rank vector. The fgure contans three detected canddates. Each hghlghted area corresponds to one detected band. The fundamental problem of analyss s to compute peaks n the graph of vertcal projecton. The peaks correspond to the bands wth possble canddates for number plates. The mamum p y correspondng to the ale of band can be computed as: value of ( ) y y0 y y1 { y ( )} y = arg ma p y The y b0 and y b1 are coordnates of band, whch can be detected as: bm { ( ) ( )} { ( ) ( )} y = ma y p y c p y b0 y y y bm y0 y ybm y = mn y p y c p y b1 y y y bm ybm y y1 c y s a constant, whch s used to determne the foot of peak y bm. In pras, the constant s calbrated to c 1 = 0.55 for the frst phase of detecton, and c 2 = 0.42 for the second phase. Fgure 2.6: The band detected by the analyss of vertcal projecton Ths prncple s appled teratvely to detect several possble bands. The y b0 and y b1 coordnates are computed n each step of teratve process. After the detecton, values of projecton p y n nterval yb0, y b1 are zerozed. Ths dea s llustrated by the followng pseudo-code: let L to be a lst of detected canddates for :=0 to number_of_bands_to_be_detected do begn detect y b0 and y b1 by analyss of projecton save y b0 and y b1 to a lst L zeroze nterval yb0, y b1 end p y The lst L of coordnates y b0 and y b1 wll be sorted accordng to value of peak ( y bm ). The band clppng s followed by an operaton, whch detects plates n a band. 10

18 2.3.2 Horzontal detecton plate clppng In contrast wth the band clppng, there s a dfference between the frst and second phase of plate clppng. Frst phase There s a strong analogy n a prncple between the band and plate clppng. The plate clppng s based on a horzontal projecton of band. At frst, the band must be processed by a vertcal detecton flter. If w s a wdth of the band (or a wdth of the analyzed mage), the r p contans w values: correspondng horzontal projecton ( ) Please notce that p ( ) y b1 ( ) (, ) p = f j j= yb 0 s a projecton of the band, not of the whole mage. Ths can be acheved by a summaton n nterval yb0, y b1, whch represents the vertcal boundares of the r p may have a bg statstcal dsperson, we decrease band. Snce the horzontal projecton ( ) t by convolvng wth a rank vector ( p ( ) = p r ( ) ɶ m [ ] vr usually equal to a half of an estmated wdth of the number plate. Then, the mamum value correspondng to the plate can be computed as: 0 y 1 { ( )} = arg ma p bm The b0 and b1 are coordnates of the plate, whch can be then detected as: { ( ) ( )} { ( ) ( )} = ma p c p b0 bm 0 bm = mn p c p b1 bm bm 1 where c s a constant, whch s used to determne the foot of peak calbrated to c = 0.86 for the frst phase of detecton. Second phase ). The wdth of the rank vector s bm. The constant s In the second phase of detecton, the horzontal poston of a number plate s detected n another way. Due to the skew correcton between the frst and second phase of analyss, the wder plate f, y be a correspondng functon of such area must be duplcated nto a new btmap. Let ( ) ( ) btmap. Ths pcture has a new coordnate system, such as [0,0] represents the upper left corner and [ w 1, h 1] the bottom rght, where w and h are dmensons of the area. The wder area of the number plate after deskewng s llustrated n fgure 2.8. In contrast wth the frst phase of detecton, the source plate has not been processed by the vertcal detecton flter. If we assume that plate s whte wth black borders, we can detect that borders as black-to-whte and whte-to-black transtons n the plate. The horzontal projecton p of the mage s llustrated n the fgure 2.7.a. To detect the black-to-whte and whte-toblack transtons, there s a need to compute a dervatve p ( ) of the projecton p ( ) n. Snce the projecton s not contnuous, the dervaton step cannot be an nfntely small number 11

19 ( h lm ). If we derve a dscrete functon, the dervaton step h must be an ntegral number 0 (for eample 4 h = ). Let the dervatve of p ( ) be defned as: ( ) p = ( ) ( ) p p h h Where h = 4. p 100% ( ) 0% p ( ) w 1 0 w 1 Fgure 2.7: (a) The horzontal projecton p ( ) of p ( ) of the plate n fgure 2.8. (b) The dervatve. Arrows denote the BW and WB transtons, whch are used to determne the boundares of the plate. Fgure 2.8: The wder area of the number plate after deskewng. The left and rght boundary of the plate can be determned by an analyss of the projecton p ( ). The left corner p0 s represented by the black-to-whte transton (postve peak n fgure 2.7.b), and rght corner p1 by the whte-to-black transton (negatve peak n fgure 2.7.b): { ( ) ma { ( )}} = mn p c p p0 w d 0 < 0 < w 2 { ( ) mn { ( )}} = ma p c p p1 w d < w 0 < w 2 where c d s a constant used to determne the most left negatve and the most rght postve peak. The left and rght corners must le on the opposte halves of the detected plate accordng to the w w constrants 0 < for p0, and < w for p

20 In ths phase of the recognton process, t s not possble to select a best canddate for a number plate. Ths can be done by a heurstc analyss of characters after the segmentaton. 2.4 Heurstc analyss and prorty selecton of number plate canddates In general, the captured snapshot can contan several number plate canddates. Because of ths, the detecton algorthm always clps several bands, and several plates from each band. There s a predefned value of mamum number of canddates, whch are detected by analyss of projectons. By default, ths value s equals to nne. There are several heurstcs, whch are used to determne the cost of selected canddates accordng to ther propertes. These heurstcs have been chosen ad hoc durng the practcal epermentatons. The recognton logc sorts canddates accordng to ther cost from the most sutable to the least sutable. Then, the most sutable canddate s eamned by a deeper heurstc analyss. The deeper analyss defntely accepts, or rejects the canddate. As there s a need to analyze ndvdual characters, ths type of analyss consumes bg amount of processor tme. The basc concept of analyss can be llustrated by the followng steps: 1. Detect possble number plate canddates. 2. Sort them accordng to ther cost (determned by a basc heurstcs). 3. Cut the frst plate from the lst wth the best cost. 4. Segment and analyze t by a deeper analyss (tme consumng). 5. If the deeper analyss refuses the plate, return to the step Prorty selecton and basc heurstc analyss of bands The basc analyss s used to evaluate the cost of canddates, and to sort them accordng to ths cost. There are several ndependent heurstcs, whch can be used to evaluate the cost α. The heurstcs can be used separately, or they can be combned together to compute an overall cost of canddate by a weghted sum: α = 0.15 α α α α Heurstcs Illustraton Descrpton α 1 = yb0 y The heght of band n pels. Bands b1 wth a lower heght wll be preferred. α 2 = 1 p p ( y ) ( ) y y The ( ) bm y bm y bm p y s a mamum value of peak of vertcal projecton of snapshot, whch corresponds to the processed band. Bands wth a hgher amount of vertcal edges wll be preferred. α = 3 y b1 py ( y) b0 y= yb0 1 y y b 1 Ths heurstcs s smlar to the prevous one, but t consders not only the value of the greatest peak, but a value of area under the graph between ponts y b0 and y b1. These ponts defne a vertcal poston of the evaluated band. 13

21 p0 p 1 α4 = 5 y y b0 b1 The proportons of the one-row number plates are smlar n the most countres. If we assume that wdth/heght rato of the plate s about fve, we can compare the measured rato wth the estmated one to evaluate the cost of the number plate Deeper analyss The deeper analyss determnes the valdty of a canddate for the number plate. Number plate canddates must be segmented nto the ndvdual characters to etract substantal features. The lst of canddates s teratvely processed untl the frst vald number plate s found. The canddate s consdered as a vald number plate, f t meets the requrements for valdty. Assume that plate p s segmented nto several characters p0 pn 1, where n s a number of characters. Let w be a wdth of th character (see fgure 2.9.a). Snce all segmented characters have roughly unform wdth, we can use a standard devaton of these values as a heurstcs: 1 1 n = 0 ( w w) β1 = n where w s an arthmetc average of character wdths n w n = 0 w =. If we assume that the number plate conssts of dark characters on a lght background, we can use a brghtness hstogram to determne f the canddate meets ths condton. Because some country-specfc plates are negatve, we can use the hstogram to deal wth ths type of plates (see fgure 2.9.b). Let ( ) b be a value of a darkest and lghtest pont. Then, ( ) ma H b be a brghtness hstogram, where b s a certan brghtness value. Let b mn and are equal to b. The plate s negatve when the heurstcs β 2 s negatve: bma bmd ( ) H ( b) β2 = H b b= bmd b= bmn bma bmn where b md s a mddle pont n the hstogram, such as bmd =. 2 H b s a count of pels, whose values 14

22 p0 p1 p2 p3 p4 p5 p6 p7 p8 p9 ( ) H b b mn w( p 2 ) bmd bma Pel numbers b Fgure 2.9: (a) The number plate must be segmented nto ndvdual characters for deeper heurstc analyss. (b) Brghtness hstogram of the number plate s used to determne the postvty of the number plate. 2.5 Deskewng mechansm The captured rectangular plate can be rotated and skewed n many ways due to the postonng of vehcle towards the camera. Snce the skew sgnfcantly degrades the recognton abltes, t s mportant to mplement addtonal mechansms, whch are able to detect and correct skewed plates. The fundamental problem of ths mechansm s to determne an angle, under whch the plate s skewed. Then, deskewng of so evaluated plate can be realzed by a trval affne transformaton. It s mportant to understand the dfference between the sheared and rotated rectangular plate. The number plate s an object n three-dmensonal space, whch s projected nto the twodmensonal snapshot durng the capture. The postonng of the object can sometmes cause the skew of angles and proportons. If the vertcal lne of plate v p s not dentcal to the vertcal lne of camera objectve v c, the plate may be sheared. If the vertcal lnes v p and v c are dentcal, but the as a p of plate s not parallel to the as of camera a c, the plate may be rotated. (see fgure 2.10) 15

23 a p a p a p v = v c p ac ap ac vp = vc a c v c = v p a a v = v p c p c v c a c v p a a v v p c p c Fgure 2.10: (a) Number plate captured under the rght angle (b) rotated plate (c) Sheared plate Detecton of skew Hough transform s a specal operaton, whch s used to etract features of a specfc shape wthn a pcture. The classcal Hough transform s used for the detecton of lnes. The Hough transform s wdely used for mscellaneous purposes n the problematc of machne vson, but I have used t to detect the skew of captured plate, and also to compute an angle of skew. It s mportant to know, that Hough transform does not dstngush between the concepts such as rotaton and shear. The Hough transform can be used only to compute an appromate angle of mage n a two-dmensonal doman. The mathematcal representaton of lne n the orthogonal coordnate system s an equaton y = a + b, where a s a slope and b s a y-as secton of so defned lne. Then, the lne s a set of all ponts [, y ], for whch ths equaton s vald. We know that the lne contans an nfnte number of ponts as well as there are an nfnte number of dfferent lnes, whch can cross a certan pont. The relaton between these two assertons s a basc dea of the Hough transform. The equaton y = a + b can be also wrtten as b = a + y, where and y are parameters. Then, the equaton defnes a set of all lnes ( a, b ), whch can cross the pont [, y ]. For each pont n the XY coordnate system, there s a lne n an AB coordnate system (so called Hough space ) y b [, y ] 0 0 m k l m l k b = 0 a + y0 a y n the Fgure 2.11: The XY and AB ( Hough space ) coordnate systems. Each pont [, ] XY coordnate system corresponds to one lne n the Hough space (red color). The are several ponts (marked as k, l, m ) n the Hough space, that correspond to the lnes n the XY coordnate system, whch can cross the pont.[, y ]

24 Let f (, y ) be a contnuous functon. For each pont [ a, b ] n Hough space, there s a lne n the XY coordnate system. We compute a magntude of pont [ a, b ] as a summary of all ponts n the XY space, whch le on the lne a + b. f, y s a dscrete functon, whch represents the snapshot wth defnte Assume that ( ) dmensons ( w h). To compute the Hough transform of the functon lke ths, t s necessary to normalze t nto a unfed coordnate system n the followng way: 2 = 1 ; w 2 y y = 1 h Although the space defned by a unfed coordnate system s always dscrete (floatng pont) on dgtal computers, we wll assume that t s contnuous. Generally, we can defne the Hough h a, b f, y n the unfed coordnate system as: transform ( ) of a contnuous functon ( ) 1 (, ) = (, + ) h a b f a b d 1 π 2 0 θ π 0 2 a y b b Fgure 2.12: (a) Number plate n the unfed XY coordnate system after applcaton of the horzontal edge detecton flter (b) Hough transform of the number plate n the θ B coordnate system (c) Colored Hough transform n the AB coordnate system. We use the Hough transform of certan mage to evaluate ts skew angle. You can see the colored Hough transform on the fgure 2.12.c. The pels wth a relatvely hgh value are marked by a red color. Each such pel corresponds to a long whte lne n the fgure 13.a. If we assume that the angle of such lnes determnes the overall angle, we can fnd the longest lne as: ( a, b ) = arg ma h ( a, b ) m m { } 0 1 a 0 b 1 To compute the angle of such a lne, there s a need to transform t back to the orgnal coordnate system: am 1 bm 1 am, bm = w, h 2 2 [ ] where w and h are dmensons of the evaluated mage. Then, the overall angle θ of mage can be computed as: θ = arctan ( a ) m 17

25 The more sophstcated soluton s to determne the angle from a horzontal projecton of the Hough transform h. Ths approach s much better because t covers all parallel lnes together, not only the longest one: where p ( a ) a ˆ aˆ 1 θ arctan w m = 2 ; aˆ = arg ma p ( a ) m { } a 1 1 a s a horzontal projecton of the Hough space, such as: a 1 = ( ) (, ) p a f a b db Correcton of skew The second step of a deskewng mechansm s a geometrc operaton over an mage f (, ) y. As the skew detecton based on Hough transform does not dstngush between the shear and rotaton, t s mportant to choose the proper deskewng operaton. In pras, plates are sheared n more cases than rotated. To correct the plate sheared by the angle θ, we use the affne transformaton to shear t by the negatve angle θ. For ths transformaton, we defne a transformaton matr A : where S and drecton of the Y-as. S y are shear factors. The ( θ ) 1 S y 0 1 tan 0 A = S 1 0 = S s always zero, because we shear the plate only n a Let P be a vector representng the certan pont, such as = [, y,1] coordnates of that pont. The new coordnates = [, y,1] be computed as: P s = s s s P where and y are P of that pont after the shearng can P A where A s a correspondng transformaton matr. Let the deskewed number plate be defned by a functon computed n the followng way: s T ( ) (, ) = [,,1] A [ 1,0,0 ] T,[,,1] A [ 0,1,0 ] f y f y y f s. The functon f s can be After the substtuton of the transformaton matr A : ( θ ) ( θ ) 1 tan tan 0 0 fs (, y) = f [, y,1] , [, y,1]

26 Fgure 2.13: (a) Orgnal number plate. (b) Number plate after deskewng. 19

27 Chapter 3 Prncples of plate segmentaton The net step after the detecton of the number plate area s a segmentaton of the plate. The segmentaton s one of the most mportant processes n the automatc number plate recognton, because all further steps rely on t. If the segmentaton fals, a character can be mproperly dvded nto two peces, or two characters can be mproperly merged together. We can use a horzontal projecton of a number plate for the segmentaton, or one of the more sophstcated methods, such as segmentaton usng the neural networks. If we assume only one-row plates, the segmentaton s a process of fndng horzontal boundares between characters. Secton 3.2 deals wth ths problematc. The second phase of the segmentaton s an enhancement of segments. The segment of a plate contans besdes the character also undesrable elements such as dots and stretches as well as redundant space on the sdes of character. There s a need to elmnate these elements and etract only the character. Secton 3.3 deals wth these problems. 3.1 Segmentaton of plate usng a horzontal projecton Snce the segmented plate s deskewed, we can segment t by detectng spaces n ts horzontal projecton. We often apply the adaptve thresholdng flter to enhance an area of the plate before segmentaton. The adaptve thresholdng s used to separate dark foreground from lght background wth non-unform llumnaton. You can see the number plate area after the thresholdng n fgure 3.1.a. After the thresholdng, we compute a horzontal projecton ( ) p of the plate f (, ) y. We use ths projecton to determne horzontal boundares between segmented characters. These boundares correspond to peaks n the graph of the horzontal projecton (fgure 3.1.b). y p ( ) v m v a Fgure 3.1: (a) Number plate after applcaton of the adaptve thresholdng (b) Horzontal projecton of plate wth detected peaks. Detected peaks are denoted by dotted vertcal lnes. v b 20

28 The goal of the segmentaton algorthm s to fnd peaks, whch correspond to the spaces between characters. At frst, there s a need to defne several mportant values n a graph of the p : horzontal projecton ( ) m v - The mamum value contaned n the horzontal projecton p ( ) v = ma p ( ), where w s a wdth of the plate n pels. m a b 0 < w { } v - The average value of horzontal projecton p ( ), such as v p ( ) a 1 1 w = w = 0, such as v - Ths value s used as a base for evaluaton of peak heght. The base value s always calculated as vb = 2 va vm. The v a must le on vertcal as between the values v b and v m. The algorthm of segmentaton teratvely fnds the mamum peak n the graph of vertcal projecton. The peak s treated as a space between characters, f t meets some addtonal condtons, such as heght of peak. The algorthm then zerozes the peak and teratvely repeats ths process untl no further space s found. Ths prncple can be llustrated by the followng steps: 1. Determne the nde of the mamum value of horzontal projecton: = arg ma p m 0 < w { ( )} 2. Detect the left and rght foot of the peak as: { ( ) ( )} { ( ) ( )} = ma p c p l m 0 m = mn p c p r < w m m 3. Zeroze the horzontal projecton ( ) 4. If p ( ) < c v, go to step 7. m w m 5. Dvde the plate horzontally n the pont m. 6. Go to step End. p on nterval, Two dfferent constants have been used n the algorthm above. The constant l r c s used to determne foots of peak m. The optmal value of c s 0.7. The constant c w determnes the mnmum heght of the peak related to the mamum value of the projecton ( v m ). If the heght of the peak s below ths mnmum, the peak wll not be consdered as a space between characters. It s mportant to choose a value of constant carefully. An nadequate small value causes that too many peaks wll be treated as spaces, and characters wll be mproperly dvded. A bg value of c w causes that not all regular peaks wll be treated as spaces, and characters wll be mproperly merged together. The optmal value of c w s To ensure a proper behavor of the algorthm, constants c and c w should meet the followng condton: (,, ) P : c v p ( ) p ( ) > l m r w m l r where P s a set of all detected peaks m wth correspondng foots l and r. c w 21

29 3.2 Etracton of characters from horzontal segments The segment of plate contans besdes the character also redundant space and other undesrable elements. We understand under the term segment the part of a number plate determned by a horzontal segmentaton algorthm. Snce the segment has been processed by an adaptve thresholdng flter, t contans only black and whte pels. The neghborng pels are grouped together nto larger peces, and one of them s a character. Our goal s to dvde the segment nto the several peces, and keep only one pece representng the regular character. Ths concept s llustrated n fgure 3.2. Horzontal segment Pece 1 Pece 2 Pece 3 Pece 4 Fgure 3.2: Horzontal segment of the number plate contans several groups (peces) of neghborng pels Pece etracton Let the segment be defned by a dscrete functon f (, ) such as [ 0,0 ] s an upper left corner of the segment, and [ w 1, h 1] where w and h are dmensons of the segment. The value of f (, ) pels, and 0 for the whte space. The pece Ρ s a set of all neghborng pels [, ] The pel [, y ] belongs to the pece Ρ f there s at least one pel [, y ] [, y ] and [, y ] are neghbors: [, y] Ρ [, y ] Ρ :[, y] N ɺɺ [, y ] y n the relatve coordnate system, s a bottom rght corner, y s 1 for the black y, whch represents a contnuous element. 4 from the Ρ, such as The notaton anɺɺ 4b means a bnary relaton a s a neghbor of b n a four-pel neghborhood : Algorthm [ ] ɺɺ [ ], y N, y = 1 y y = 1 4 The goal of the pece etracton algorthm s to fnd and etract peces from a segment of the plate. Ths algorthm s based on a smlar prncple as a commonly known seed-fll algorthm. 22

30 Let pece Ρ be a set of (neghborng) pels [, y ] Let S be a set of all peces Ρ from a processed segment defned by the functon f, y. ( ) {,, 1} Let X be a set of all black pels: X [ y] f ( y) Let A be an aulary set of pels = = Prncple of the algorthm s llustrated by the followng pseudo-code: let set S = 0/ { } let set X = [, y] f (, y) = 1 [ 0,0 ] [, y] < [ w, h] whle set X s not empty do begn let set Ρ = 0/ let set A = 0/ pull one pel from set X and nsert t nto set A whle set A s not empty do begn let [, y ] be a certan pel from A pull pel [, y ] from a set A f f (, y) 1 [, y] A [ 0,0 ] [, y] [ w, h] begn end end add Ρ to set S end = < then pull pel [, ] nsert pels [ 1, y],[ + 1, y],[, y 1],[, 1] y from set A and nsert t nto set Ρ y + nto set A Note 1: The operaton pull one pel from a set s non-determnstc, because a set s an unordered group of elements. In real mplementaton, a set wll be mplemented as an ordered lst, and the operaton pull one pel from a set wll be mplemented as pull the frst pel from a lst Note 2: The mathematcal concluson [ mn, ymn ] < [, y] < [ ma, yma ] means The pel [, y ] les n a rectangle defned by pels [, y ] and [, y ]. More formally: mn mn ma ma [, y]r[, y ] R yry where R s a one of the bnary relatons: <, >,, and = Heurstc analyss of peces The pece s a set of pels n the local coordnate system of the segment. The segment usually contans several peces. One of them represents the character and others represent redundant elements, whch should be elmnated. The goal of the heurstc analyss s to fnd a pece, whch represents character. Let us place the pece Ρ nto an magnary rectangle ( 0, y0, 1, y 1), where [ 0, y 0] s an upper left corner, and [ 1, y 1] s a bottom rght corner of the pece: 23

31 { } { } { } { } = mn [, y] Ρ y = mn y [, y] Ρ 0 0 = ma [, y] Ρ y = ma y [, y] Ρ 1 1 The dmensons and area of the magnary rectangle are defned as w = 0 1, h = y0 y1 and S = w h. Cardnalty of the set Ρ represents the number of black pels n b. The number of whte pels n w can be then computed as n w = S n b = w h Ρ. The overall magntude M of a pece s a rato between the number of black pels n b and the area S of an magnary rectangle M = n / S. b In pras, we use the number of whte pels n w as a heurstcs. Peces wth a hgher value of n w wll be preferred. The pece chosen by the heurstcs s then converted to a monochrome btmap mage. Each such mage corresponds to one horzontal segment. These mages are consdered as an output of the segmentaton phase of the ANPR process (see fgure 3.3) Fgure 3.3: The nput (a) and output (b) eample of the segmentaton phase of the ANPR recognton process. 24

32 Chapter 4 Feature etracton and normalzaton of characters To recognze a character from a btmap representaton, there s a need to etract feature descrptors of such btmap. As an etracton method sgnfcantly affects the qualty of whole OCR process, t s very mportant to etract features, whch wll be nvarant towards the varous lght condtons, used font type and deformatons of characters caused by a skew of the mage. The frst step s a normalzaton of a brghtness and contrast of processed mage segments. The characters contaned n the mage segments must be then reszed to unform dmensons (second step). After that, the feature etracton algorthm etracts approprate descrptors from the normalzed characters (thrd step). Ths chapter deals wth varous methods used n the process of normalzaton. 4.1 Normalzaton of brghtness and contrast The brghtness and contrast characterstcs of segmented characters are varyng due to dfferent lght condtons durng the capture. Because of ths, t s necessary to normalze them. There are many dfferent ways, but ths secton descrbes the three most used: hstogram normalzaton, global and adaptve thresholdng. Through the hstogram normalzaton, the ntenstes of character segments are redstrbuted on the hstogram to obtan the normalzed statstcs. Technques of the global and adaptve thresholdng are used to obtan monochrome representatons of processed character segments. The monochrome (or black & whte) representaton of mage s more approprate for analyss, because t defnes clear boundares of contaned characters Hstogram normalzaton The hstogram normalzaton s a method used to re-dstrbute ntenstes on the hstogram of the character segments. The areas of lower contrast wll gan a hgher contrast wthout affectng the global characterstc of mage. f, y. Let I be a total number Consder a grayscale mage defned by a dscrete functon ( ) of gray levels n the mage (for eample I = 256 ). We use a hstogram to determne the number of occurrences of each gray level, 0 I 1 : { } ( ) [, ] 0 0 (, ) H = y < w y < h f y = The mnmum, mamum and average value contaned n the hstogram s defned as: mn = mn { (, )} ; Hma = ma { f (, y) } ; Havg = f (, y) H f y 0 < w 0 y< h 0 < w 0 y< h 1 w h w 1 h 1 = 0 y= 0 25

33 where the values H mn, H ma and H avg are n the followng relaton: 0 H H H I 1 mn avg ma The goal of the hstogram normalzaton s to obtan an mage wth normalzed statstcal I characterstcs, such as H mn = 0, Hma = I 1, H avg =. To meet ths goal, we construct a 2 g as a Lagrange polynomal wth nterpolaton ponts transformaton functon ( ) [, y ] = [ H,0], [ ] 1 1 mn I 2, y2 = Havg, 2 and [ ] [ ] 3, y3 Hma, I 1 ( ) g = : 3 3 k = y j j= 1 k = 1 j k k j Ths transformaton functon can be eplctly wrtten as: g = y + y + y ( ) After substtuton of concrete ponts, and concrete number of gray levels I = 256 : H H H Havg H H H H H H H H mn ma ( ) g mn = I 1 avg mn avg ma ma mn ma avg g ( ) I 2 0 H mn Havg Hma brghtness before transformaton Fgure 4.1: We use the Lagrange nterpolatng polynomal as a transformaton functon to normalze the brghtness and contrast of characters. The Lagrange nterpolatng polynomal as a transformaton functon s a costly soluton. It s lke harvestng one potato by a tractor. In pras, there s more useful to construct the transformaton usng a smple lnear functon that spreads the nterval Hmn, H ma nto the unfed nterval 0, I 1 : 26

34 H H H mn ( ) = ( I 1) g ma The normalzaton of mage s proceeded by the transformaton functon n the followng way: n mn ( ) (, ) = (, ) f y g f y Global Thresholdng The global thresholdng s an operaton, when a contnuous gray scale of an mage s reduced nto monochrome black & whte colors accordng to the global threshold value. Let 0,1 be a gray scale of such mage. If a value of a certan pel s above the threshold t, the new value of the pel wll be zero. Otherwse, the new value wll be one for pels wth values above the threshold t. Let v be an orgnal value of the pel, such as v 0,1. The new value v s computed as: 0 f v 0, t v = 1 f v t,1 The threshold value t can be obtaned by usng a heurstc approach, based on a vsual nspecton of the hstogram. We use the followng algorthm to determne the value of t automatcally: 1. Select an ntal estmate for threshold t (for eample t = 0.5) 2. The threshold t dvdes the pels nto the two dfferent sets: [, ] (, ) { } S = y f y t. and [, ] (, ) b ) a { } S = y f y < t, 3. Compute the average gray level values µ a and µ b for the pels n sets S a and S b as: 1 1 µ a = f (, y) ; µ b S = f (, y) a [, y] S S a b [, y] Sb 1 4. Compute a new threshold value t = ( µ a + µ b ) 2 5. Repeat steps 2, 3, 4 untl the dfference t n successve teratons s smaller than predefned precson t p Snce the threshold t s global for a whole mage, the global thresholdng can sometmes fal. Fgure 4.2.a shows a partally shadowed number plate. If we compute the threshold t usng the algorthm above, all pels n a shadowed part wll be below ths threshold and all other pels wll be above ths threshold. Ths causes an undesred result llustrated n fgure 4.2.b. 27

35 H ( b) t Pel numbers b A B C Fgure 4.2: (a) The partally shadowed number plate. (b) The number plate after thresholdng. (c) The threshold value t determned by an analyss of the hstogram Adaptve thresholdng The number plate can be sometmes partally shadowed or nonunformly llumnated. Ths s most frequent reason why the global thresholdng fal. The adaptve thresholdng solves several dsadvantages of the global thresholdng, because t computes threshold value for each pel separately usng ts local neghborhood. Chow and Kaneko approach There are two approaches to fndng the threshold. The frst s the Chow and Kaneko approach, and the second s a local thresholdng. The both methods assumes that smaller rectangular regons are more lkely to have appromately unform llumnaton, more sutable for thresholdng. The mage s dvded nto unform rectangular areas wth sze of m n pels. The local hstogram s computed for each such area and a local threshold s determned. The threshold of concrete pont s then computed by nterpolatng the results of the submages. 1? Fgure 4.3: The number plate (from fgure 4.2) processed by the Chow and Kaneko approach of the adaptve thresholdng. The number plate s dvded nto the several areas, each wth own hstogram and threshold value. The threshold value of a concrete pel (denoted by ) s computed by nterpolatng the results of the submages (represented by pels 1-6). Local thresholdng The second way of fndng the local threshold of pel s a statstcal eamnaton of, y be a pel, for whch we compute the local threshold t. For neghborng pels. Let [ ] 28

36 smplcty we condder a square neghborhood wth wdth 2 r 1 [ r, y + r], [ + r, y r] and [ r, y r] approaches of computng the value of threshold: Mean of the neghborhood : t (, y) = mean f (, j) +, where [ r, y r], + + are corners of such square. There are severals r + r y r j y+ r { } { } Medan of the neghborhood : t (, y) = medan f (, j) r + r y r j y+ r Mean of the mnmum and mamum value of the heghborhood: 1 t (, y) = mn { f (, j) } + ma { f (, j) } 2 r + r r + r y r j y+ r y r j y+ r The new value f (, y) of pel [, ] y s then computes as: ( ) ( )) ( ) ( ) 0 f f, y 0, t, y f (, y) = 1 f f, y 0, t, y 4.2 Normalzaton of dmensons and resamplng Before etractng feature descrptors from a btmap representaton of a character, t s necessary to normalze t nto unfed dmensons. We understand under the term resamplng the process of changng dmensons of the character. As orgnal dmensons of unnormalzed characters are usually hgher than the normalzed ones, the characters are n most cases downsampled. When we downsample, we reduce nformaton contaned n the processed mage. There are several methods of resamplng, such as the pel-resze, blnear nterpolaton or the weghted-average resamplng. We cannot determne whch method s the best n general, because the successfulness of partcular method depends on many factors. For eample, usage of the weghed-average downsamplng n combnaton wth a detecton of character edges s not a good soluton, because ths type of downsamplng does not preserve sharp edges (dscussed later). Because of ths, the problematc of character resamplng s closely assocated wth the problematc of feature etracton. We wll assume that m n are dmensons of the orgnal mage, and m n are dmensons of the mage after resamplng. The horzontal and vertcal aspect rato s defned as r = m / m and r = n / n, respectvely. y Nearest-neghbor downsamplng The prncple of the nearest-neghbor downsampng s a pckng the nearest pel n the orgnal f, y be mage that corresponds to a processed pel n the mage after resamplng. Let ( ) a dscrete functon defnng the orgnal mage, such as 0 < m and 0 y < n. Then, the functon f (, y ) of the mage after resamplng s defned as: y f (, y ) = f, r r y 29

37 where 0 < m and 0 y < n. If the aspect rato s lower than one, then each pel n the resampled (destnaton) mage corresponds to a group of pels n the orgnal mage, but only one value from the group of source pels affects the value of the pel n the resampled mage. Ths fact causes a sgnfcant reducton of nformaton contaned n orgnal mage (see fgure 4.5). Fgure 4.4: One pel n the resampled mage corresponds to a group of pels n the orgnal mage Although the nearest neghbor downsampng sgnfcantly reduces nformaton contaned n the orgnal mage by gnorng a bg amount of pels, t preserves sharp edges and the strong bpolarty of black and whte pels. Because of ths, the nearest neghbor downsampng s sutable n combnaton wth the edge detecton feature etracton method descrbed n secton Weghed-average downsamplng In contrast wth the nearest-neghbor method, the weghted-average downsampng consders all pels from a correspondng group of pels n the orgnal mage. Let r and r y be a horzontal and vertcal aspect rato of the resampled mage. The value of n the destnaton mage s computed as a mean of source pels n the range the pel [, y ] [, y ] to [, ] mn where: mn y : ma ma 1 f, y = f, j ( ) ma ma mn j ymn ( ) ( ) ma mn yma y mn = = y ( ) mn = r y ; mn y = ry ; ma + 1 = r y ; ma y + 1 = ry 30

38 The weghted-average method of downsamplng does not preserve sharp edges of the mage (n contrast wth the prevous method). You can see the vsual comparson of these two methods n Fgure 4.5. n n n n m m Fgure 4.5: (a) Nearest-neghbor resamplng sgnfcantly reduces nformaton contaned n the orgnal mage, but t preserves sharp edges. (b) Weghted average resamplng gves a better vsual result, but the edges of the result are not sharp. m m 4.3 Feature etracton Informaton contaned n a btmap representaton of an mage s not sutable for processng by computers. Because of ths, there s need to descrbe a character n another way. The descrpton of the character should be nvarant towards the used font type, or deformatons caused by a skew. In addton, all nstances of the same character should have a smlar descrpton. A descrpton of the character s a vector of numeral values, so-called descrptors, or patterns : = ( ) 0,, n 1 Generally, the descrpton of an mage regon s based on ts nternal and eternal representaton. The nternal representaton of an mage s based on ts regonal propertes, such as color or teture. The eternal representaton s chosen when the prmary focus s on shape characterstcs. The descrpton of normalzed characters s based on ts eternal characterstcs because we deal only wth propertes such as character shape. Then, the vector of descrptors ncludes characterstcs such as number of lnes, bays, lakes, the amount of horzontal, vertcal and dagonal or dagonal edges, and etc. The feature etracton s a process of transformaton of data from a btmap representaton nto a form of descrptors, whch are more sutable for computers. If we assocate smlar nstances of the same character nto the classes, then the descrptors of characters from the same class should be geometrcally closed to each other n the vector space. Ths s a basc assumpton for successfulness of the pattern recognton process. Ths secton deals wth varous methods of feature etracton, and eplans whch method s the most sutable for a specfc type of character btmap. For eample, the edge detecton method should not be used n combnaton wth a blurred btmap Pel matr The smplest way to etract descrptors from a btmap mage s to assgn a brghtness of each pel wth a correspondng value n the vector of descrptors. Then, the length of such vector s equal to a square ( w h ) of the transformed btmap: 31

39 = f,modw ( ) w where 0,, w h 1. Bgger btmaps produce etremely long vector of descrptors, whch s not sutable for recognton. Because of ths, sze of such processed btmap s very lmted. In addton, ths method does not consder geometrcal closeness of pels, as well as ts neghborng relatons. Two slghtly based nstances of the same character n many cases produce very dfferent descrpton vectors. Even though, ths method s sutable f the character btmaps are too blurry or too small for edge detecton. h w 251, 181, 068, 041, 032, 071, 197, 196, 014, 132, 213, 187, 043, 041, 174, 011, 200, 254, 254, 232, 164, 202, 014, 012, 128, 242, 255, 255, = 253, 212, 089, 005, 064, 196, 253, 255, 255, 251, 196, 030, 009, 165, 127, 162, 251, 254, 197, 009, 105, 062, 005, 100, 144, 097, 006, 170, 207, 083, 032, 051, 053, 134, 250 Fgure 4.6: The pel matr feature etracton method Detecton of character edges In contrast wth the prevous method, the detecton of character edges does not consder absolute postonng of each pel, but only a number of occurrences of ndvdual edge types n a specfc regon of the character btmap. Because of ths, the resultng vector s nvarant towards the ntra-regonal dsplacement of the edges, and towards small deformatons of characters. Btmap regons Let the btmap be descrbed by a dscrete functon f (, ) y, where w and h are dmensons, such as 0 < w and 0 y < h. We dvde t nto s equal regons organzed to three rows and two columns n the followng way: ( ) ( ) Let mn, y mn and ( ) ( ) ma, y ma be an upper left and bottom rght pont of a rectangle, whch determnates the regon r, such as: (0) (0) (0) w (0) h Regon r 0 : mn = 0, y mn = 0, ma = 1 2, yma = 1 3 (1) w Regon r 1 : mn = 2, (1) (1) (1) h y mn = 0, ma = w 1, yma = 1 3 (2) (2) h Regon r 2 : mn = 0, ymn = 3, (2) w (2) 2 h ma = 1 2, yma = 1 3 (3) w Regon r 3 : mn = 2, (3) h ymn = 3, (3) (3) 2 h ma = w 1, yma =

40 (4) (4) 2 h Regon r 4 : mn = 0, ymn = 3, (4) w ma = 2, (4) yma = h 1 (5) w Regon r 5 : mn = 2, (5) 2 h ymn = 3, (5) (5) ma = w 1, yma = h 1 There are several ways how to dstrbute regons n the character btmap. The regons can be dsjunctve as well as they can overlap each other. The fgure 4.7 shows the several possble layouts of regons. Fgure 4.7: Layouts of regons n the character btmap. The regons can be dsjunctve as well as they can overlap each other. Edge types n regon Let us defne an edge of the character as a 22 whte-to-black transton n a btmap. Accordng to ths defnton, the btmap mage can contan fourteen dfferent edge types llustrated n fgure 4.8. Fgure 4.8: The processed btmap can contan dfferent types of 22 edges. The statstcs of occurrence of each edge type causes uselessly long vector of descrptors. Because of ths, the smlar types of edges are consdered as the same. The followng lsts shows how the edges can be grouped together: (vertcal edges) (horzontal edges) ( / -type dagonal edges) ( \ -type dagonal edges) (bottom rght corner) (bottom left corner) (top rght corner) (top left corner) For smplcty, assume that edge types are not grouped together. Let η be a number of dfferent edge types, where h s a 22 matr that corresponds to the specfc type of edge: h 0 = 1 0, h 1 = 0 1, h 2 = 0 0, h 3 = 1 1, h 4 = 0 1, h 5 = 1 0, h 6 = h 7 = 0 0, h 8 = 1 0, h 9 = 0 1, h 10 = 1 1, h 11 = 1 1, h 12 = 0 1, h 13 =

41 Let ρ be a number of rectangular regons n the character btmap, where ( ) ma ( ) mn, ( ) mn y, ( ) ma and y are boundares of the regon r ( 0 ρ 1 ). If the statstcs consder η dfferent edge types for each of ρ regons, the length of the resultng vector s computed as η ρ : = (,,, η ρ ) Feature etracton algorthm At frst, we have to embed the character btmap f (, ) y nto a bgger btmap wth whte paddng to ensure a proper behavor of the feature etracton algorthm. Let the paddng be one pel wde. Then, dmensons of the embeddng btmap wll be w + 2 and h + 2. The f, y s then defned as: embeddng btmap ( ) 1 f = 0 y = 0 = w + 1 y = h + 1 f (, y) = f y f y w y h ( 1, 1) ( = 0 = 0 = + 1 = + 1) where w and h are dmensons of character btmap before embeddng. Color of the paddng s whte (value of 1). The coordnates of pels are shfted one pel towards the orgnal poston. The structure of vector of output descrptors s llustrated by the pattern below. The notaton h r means number occurrences of an edge represented by the matr h j n the regon r. ( r0, r0,, η r0, r1, r1,, η r1, rρ 1, rρ 1,, η rρ 1 ) = h h h h h h h h h regon r regon 0 1 regon rρ 1 We compute the poston k of the r r n the vector as k = η + j, where η s the j number of dfferent edge types (and also the number of correspondng matrces). The followng algorthm demonstrates the computaton of the vector of descrptors : zeroze vector for each regon begn for each pel [, ] begn for each matr begn f begn r, where 0,, ρ 1 do y n regon j r,where ( ) ( ) h, where j 0,, η 1 (, ) ( + 1, ) (, + 1) ( + 1, + 1) f y f y f y f y mn do h j = then let k = η + j let k = k + 1 end end end end and y ( ) y y ( ) do ma mn ma 34

42 4.3.3 Skeletonzaton and structural analyss The feature etracton technques dscussed n the prevous two chapters are based on the statstcal mage processng. These methods do not consder structural aspects of analyzed mages. The small dfference n btmaps sometmes means a bg dfference n the structure of contaned characters. For eample, dgts 6 and 8 have very smlar btmaps, but there s a substantal dfference n ther structures. The structural analyss s based on hgher concepts than the edge detecton method. It does not deal wth terms such as pels or edges, but t consders more comple structures (lke junctons, lne ends or loops). To analyze these structures, we must nvolve the thnnng algorthm to get a skeleton of the character. Ths chapter deals wth the prncple of skeletonzaton as well as wth the prncple of structural analyss of skeletonzed mage. The concept of skeletonzaton The skeletonzaton s a reducton of the structural shape nto a graph. Ths reducton s accomplshed by obtanng a skeleton of the regon va the skeletonzaton algorthm. The skeleton of a shape s mathematcally defned as a medal as transformaton. To defne the medal as transformaton and skeletonzaton algorthm, we must ntroduce some elementary prerequste terms.,, y, such as anɺɺ b means a Let N ɺɺ be a bnary relaton between two pels [ y ] and [ ] s a neghbor of b. Ths relaton s defned as: [ ] ɺɺ 8 [ ] [ ] ɺɺ [ ], y N, y = 1 y y = 1 for eght-pel neghbourhood, y N, y = 1 y y = 1 for four-pel neghbourhood 4 The border B of character s a set of boundary pels. The pel [, y ] s a boundary pel, f t s black and f t has at least one whte neghbor n the eght-pel neghborhood: [, y] B f (, y) = 0 [, y ]: f (, y) = 1 [, y] N ɺɺ [, y ] The nner regon I of character s a set of black pels, whch are not boundary pels: [, ] (, ) 0 [, ] y I f y = y B 8 Fgure 4.9: (a) Illustraton of the four-pel and eght-pel neghborhood. (b) The set of boundary and nner pels of character. The pece Ρ s then a unon of all boundary and nner pels ( B I Ρ = ). Snce there s only one contnuous group of black pels, all black pels belong to the pece Ρ. The prncple and the related termnology of the skeletonzaton are smlar to the pece etracton algorthm dscussed n secton

43 Medal as transformaton The medal as transformaton of the pece Ρ defned as follows. For each nner pel p I, we fnd the closest boundary pel pb B. If a pel p has more than one such neghbor, t s sad to belong to the medal as (or skeleton) of the Ρ. The concept of the closest boundary pel depends on the defnton of the Eucldean dstance between two pels n the orthogonal coordnate system. Mathematcally, the medal as (or skeleton) S s a subset of the Ρ defned as: { } ( ) ( ) ( ) p S p p : p B p B d p, p = d p, p = mn d p, p The pel p belongs to the medal as S f there ests at least two pels p 1 and p 2, such as Eucldean dstance between pels p and p 1 s equal to the dstance between pels p and p 2, and these pels are closest boundary pels to pel p. p B The Eucldean dstance between two pels p = [, y ] and p [, y ] Skeletonzaton algorthm (, ) = 2 ( ) ( ) d p p = s defned as: Drect mplementaton of the mathematcal defnton of the medal as transformaton s computatonally epensve, because t nvolves calculatng the dstance from every nner pel from the set I to every pel on the boundary B. The medal as transformaton s ntutvely defned by a so-called fre front concept. Consder that a fre s lt along the border. All fre fronts wll advance nto the nner of character at the same speed. The skeleton of a character s then a set of pels reached by more than one fre front at the same tme. The skeletonzaton (or thnnng) algorthm s based on the fre front concept. The thnnng s a morphologcal operaton, whch preserves end-pels and does not break connectvty. Assume that pels of the pece are black (value of zero), and background pels are whte (value of one). The thnnng s an teratve process of two successve steps appled to boundary pels of a pece. Wth reference to the eght-pel neghborhood notaton n fgure 4.9, the frst step flags a boundary pel p for deleton f each of the followng condtons s satsfed: At least one of the top, rght and bottom neghbor of the pel p must be whte (the pel p s whte just when t does not belong to the pece Ρ ). t r b p Ρ p Ρ p Ρ At least one of the left, rght and bottom neghbor of pel p must be whte. l r b p Ρ p Ρ p Ρ The pel p must have at least two, and at most s black neghbors from the pece Ρ. Ths condton prevents the algorthm from erasng end-ponts and from breakng the connectvty. 36

44 { p pɺɺ 8 p p } 2 N Ρ 6 The number of whte-to-black transtons n the ordered sequence t tr r br b bl l tl t p, p, p, p, p, p, p, p, p must be equal to one. ( t Ρ tr Ρ ) + ( tr Ρ r Ρ ) + ( r Ρ br Ρ ) + ( br Ρ b Ρ) ( b Ρ bl Ρ ) + ( bl Ρ l Ρ ) + ( l Ρ tl Ρ ) + ( tl Ρ t Ρ ) = 1 v p p v p p v p p v p p v p p v p p v p p v p p ( ) v 0 = 1 f f The frst step flags pel p for deleton, f ts neghborhood meets the condtons above. However, the pel s not deleted untl all other pels have been processed. If at least one of the condtons s not satsfed, the value of pel p s not changed. After step one has been appled to all boundary pels, the flagged pels are defntely deleted n the second step. Every teraton of these two steps thns the processed character. Ths teratve process s appled untl no further pels are marked for deleton. The result of thnnng algorthm s a skeleton (or medal as) of the processed character. Let the pece Ρ be a set of all black pels contaned n skeletonzed character. Let B be a set of all boundary pels. The followng pseudo-code demonstrates the thnnng algorthm more formally. Ths algorthm proceeds the medal as transformaton over a pece Ρ. do // teratve thnnng process let contnue = false let B = 0/ for each pel p n pece Ρ do // create a set of boundary pels p : p Ρ pn ɺɺ p then // f the pel p has at least one whte neghbor nsert pel p nto set B // but keep t also n Ρ f 8 for each pel p n set B do // 1.step of the teraton begn // f at least one condton s volated, skp ths pel f t ( p r p b p ) f l ( p r p b p ) f { p p 8 p p } f Ρ Ρ Ρ then contnue Ρ Ρ Ρ then contnue ( 2 N 6 ) ɺɺ then contnue Ρ ( t Ρ tr Ρ ) + ( tr Ρ r Ρ ) + ( r Ρ br Ρ ) + ( br Ρ b Ρ) ( b Ρ bl Ρ ) + ( bl Ρ l Ρ ) + ( l Ρ tl Ρ ) + ( tl Ρ t Ρ) 1 v p p v p p v p p v p p v p p v p p v p p v p p then begn contnue end // all tests passed flag pont p for deleton let contnue = true 37

45 end for each pel p n set B do // 2.step of the teraton f p s flagged then pull pont p from pece Ρ whle contnue = true Note: The pel p belongs to the pece Ρ when t s black: p Ρ f ( p) = 0 Fgure 4.10: (a) The character btmap before skeletonzaton. (b) The thnnng algorthm teratvely deletes boundary pels. Pels deleted n the frst teraton are marked by a lght gray color. Pels deleted n the second and thrd teraton are marked by dark gray. (c) The result of the thnnng algorthm s a skeleton (or a medal as). Structural analyss of skeletonzed character The structural analyss s a feature etracton method that consders more comple structures than pels. The basc dea s that the substantal dfference between two compared characters cannot be evaluated by the statstcal analyss. Because of ths, the structural analyss etracts features, whch descrbe not pels or edges, but the more comple structures, such as junctons, lne ends and loops. Juncton The juncton s a pont, whch has at least three black neghbors n the eght-pel neghborhood. We consder only two types of junctons: the juncton of three and four lnes. The number of junctons n the skeletonzed pece Ρ s mathematcally defned as: Lne end { :{, } Nɺɺ 8 } { :{, } Nɺɺ 8 } 3 3 j n = p p p p Ρ p p 3 4 j n = p p p p Ρ p p The lne end s a pont, whch has eactly one neghbor n the eght-pel neghborhood. The number of lne-ends n a skeletonzed pece Ρ s defned as: e {! 1 : {, 1} N 8 1} n = p p p p Ρ p ɺɺ p The followng algorthm can be used to detect the number of junctons and number lne-ends n a skeletonzed pece Ρ : 38

46 let n j = 0 let n e = 0 for each pel p n pece Ρ do begn let neghbors = 0 end for each pel p n neghborhood { t, tr, r, br, b, bl, l, tl } f p Ρ then let neghbors = neghbors+1 f neghbors = 1 then let ne = ne +1 else f neghbors 3 then let n = n +1 j j p p p p p p p p do Fgure 4.11: (a, b) The juncton s a pel, whch as at least three neghbors n eghtpel neghborhood. (c) The lne end s a pel, whch has only one neghbor n eghtpel neghborhood (d) The loop s a group of pels, whch encloses the contnuous whte space. Loops It s not easy to determne the number of loops n l n the skeletonzed character. The algorthm s based on the followng prncple. At frst, we must negate the btmap of the skeletonzed character. Black pels wll be consdered as background and whte pels as foreground. The number of loops n the mage s equal to a number of lakes, whch are surrounded by these loops. Snce the lake s a contnuous group of whte pels n the postve mage, we apply the pece etracton algorthm on the negatve mage to determne the number of black peces. Then, the number of loops s equal to the number of black peces mnus one, because one pece represents the background of the orgnal mage (negated to the foreground). Another way s to use a seres of morphologcal erosons. Fgure 4.12: (a) We determne the number of lakes n skeleton by applyng the pece-etracton algorthm on negatve mage. The negatve mage (b) contans three peces. Snce the pece 3 s a background, only two peces are consdered as lakes. (c)(d) The smlar skeletons of the same character can dffer n the number of junctons 39

47 Snce we do not know the number of edges of the skeleton, we cannot use the standard cyclomatc equaton know from the graph theory. In addton, two smlar skeletons of the same character can sometmes dffer n a number of junctons (see fgure 4.12). Because of ths, t s not recommended to use constrants based on the number of junctons. Structural constrants To mprove the recognton process, we can assume structural constrants n the table 4.1. The syntactcal analyss can be combned by other methods descrbed n prevous chapters, such as edge detecton method or pel matr. The smplest way s to use one global neural network that returns several canddates and then select the best canddate that meets the structural constrants (fgure 4.13.a). More sophstcated soluton s to use the structural constrants for adaptve selecton of local neural networks (fgure 4.13.b). Lne ends Loops Junctons 0 BDO08 CEFGHIJKLMNSTUVWXYZ CDGIJLMNOSUVWZ PQ69 ADOPQR09 EFKPQTXY469 2 ACGIJLMNRSUVWZ B8 ABHR8 3 EFTY 4 HKX Table 4.1: Structural constrants of characters. A B C Fgure 4.13: (a, b) Structural constrants can be appled before and after the recognton by the neural network. (c) Eample of the skeletonzed alphabet. Feature etracton In case we know the poston of structural elements, we can form a vector of descrptors drectly from ths nformaton. Assume that there are several lne-ends, loops, and junctons n the 40