FEATURE BASED RECOGNITION OF TRAFFIC VIDEO STREAMS FOR ONLINE ROUTE TRACING

FEATURE BASED RECOGNITION OF TRAFFIC VIDEO STREAMS FOR ONLINE ROUTE TRACING Christoph Busch, Ralf Dörer, Christia Freytag, Heike Ziegler Frauhofer Istitute for Computer Graphics, Computer Graphics Ceter Darmstadt, Germay Email: {busch, doerer, freytag, heike}@igd.fhg.de Abstract I this paper a approach is preseted for the acquisitio of route related data by meas of recogizig vehicles that pass differet survey poits. This recogitio approach essetially relies o vehicle licese plate matrices that are provided by stadard video cameras. A recogitio pipelie is coceived where the mai steps are a frame selectio method, a segmetatio techique for the localizatio of plate matrices, a feature extractio method ad a recogitio process. The sigle steps are discussed i detail ad recogitio rates that have bee achieved i a field trial are preseted. Keywords: Computer Visio, motio image aalysis, feature extractio, object recogitio, correspodig problem, traffic aalysis, route tracig I. INTRODUCTION Due to upcomig low cost hardware ad the progress i algorithmic research, Computer Visio has become a promisig base techology for traffic sesorig systems. Various research projects [,5] poited out advatages of video based techology over covetioal systems. Amog these advatages are portability, ease of istallatio, reliability, log lifetime ad low cost. These advatages are stadard features i itegrated video based traffic aalysis systems [3,6]. Aother major ew feature is the acquisitio of route related data [4], which will be elaborated withi this paper. The term route related data describes the combied processig of data stemmig from differet survey poits. Likig the poits of recogitio to cotiuous routes, the traffic flow statistics ca be computed from the data acquired. This data represets the curret traffic situatio ad ca be optimally used for the cotrol of variable directio sigs. Fig. : Traffic situatio aroud a major city. The modified route directio towards lie A-C produces a traffic jam o lie C-B The importace of the traffic flow survey for a optimal trip distributio is illustrated i Figure. This figure represets the traffic situatio aroud may major cities. Assume dese traffic ruig o the log-distace route from A -B-C (Figure a). As soo as traffic icreases, local detectors o route segmet A-B (i Figure b) will register higher umbers of vehicles per time. Dyamic traffic cotrol systems would modify variable road sigs i a way that the assumed mai traffic flow would the ru o the route segmet A-C. This cotrol reactio is cotra productive if most of the vehicles are ruig from A towards B (see Figure c), sice superpositio of the diverted traffic with the dese traffic flow o lie segmet C-B will the lead to a complete traffic jam. The major subject of this paper will be a approach for the acquisitio of route related data by meas of recogizig vehicles that pass differet survey poits. This recogitio approach essetially relies o vehicle licese plate matrices that are provided by stadard video cameras. II. IMAGE PROCESSING PIPELINE As itroduced i Sectio I, the survey of et-liked traffic data relies o recogitio of sigle vehicles at differet poits. The recogitio is accomplished by usig features that characterize a vehicle. It ca be show that features extracted from wideagle video frames (showig the etire highway) such as vehicle class (va, car, motor bike), color ad predicted arrival time are ot sufficiet for reliable vehicle recogitio. The aalysis of licese plates as a feature cotributor is therefore cosidered i our approach. I this cotext, video recordigs which show oly sigle laes are used. This sectio gives a overview of how the video recordigs are processed i a image processig pipelie. A ordiary video stream of theoretically ifiite legth is used as iput data for the pipelie at each survey poit. A umber of frames are redudat sice they do ot cotai ay relevat iformatio (e.g. showig the pavemet ad o vehicle). Therefore, the task of the first step, the relevat frame selectio, is to filter frames of a video sequece that iclude a vehicle for further processig. The secod step, the segmetatio process, is performed i two substeps. I the first substep, the image area roughly cotaiig the licese plate is cut from the etire frame. Secodly, a fier segmetatio is executed i order to obtai the charac-

ter field. Due to our applicatio costraits we have developed a kowledge based solutio for licese plate segmetatio that relies o the detectio of horizotal ad vertical plate edges. The segmetatio of plate images is iflueced by plate properties ad local illumiatio coditios (e.g. shadows). I order to provide comparable iformatio vectors for further processig, a additioal segmetatio ad ormalizatio is required that elaborates the essetial data. Based o this data, step 3, feature extractio, derives features from the plate matrix. The feature extractio scheme is resposible for calculatig sigificat feature vectors that ca be used for compariso of two plate matrices. Fially, step 4, the recogitio process, is applied to the feature vectors. I order to cope with the requiremets of the parallel processig of vectors stemmig from differet survey poits the approach uses a et-orieted recogitio process. The data structure developed for recogitio purposes across differet survey poits must provide fast isertio ad retrieval of feature vectors. Therefore, a multidimesioal search tree has bee chose that splits the set of feature vectors ito smaller subsets regardig preselected features (such as color, etc.). To sum up, the four steps of our image processig pipelie are:. Relevat frame selectio. Plate segmetatio 3. Feature extractio 4. Recogitio The followig sectios discuss each process i more detail. III. RELEVANT FRAME SELECTION The iput video stream with a frame size of 5 x 5 ad a frame rate of 5 frames per secod is digitized usig a frame grabber card resultig i a high data iput rate of 6 MB per secod. Depedet o the traffic desity, a percetage of frames will show a empty traffic lie ad thus do ot provide ay iformatio. Therefore, the task of the relevat frame selectio process is to filter frames of a video sequece that iclude a vehicle for further processig. A special requiremet i the desig for a suitable algorithm is the ecessity for actio i real time, implyig that the processig time betwee cosecutive frames must be less tha 0.04 secods. This is the upper boudary for the process duratio. As a cosequece of this requiremet the umber of pixels ivolved i the decisio of whether a frame cotais a vehicle has to be reduced to a miimum. The reductio is achieved through the use of detectio stripes spread over the frame. Their localizatio i the frame is ot chaged while processig. Because the vehicles move maily vertically through the image, the detectio stripes are arraged horizotally o the frame. We foud pairs of stripes with sectios for the upper stripe ad 3 sectios for the lower stripe to be appropriate for this task (Fig. ). Fig. : Relevat frame selectio method: deactivatio of detectio stripe ad simultaeous activatio of stripe idicates a frame of iterest The state of a detectio stripe is importat for the relevat frame selectio. A detectio stripe is said to be active if a certai percetage of its pixel are covered by a movig object. I Figure, we see that certai detectio stripes are activated accordig to the positio of the car i the frame. I this example a frame is selected whe detectio stripe is active while detectio stripe becomes iactive. This state idicates a vehicle positio where the licese plate is approximately i the cetre of the frame. To fid out which areas are assumed to cotai a movig object, the backgroud-differece-techique [3] is applied with the modificatio that it is restricted to the pixels o the detectio stripes. We made further modificatios to the adaptatio of the backgroud image (stripes): The etire frame iformatio is simply trasferred to the backgroud if o detectio stripes are active. Cosiderig the umber of calculatios that are ecessary i processig oe pixel, it turs out that a maximum of four operatios are eeded. These operatios are visualized i the data flow graph i Figure 3, where s is the threshold which determies if a pixel is i a movig area. Fig. 3: Dataflow graph of pixel operatios IV. PLATE SEGMENTATION Feature aalysis of licese plates requires a importat preprocessig step: segmetatio withi frames of iterest. This is performed i two steps, as outlied i Figure 4. I the first step, the image area roughly cotaiig the licese plate is cut out of the etire frame (Sectio IV.A). Secodly, a fier segmetatio is executed i order to obtai the character field. This is the topic of Sectio IV.B.

Fig. 4: The segmetatio process as part of the processig pipelie A. Licese Plate Segmetatio A robust licese plate segmetatio is essetial for all further processig steps. Its fuctioality, amely the determiatio of licese plate positio, ca be foud i a variety of video based applicatios such as access cotrol systems, as well as kowledge based approaches which check image lies for whiteblack-white chagig of character edges. Plate segmets are idetified whe a group of image lies fulfills predefied feature thresholds. Aother approach is the adaptio of artificial eural etworks, which have prove their capabilities i a variety of applicatios i the segmetatio task. The commo disadvatages of segmetatio methods proposed so far are:. They allow a more geerous plate-to-frame-area relatio. The projectio of a licese plate ito a image produces a presetatio which is 9 times larger tha i our applicatio;. They do ot require real time solutios for their applicatio. Due to our applicatio costraits we have developed a kowledge based solutio that relies o the detectio of horizotal ad vertical plate edges. Accordigly, we apply a horizotal ad a vertical Sobel operator [] sice it is less sesitive to oise tha a Laplace filter ad i additio slightly smoothes the iput. Typical plate characteristics such as white backgroud ad black border differetiate the plate s properties from other parts of the vehicle. This is highly advatageous for a edge based algorithm. The algorithm idetifies the border of the licese plate by aalyzig the detected edges. The kowledge about the stadardized size of a licese plate is also used i the algorithm. Depedig o the camera positio ad the system camera calibratio the projected size of a licese plate ito the image plae is differet for each survey poit. Therefore, the algorithm is parametrized to adapt to variable locatio coditios at differet road cotrol statios. Fig. 5: Edge based licese plate segmetatio Our border idetifyig algorithm sequetially examies the edge image E H i Figure 5 lie by lie. As soo as the umber of edge poits i a lie correspods to a parametrized threshold, the algorithm searches for lies a certai distace away depedet o the plate size. If such a pair of lies (as lower ad upper border cadidates) is foud, a similar search is performed for colums i edge image E V (Figure 5). This search is restricted to the area betwee the two lies used as pair cadidates. If this detectio process is successful, a subsequet cosistecy check esures that edge pixels i determied lie pairs are coheret ad furthermore lie betwee the vertical bouds. B. Character Field Segmetatio As Figure 6 illustrates, the segmetatio of plate images is iflueced by plate properties ad local illumiatio coditios (e.g. shadows). I order to provide comparable iformatio for further processig, a additioal segmetatio which elaborates the essetial iformatio is required: amely the idetificatio of letters iside a character field. Fig. 6: Segmetatio results The segmetatio starts searchig for a suitable determiatio threshold i order to distiguish betwee plate backgroud ad letters. This search is realized by selectig a rectagular regio i the cetre of the licese plate image ad performig a histogram aalysis of the pixel values. After subsequet determiatio of the etire licese plate image, the upper ad lower boudary of the character field are searched. A upper boudary is foud if a log coheret area of white values i a lie is detected which is followed by at least three lies cotaiig substatially shorter coheret areas of white ad black values (see Figure 7). I a correspodig maer, the lower ad vertical boudaries are determied.

For the implemetatio of the similarity measure rec we have chose the correlatio c( x, y) which is mathematically defied for two real fuctios f(x,y) ad g(x,y) i equatio. Fig. 7: Detectio of the character field V. FEATURE EXTRACTION AND RECOGNITION I order to realize the recogitio, a fuctio rec is defied as a measure for the similarity of two plate image feature vectors with the sigature: rec: feature-vector x feature-vector [0,] This fuctio provides a uique value for the recogitio of a vehicle at two survey poits M i ad M j. It ca be used as a weightig factor i the calculatio of route related traffic data. Furthermore, every survey poit M is assiged a uique idetificatio umber i. For every M i the set D i of feature vectors is defied which is the represetatio of vehicles passig survey poit M i. The storage implemetatio of D i uses a data structure that allows efficiet isertio ad retrieval of feature vectors. The data structure represets iterally the geometrical topology of survey poits. Thus, every topological predecessor ad successor of a survey poit ca be determied. Additioally, stored iformatio are distaces betwee two cosecutive survey poits, ad the average travel time betwee selected poits. With these elemets the recogitio process is coceived as follows:. A feature vector v occurs at the measure poit M i. The set of idetifiers J of all direct predecessors of M i is determied 3. Calculate the set of cadidates V = D j j J It is possible to remove all elemets v from V for which it is improbable that v ad v are similar (for example the occurrece time differece betwee v ad v is too small). 4. Calculate m= max{ rec( v, v) v V} 5. Idetify assigmet: if m is greater tha the threshold S rec, the v has bee recogized. 6. Update the data structures ad i the case of a recogitio collect route iformatio ad iitialize calculatio of route related data. The calculatio of the feature vector for a plate matrix is the task of the feature extractio process, while the determiatio of m is performed by the recogitio process. c( x, y) = f ( x, y) g( x+ x, y+ y) d xdy () xy, Applied to our applicatio we cosider two greyscale plate images N ad N (see Figure 4) which ca be thought of as greyscale matrices or discrete fuctios u(m,) ad v(m,). They correspod to the fuctios f ad g i the defiitio above. The correlatio value, which is calculated depedet o the traslatios, represets a measure for the similarity of the images beig overlaid i a certai aligmet. As a characteristic value for the similarity of two images, the maximum correlatio value c is calculated over all possible aligmets of the two images that ca be obtaied by traslatig the images i both directios parallel to the mai axis. So far, the use of the correlatio appears to be a proper solutio for our similarity measure rec. However, c caot be used for the compariso of differet image pairs sice its expressiveess is limited to a fixed choice of a sigle image pair. Moreover, c is ot ivariat regardig differet illumiatio coditios that might occur at differet survey poits. The problems ca be tackled by ormalizig the correlatio value []. Workig with the above approach leads to a umber of issues which must be resolved. First, a computatioal time problem eeds to be solved sice correlatio has a computatioal time complexity of O( 4 ) cosiderig images of size x. Oe solutio is the trasformatio i the frequecy domai. Accordig to the Wieer-Khichi-theorem [], the correlatio ca be calculated by multiplyig both Fourier trasformatios. Other solutios exist for the spatial domai as for example the limited cosideratio of possible aligmets of the images or the omissio of maximum value calculatio. Cosiderig the computatioal overhead while trasformig a image ito the frequecy domai, the latter solutios i the spatial domai are less computatioally expesive. Secodly, the questio arises whether to use black ad white images istead of grey scale images. It ca be show, however, that the average correlatio error is 7 times higher while usig black ad white images tha the error whe usig images of 56 grey toes. Additioal difficulties arise i the decisio of a suitable determiatio threshold. The result of our cosideratios is the followig algorithm, which uses as iput the character field of a licese plate segmeted as described i Sectio IV.B:. Calculate the feature vector from the image: a) Scale the plate image C to a stadard size image N b) Extract grey values from the image ad write them to field (feature stripes) c) Calculate the expectacy m = field[] i

d) Calculate the square root from the auto correlatio a = ( field [ i ] m ) e) Create the feature vector v = (field, m, a). Calculate rec(v,v ) of two feature vectors v ad v a) Calculate r = ( field[ i] m) ( field[ i]' m') a a' 00 r if v > 0 b) Calculate: rec( v, v') = 0 otherwise Referece licese plate: 8 8 8 4 37 4 VII. RESULTS Fig. 8: Correlatio based similarity measure of data set to referece licese plate The algorithms preseted above have bee itegrated i VESUV, a comprehesive, itegrated machie visio system for traffic surveillace. The VESUV system has bee evaluated i a field trial o a Germa highway. Usig 4 video cameras traffic data was recorded at a poit of covergece i order to cover every traffic etrace ad exit. O the base of this test data (see Fig. 8), the system s reliability as well as its time cosumptio were evaluated. Results are give i Table ad show that 60% of vehicles were recogized correctly. This recogitio rate is satisfyig the specified requiremet, sice oly represetatives of traffic streams are eeded i order to calculate route related traffic data. The evaluatio of the time cosumptio showed that VESUV has bee able to process the video data i real time o a SUN Ultra with 67 MHz. The work preseted i this paper shows promisig results that will efficietly improve a ew geeratio of traffic cotrol systems. Percetage correct Percetage correct (idividual error) (superpos. of errors) Selectig frames 98% 98% of iterest Detectio of licese plate 7% 70% Correlatio based 84% 60% idetity Tab. : Precisio of et-liked data 7 80 56 44 40 45 60 VIII. CONCLUSION I this paper we preseted a recogitio pipelie for processig video data i order to perform olie route tracig by a machie visio system. The objective of the preseted approach is to miimize the computatio time for the recogitio by preselectig feature vectors accordig to the well kow topology of the road etwork ad efficietly reducig the data volume for cosideratio. Moreover, the time eeded to compare two feature vectors is miimized by precalculatio of ecessary iformatio at creatio time. Due to the use of the correlatio method, our approach does ot deped o optical character recogitio (OCR) of the plate s cotets. Eve though iterpretig sigle characters is performed successfully i a variety of related work, the correlatio approach is preferable with regard to privacy, sice o idetificatio of idividuals is possible. The outcome of our approach is the olie acquisitio of route related data such as travel times ad tracig of traffic streams, which ca highly icrease the efficiecy of traffic cotrol systems. REFERENCES [] Buxto, H.; Gog, S.: Advaced Visual Surveillace usig Bayesia Networks. I: Iteratioal Coferece o Computer Visio, Cambridge, Massachusetts, (995). [] Jai, A.K.: Fudametals of Digital Image Processig, Pretice Hall, (989) [3] M. Kilger: A Shadow Hadler i a Video-based Real-time Traffic Moitorig System. IEEE Workshop o Applicatios of Computer Visio, pp.060-066, (99). [4] Kuehe, R. et al: Loop-based travel time measuremet. I: Traffic Techology Iteratioal, pp. 57-6, (997). [5] Malik, J.: Robust Multiple Car Trackig with Occlusio Reasoig. I: Proc. Third Europea Coferece o Computer Visio, Stockholm, Swede, May -6, pp. 89-96, (994). [6] P.G. Michalopoulus et al.: Vehicle detectio video through image processig: the Autoscope system. IEEE Trasactios o Vehicle Techology, vol. 40, o., (99)