Using Mean-Shift Tracking Algorithms for Real-Time Tracking of Moving Images on an Autonomous Vehicle Testbed Platform

Transcription

1 Usng Mean-Shft Trackng Algorthms for Real-Tme Trackng of Movng Images on an Atonomos Vehcle Testbed Platform Benjamn Gorry, Zezh Chen, Kevn Hammond 2, Andy Wallace 3, and Greg Mchaelson () Dept. of Compter Scence, Herot-Watt Unversty, Rccarton, Scotland; (2) School of Compter Scence, Unversty of St Andrews, St Andrews, Scotland; (3) Dept. of Electrcal & Compter Engneerng, Herot-Watt Unversty, Rccarton, Scotland. ABSTRACT Ths paper descrbes new compter vson algorthms that have been developed to track movng objects as part of a long-term stdy nto the desgn of (sem-)atonomos vehcles. We present the reslts of a stdy to explot varable kernels for trackng n vdeo seqences. The bass of or work s the mean shft object-trackng algorthm; for a movng target, t s sal to defne a rectanglar target wndow n an ntal frame, and then process the data wthn that wndow to separate the tracked object from the backgrond by the mean shft segmentaton algorthm. Rather than se the standard, Epanechnkov kernel, we have sed a kernel weghted by the Chamfer dstance transform to mprove the accracy of target representaton and localzaton, mnmsng the dstance between the two dstrbtons n RGB color space sng the Bhattacharya coeffcent. Expermental reslts show the mproved trackng capablty and versatlty of the algorthm n comparson wth reslts sng the standard kernel. These algorthms are ncorporated as part of a robot test-bed archtectre whch has been sed to demonstrate ther effectveness.. INTRODUCTION Ths paper compares and contrasts three compter vson algorthms for trackng movng objects n vdeo seqences. Detectng and followng movng objects aganst complex backgronds s crtcal n a nmber of atonomos vehcle (AV) applcatons. For example, n a fll-scale AV t may allow s to track and avod collsons wth pedestrans or movng vehcles, and n a robotc context, t may allow mproved navgaton and enhance safety. Good solaton of ndvdal movng objects wll allow s develop applcatons that nvolve followng targets of nterest. All these applcatons reqre s to process fll-color vdeo seqences n real tme. Or work s ndertaken n the context of a robotc AV testbed platform, based on the Poneer P3-AT all-terran robot, as part of a broader UK project that s developng new sensor technology for atonomos vehcles, and fnded by the Systems Engneerng for Atonomos Systems (SEAS) Defence Technology Centre (DTC). The SEAS DTC s operated by a UK ndstral consortm and ams to research nnovatve technologes relevant to atonomos systems, at both whole-system and sb-system level and, throgh the adopton of Systems Engneerng approaches, to facltate pll-throgh of the technology nto mltary capabltes.. Contrbtons Ths paper makes a nmber of novel contrbtons. Frstly, we descrbe new algorthms for trackng movng mages aganst complex, clttered backgronds, based on prevosly stded mean-shft algorthms. Secondly, we show

2 that or algorthms are capable of trackng movng targets for fll-sze vdeo mages n real tme. Thrdly, we show how the algorthm can be deployed n a smple AV testbed, based on a Poneer P3-AT all-terran robot. Fnally, or mplementaton s nsal n beng wrtten sng the novel programmng langage Hme [, 2], a langage that combnes fnctonal programmng concepts wth fnte-state atomata for programmng real-tme reactve systems. 2. THE MEAN-SHIFT VISION ALGORITHM 2. Mean Shft Segmentaton We present the reslts of a stdy to explot varos kernels for real-tme trackng of movng objects n vdeo seqences. The kernels are defned by the nderlyng, smple, robst, and dverse mean shft, clsterng algorthm [3], frst appled to mage segmentaton by Comanc and Meer [4, 5, 6]. For the atonomos vehcle applcaton, the task s to frst defne an object of nterest, by segmentaton and/or by nteractve selecton, then by trackng the object as t moves wthn the camera feld of vew. The mean shft algorthm s desgned to fnd modes (or the centers of the regons of hgh concentraton) of data represented as arbtrary-dmensonal vectors. The algorthm proceeds as follows [7]. Choose the rads of the search wndow. Choose the ntal locaton (center) of the wndow. Repeat Compte the mean (average) of the data ponts over the wndow and translate the centre of the wndow nto ths pont. Untl the translaton dstance of the center becomes less than a preset threshold. Hgh-densty regons n the featre space correspond to sffcently large nmbers of pxels n a narrow range of ntenstes/colors n the mage doman. Therefore, provded the pxels form connected regons (as s often the case for relatvely smooth mages); the algorthm essentally fnds relatvely large connected regons that have sffcently small varatons n ntensty/color (and are ths perceved as well defned regons by hmans). In practce, the algorthm proceeds by placng randomly one search wndow at a tme, fndng the correspondng mode, and removng all the featre vectors n the fnal wndow from the featre space. Ths one wold expect to fnd larger regons frst. (a) A 320x240 color mage. (b) The correspondng RGB color space. (c) The correspondng Lv color space. Fgre : Relatonshp between mage, RGB and Lv space. In or mplementatons, before segmentng color mages, pxels, sally represented n the RGB color space, are mapped nto the Lv color space whch has a brghtness component represented by L and two chromatc

3 components represented by and v. It has been arged that the latter color space s more sotropc and ths s better stable for the mode fndng algorthm. Fnally, when defnng a varable kernel, ths s constraned wthn a rectanglar wndow that we also sed when dsplayng the trackng reslt. Fgre llstrates an example of the relatonshp between mage and featre space. Fgre 2 shows the segmentaton reslts n RGB and Lv space, respectvely. Sbjectvely at least, ths shows an mprovement n sng the Lv parameterzaton for ths partclar example. The flowchart of mplementaton of RGB to Lv by Hme s shown n Fgre 3. The flowchart of mean shft segmentaton s shown n Fgre 4. (a) Segmentaton reslt n RGB space. (b) Segmentaton reslt n Lv space. Fgre 2 : Segmentaton Reslts Fgre 3 : The flowchart of RGB to Lv (Hme) 2.2 Mean Shft Object Trackng A rectanglar wndow s defned abot the regon of nterest n an ntal frame. Then the mean shft algorthm s appled to separate the tracked object from the backgrond n Lv color space. As the object moves, an nsal kernel weghted by the Chamfer dstance transform mproves the accracy of target representaton and localzaton, mnmsng the dstance between two color dstrbtons sng the Bhattacharya coeffcent. In trackng an object throgh a color mage seqence, we assme that we can represent t by a dscrete dstrbton of samples from a regon n color space, localsed by a kernel whose centre defnes the crrent poston. Hence, we want to fnd the maxmm n the dstrbton of a fncton, ρ, that measres the smlarty between the weghted color dstrbtons as a fncton of poston (shft) n the canddate mage wth respect to a prevos model mage. If we have two sets of parameters for the respectve denstes p( x) and q ( x ), the Bhattacharyya coeffcent [8] s an approxmate measrement of the amont of overlap, defned by: ρ = p ( x) q( x) dx () Snce we are dealng wth dscretely sampled data from color mages, we se dscrete denstes stored as m-bn hstograms n both the model and canddate mage. The dscrete densty of the model s defned as:

4 m q = { q }, =,2, L, m = q = (2) Smlarly, the estmated hstogram of a canddate at a gven locaton y n a sbseqent frame s: m p( y) = { p ( y) }, =,2, L, m = p = (3) Accordng to the defnton of Eqaton (), the sample estmate of the Bhattacharyya coeffcent s gven by: m ρ () y = ρ[ p() y, q] () y (4) = = p q (Inpt) frst2layer o o2 o3 maxmn_box j o k s l m scale_box o o2 K s l m k s m colormap_box o k s m m st nex t s cr smvale tm tm meanshft mode m st next s cr smvale tm tm L 2 k sl left ps s tp l so state msot o o2 o3 left k oterror oterror otptscreeen sl ps s tp l' so state Fgre 4 : The flowchart of mean shft segmentaton (Hme) Let { x, x, L, x 2 n} be an ndependent random sample drawn from f ( x ), the color densty fncton. If K s the normalzed kernel fncton, then the kernel densty estmate s gven by: q n = = K ( x ) δ b( x ) [ ] Estmatng the color densty n ths way, the mean shft algorthm s sed to teratvely shft locaton y n the target frame, to fnd a mode n the dstrbton of the Bhattacharya coeffcent (Eqaton 4). Usng Taylor expanson arond the vales, p ( ), the Bhattacharyya coeffcent s approxmated by [8]: y 0 (5)

5 ( y), ρ [ p q] 2 m = p = n ( y ) q + w K( x ) 0 2 (6) where q = δ (7) ( x ) m w [ b ] = p y 0 ( ) To maxmze Eqaton (4), the second term n Eqaton (6) s maxmzed as the frst term s ndependent of y. In the mean shft algorthm, the kernel s recrsvely moved from the crrent locaton y 0 to a new locaton y accordng to the relaton: n n = x wg( y0 x ) wg( y0 x ) = = y (8) where G s the gradent fncton compted on K. Ths s eqvalent to a steepest ascent over the gradent of the kernel-fltered smlarty fncton based on the color hstograms. The flowchart of mean shft object trackng s shown n Fgre 5. Fgre 5 : The flowchart of the mean shft object trackng (Hme) 3. DEPLOYING THE TRACKING ALGORITHMS ON AN AUTONOMOUS VEHICLE TESTBED PLATFORM Or hardware testbed conssts of a Poneer P3-AT all-terran robot [9], SEBO (SEas robot, Fgre 6). We have confgred SEBO wth a front array of sonar dscs, rado Ethernet, front and back safety bmpers, and a srface-

6 monted camera that s sed to collect data for the mean-shft vson algorthms. Or Hme mplementaton nterfaces to standard software sppled wth the Poneer robot: ARIA (Advanced Robotcs Interface for applcatons), an open-sorce development envronment whch nterfaces to the robot s mcrocontroller, and provdes access to basc motor and camera fnctons; and VsLb, an open-sorce C-based vson-processng lbrary that provdes basc mage processng capabltes. Fgre 6: SEBO - the Herot-Watt/St Andrews Poneer P3-AT 3. Software Archtectre The software archtectre for or testbed mplementaton s shown n Fgre 7. Sold arrows represent local socket commncaton on the robot whle broken arrows represent wreless socket commncaton. All sorce code s located on the robot apart from the Java GUI, whch rns on a laptop compter. Real-tme mages are captred from the robot camera, passed throgh an mage-processng program located on the robot and wrtten n Hme, and then sent wrelessly to a laptop compter whch dsplays the mages n real-tme. For each mage a Red, Ble, and Green component s captred. The mage sze whch s captred s 240 x 320, ths reqrng a storage strctre of sze 3 x 240 x 320. Fgre 7 : Robot Test-bed Archtectre Fgre 8 : Screenshot of Interface

7 We have mplemented a smple command nterface on the laptop. When the ser decdes to move the robot, a sgnal s sent wrelessly from the laptop compter to a Hme program located on the robot. The Hme program then commncates wth a C++ ARIA program whch sends the basc motor commands to the robot. The camera s controlled n a smlar way to the movements of the robot. Fgre 7 shows that as the ser selects to control the camera a sgnal s sent wrelessly from the laptop compter to a Hme program located on the robot. The Hme program then commncates wth a C++ ARIA program whch sends commands to the camera. The camera panel, dsplayed on the top left part of Fgre 8, s sed to control the pan, tlt, and zoom featres of the camera. Two sets of camera controls are provded. The frst set allows mnmal movement or focs vales for the camera whle the second set allows maxmal movement or focs vales. 3.2 Incorporatng the vson algorthms The Hme pass-throgh box marked wth an astersk n Fgre 7 has been sccessvely replaced wth: the LUV converson algorthm; 2 the mean-shft segmentaton algorthm; 3 the mean-shft object-trackng algorthm. These algorthms prodce dfferent mage reslts. From the ntal experments, each of these algorthms can be ncorporated as part of the test-bed archtectre by smply replacng the Hme box whch passes the mages from the camera to the Java nterface. Work has begn on dentfyng dependences between each algorthm and establshng effcent lnk ponts where reqred. For the LUV converson algorthms, mages are presented n the LUV color space. For the mean-shft segmentaton algorthm, experments have nclded sng a varety of mage types and szes. Normally, mages of sze 240 x 320 are processed by the algorthm. Intal work wth the mean-shft object-trackng algorthm shows encoragng reslts. For an object placed n central vew of the camera, the robot or camera s moved at a steady rate, the object s tracked on the screen. Crrent work ncldes ntrodcng an opton where the ser can hghlght an object on the nterface screen by rbber-bandng the object of nterest. The co-ordnates of ths object, relatve to ts poston on the screen, are then passed to the mean-shft object trackng algorthm. From ths, f ether the object moves, the robot moves; or the camera on the robot moves, then the object s tracked sng the algorthm dscssed n secton 2.2. Ths s vsble on the nterface screen. 3.3 Implementaton and expermental evalaton Fgre 9 shows the frst frame and the foregrond mage of the tracked object. In ths case a smple regonal homogenety crteron has been appled as the target had relatvely nform ntensty. Partal reslts of trackng a male pedestran sng the NCDT kernel are shown n Fgre 0. Fgre 9 : Rectanglar wndow and segmentaton

8 Fgre 0 : Partal reslts of trackng a male pedestran by NCDT kernel. 3.4 The robot platform Crrently, the robot platform llstrated n Fgre 7 s beng sed to as a deployment archtectre for the vson algorthms that have been developed n Hme. The se of these algorthms s proof of concept that the Hme mplementatons work, and also that Hme can be sed n conjncton wth other ndstry-standard langages sch as C, C++, and Java. Each of the three algorthms dscssed n Secton 2 process mages n real-tme as they are captred from the camera monted on the srface of the robot. Possble extensons to the robot platform may nvolve de-localzng the sectons of Hme code whch lnk wth the robot API. By dong ths we cold cost these ndvdal programs to gan a measre of performance analyss. Alternatvely, the three Hme programs ndcated n Fgre 7 cold be combned. Ths one program cold then be analysed for ts performance and we cold also assess the reactvty rates when a robot or camera movement reqest s sent from the Java nterface. For the mean-shft trackng algorthm a nmber of experments are nderway. These nvolve trackng objects of dfferent szes, colors, objects aganst smlar and dssmlar backgrond colors, and objects of dfferent shapes. 4. RELATED WORK Work on real-tme trackng algorthms has taken place n a nmber of applcaton areas. By captrng mages n real-tme and then performng mage segmentaton we can partton an mage nto several dfferent regons. We can then se ths nformaton to track hghlghted objects. In the area of processng these algorthms work has been carred ot sng FPGAs (Feld Programmable Gate Arrays) nstead of mcroprocessors [0]. Ths ams to take advantage of the low cost and parallelsm assocated wth FPGAs. However, the algorthm dscssed n [0] reqres the se of clearly dstngshable florescent markers the mean shft object trackng algorthm dscssed n ths paper reqres no sch markers. Or ntal experments have shown that two smlar objects, whch dffer n color n some small way, can be dentfed ndependently of each other. In embedded real-tme applcatons the mportance of obtanng accrate bonds of tme and space sage s extremely valable [, 2]. If we can predct how or system shold behave we can place an pper bond on the expected execton tme of one program cycle. Usng Hme, ths can be done. The key to the desgn of Hme s ts ablty to be costed. To provde strctre to these costs, Hme has been developed as a seres of langage sbsets whch overlap [3]. Each sperset of ths overlappng seres of sbsets adds expressblty to the langage. By choosng the approprate level of langage, the programmer can obtan the balance they reqre between langage

9 expressveness and the reqred degree of costablty. Therefore, we can dentfy the tme and space bonds whch are reqred ths allows s to dentfy exact qanttes of hardware resorces we reqre. What wold be nterestng s to deploy the Hme algorthms dscssed n ths paper on FPGAs. We cold then se the Hme approach to cost the algorthms and compare the reslts aganst those obtaned from costng the algorthms on a mcroprocessor. 5. CONCLUSIONS AND FURTHER WORK In ths paper we have explored the se of varable kernels to enhance mean-shft segmentaton.. Expermental reslts show the mproved trackng capablty and versatlty of or mplementaton of mean-shft object trackng algorthms when compared wth reslts sng the standard kernel. These algorthms are ncorporated as part of a robot test-bed archtectre whch has been sed to demonstrate ther effectveness. Each algorthm has been developed sng Hme. By processng real-tme mages and commncatng wrelessly wth or robot, we can track movng mages aganst complex, clttered backgronds. Work s crrently nderway to extend ot testbed platform by:. developng new mage processng algorthms for AV deployment, and 2. complementng or moton-trackng algorthms by addng a lne followng algorthm. Ths wll nvolve extendng the nterface sed to control the physcal moton of the robot and camera. These extensons shold frther demonstrate:. how Hme can be sed to develop algorthms whch perform real-tme processng, 2. the flexblty of or testbed platform, and 3. the accracy of or trackng algorthms. 6. ACKNOWLEDGMENTS The work reported n ths paper was fnded by the Systems Engneerng for Atonomos Systems (SEAS) Defence Technology Centre establshed by the UK Mnstry of Defence. We wold lke to thank or collaborators n the EU FP6 EmBonded project, n partclar Chrstan Ferdnand, Renhold Heckmann, Hans-Wolfgang Lodl, Robert Ponton and Steffen Jost. 7. REFERENCES [] K. Hammond and G. Mchaelson, The Hme Report, Verson 0.3, 2006 [2] K. Hammond and G. Mchaelson, Hme: a Doman-Specfc Langage for Real-Tme Embedded Systems, Proc. of Int. Conf. on Generatve Programmng and Component Engneerng, Erfrt, Germany, Sept. 2003, Sprnger-Verlag Lectre Notes n Comp. Sc., pp [3] Y. Z. Cheng. Mean shft, model seekng, and clsterng. PAMI, 7(8): , 995. [4] D. Comanc, P. Meer. Robst analyss of featre space: Color mage segmentaton. In IEEE Conf. Compter vson and Pattern Recognton, , 997. [5] D. Comanc and P. Meer, Mean shft: A robst approach toward featre space analyss. IEEE Transactons on Pattern Analyss and Machne Intellgence, 24(5), , [6] D. Comanc, V. Ramesh, P. Meer, Kernel-based object trackng. IEEE Transactons on Pattern Analyss and Machne Intellgence, 25(5), pp , [7] Y. Keselman and E. Mchel-Tzanako. Extracton and characterzaton of regons of nterest n bomedcal mages. In Proceedng of IEEE Internatonal conference on Informaton Technology Applcaton n Bomedcne (ITAB 98), 87-90, 998.

10 [8] A. Bhattacharyya. On a measre of dvergence between two statstcal poplatons defned by ther probablty dstrbtons. Blletn of the Calctta Mathematcs Socety 35, pp99-0, 943. [9] MobleRobots Inc., Poneer 3 Operatons Manal wth MobleRobots Exclsve Advanced Control & Operatons Software, MobleRobots Inc., Janary [0] C. T. Johnston, K. T. Grbbon and D. G. Baley. FPGA based Remote Object Trackng for Real-tme Control, Internatonal Conference on Sensng Technology, Palmerston North, New Zealand, st [] G. Hager and J. Peterson, FROB: A Transformatonal Approach to the Desgn of Robot Software, Proc. of the nnth Internatonal Symposm of Robotcs Research, Utah, USA, 999. [2] D. Fjma and R. Udnk, A Case Sdy n Fnctonal Real-Tme Programmng, Techncal Report, Dept. of Compter Scence, Unv. of Twente, The Netherlands, 99 [3] K. Hammond, Explotng Prely Fnctonal Programmng to Obtan Bonded Resorce Behavor: the Hme Approach, Frst Central Eropean Smmer School, CEFP 2005, Bdapest, Hngary, Jly 4-5, 2005, Lectre Notes n Compter Scence 464, Sprnger-Verlag, 2006, pp