Developing a Real-Time Person Tracking System Using the TMS320C40 DSP

Transcription

1 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP APPLICATION BRIEF: SPRA89 Authors: Ruzo Okada Shina Yamamoto Yasushi Mae Advising Proessor: Yoshiaki Shirai Deartment o Mechanical Engineering or Comuter Controlled Machiner, Osaka Universit Digital Signal Processing Solutions Januar 996

2 IMPORTANT NOTICE Teas Instruments TI reserves the right to make changes to its roducts or to discontinue an semiconductor roduct or service without notice, and advises its customers to obtain the latest version o relevant inormation to veri, beore lacing orders, that the inormation being relied on is current. TI warrants erormance o its semiconductor roducts and related sotware to the seciications alicable at the time o sale in accordance with TI s standard warrant. Testing and other qualit control techniques are utilized to the etent TI deems necessar to suort this warrant. Seciic testing o all arameters o each device is not necessaril erormed, ecet those mandated b government requirements. Certain alication using semiconductor roducts ma involve otential risks o death, ersonal injur, or severe roert or environmental damage Critical Alications. TI SEMICONDUCTOR PRODUCTS ARE NOT DESIGNED, INTENDED, AUTHORIZED, OR WARRANTED TO BE SUITABLE FOR USE IN LIFE-SUPPORT APPLICATIONS, DEVICES OR SYSTEMS OR OTHER CRITICAL APPLICATIONS. Inclusion o TI roducts in such alications is understood to be ull at the risk o the customer. Use o TI roducts in such alications requires the written aroval o an aroriate TI oicer. Questions concerning otential risk alications should be directed to TI through a local SC sales oice. In order to minimize risks associated with the customer s alications, adequate design and oerating saeguards should be rovided b the customer to minimize inherent or rocedural hazards. TI assumes no liabilit or alications assistance, customer roduct design, sotware erormance, or inringement o atents or services described herein. Nor does TI warrant or reresent that an license, either eress or imlied, is granted under an atent right, coright, mask work right, or other intellectual roert right o TI covering or relating to an combination, machine, or rocess in which such semiconductor roducts or services might be or are used. Coright 997, Teas Instruments Incororated

3 TRADEMARKS TI is a trademark o Teas Instruments Incororated. Other brands and names are the roert o their resective owners.

4 CONTACT INFORMATION US TMS30 HOTLINE US TMS30 FAX US TMS30 BBS US TMS30 dsh@ti.com

5 Contents Abstract... 7 World Wide Web... 8 Introduction... 9 Etraction o the Otical Flows... Constraint Equation o Gradient Method... Filtering... Recursive Filtering...3 Satial Filter...3 Temoral Filter...5 Result o Etracting Otical Flows...5 Person Tracking Algorithm...7 Person-Window Prediction...8 Etraction o Velocit and Object Window...8 Determination o Person Window...8 Camera Control...9 Imlementation...0 Hardware Architecture...0 Imlementation o the Algorithm...3 Result o Tracking...5 Summar...6 Reerences...6

6 Figures Figure. Inut Images...6 Figure. Otical Flows...6 Figure 3. Tracking Algorithm...7 Figure 4. Sstem Overview... Figure 5. TMS30C40 DSP Board Block Diagram... Figure 6. Imlementation...4 Figure 7. Tracking Result...5

7 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP Abstract This alication reort describes the develoment o a real time sstem or tracking a walking erson using a high-seed image rocessor and an active camera head. The image rocessor is comosed o ive digital signal rocessing DSP boards. Each DSP board includes two Teas Instruments TI TMS30C40 DSPs that communicate with each other through the communication orts and erorm various image-rocessing in arallel. The irst three DSP boards calculate lows the velocit ield o an image based on the gradient method. Net, the region o a erson on the image is etracted rom the otical lows using knowledge about the shae o a erson. Finall, the camera head is controlled to kee the erson in the ield o view o the camera. The result is shown or tracking a erson in a cluttered background. This document was an entr in the 995 DSP Solutions Challenge, an annual contest organized b TI to encourage students rom around the world to ind innovative was to use DSPs. For more inormation on the TI DSP Solutions Challenge, see TI s World Wide Web site at Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 7

8 World Wide Web Our World Wide Web site at contains the most u to date roduct inormation, revisions, and additions. Users registering with TI&ME can build custom inormation ages and receive new roduct udates automaticall via . 8 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

9 Introduction Collision avoidance is a ke consideration or eole working with autonomous robots. Comuter vision is an eective tool in collision avoidance because it tracks a erson without requiring hsical contact. Tracking generall takes ecessive time to rocess images because o the enormous amounts o data contained in the images. A camera must be able to turn to kee a erson in its ield o view while the erson moves in the environment. Our objective is to track a walking erson in real time using an active vision sstem with a camera head and high-seed image rocessor. We use otical lows, the velocit ield o an image, to detect a erson in an image. Various methods are available to calculate the otical lows. 3 This alication reort uses the gradient method. 4 5 Two dierent satial ilters are convolved to an inut image to solve the constraint equation o the gradient method. Though the cost o the convolution oeration can be rohibitive, dividing the two-dimensional satial ilters into two one-dimensional recursive ilters can reduce the rocessing. 6 The region o the erson is segmented based on the dierence in velocit between the erson and the background. However this method cannot detect the correct region o the erson i the velocit o the erson is similar to the background velocit or a art o the erson s bod is occluded. We use the knowledge o the shae o the erson to coe with these situations. The shae o the erson is the ratio o the height to the width o the erson in the image. Since the ratio o the walking erson does not change much, the region o the erson is determined to kee the ratio b modiing the region detected rom the otical lows. The active camera head is controlled to kee the erson in the ield o view o the camera. The image rocessor has a video board and ive DSP boards. Each DSP board contains two TI TMS30C40 DSPs that communicate with each other through the communication orts. The video board is the interace between the active camera and the host comuter. The DSP boards erorm image rocessing. Our tracking is imlemented to the image rocessor. The irst three DSP boards calculate the otical lows. The ourth DSP-board detects the erson and controls the active camera head. And the ith DSP-board makes an image o the result or disla. Section describes the etraction o otical lows. The result o etraction o otical lows is shown. Section shows the tracking algorithm. Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 9

10 Section 3 describes the imlementation o the tracking algorithm to the image rocessor. The architecture o the image rocessor is shown. The result o tracking is shown. 0 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

11 Etraction o the Otical Flows The velocit ield is derived rom otical lows calculated using the gradient method. The gradient method is based on satio-temoral iltering. 3 Constraint Equation o Gradient Method Let,, t denote a brightness at a oint, in an image at time t. I this oint moves to a oint + d, + d at time t + dt, the ollowing equation holds:,,t = +d,+d,t+dt Talor eansion o the right side o the equation is,,t =,,t +,,td+,,td + t,,tdt + e, where =, =, t =, and e is high order terms o d,d,dt. t Assuming that e is negligible, we obtain the net equation: =,,tu +,,tv + t,,i = 0 3 d d where u =, v = and u and v are the and comonent o the dt dt velocit. This is the constraint equation o the gradient method. However, equation 3 has two unknowns u and v and cannot be solved. The unknowns u and v are calculated as shown in the Filtering section. Filtering Two dierent satial ilters g,h are convolved to the inut image,,t, and the temoral smoothing ilter is alied using these two iltered images. The ollowing two constraint equations are derived according to equation 3. g u + g v + g t = 0 4 h u + h v + h t = 0 5 where reresents convolution. u and v are calculated rom equations 4 and 5 as ollows: Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

12 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP t t h g h g g h h g u = 6 t t h g h g h g g h v = 7 I the absolute value o denominator o equations 6 and 7 is small, the velocit at such oint is not calculated. This is because the indeendence o equations 4 and 5 is insuicient. These ilters must have the ollowing three characteristics. To assure dierentiabilit To suress the high-order terms o d,d,dt To make two indeendent constraint equations 4 and 5 We use the horizontall dierentiated gaussian ilter or g. The horizontal variance o this gaussian is larger than the vertical variance, as shown in equation 8. Similarl, or the ilter h, the verticall dierentiated gaussian ilter. The vertical variance is larger than the horizontal variance, as shown in equation 9. = + = 3 e e e, σ πσ σ πσ σ σ σ πσ g 8 = + = 3 e e e, σ πσ σ πσ σ σ σ πσ h 9 where σ, σ are variances and σ > σ. Equation 0 is used or the temoral ilter i = β 0 t + β t - + β t β n t - n where t is the outut.

13 t is the inut image at time t. The coeicients hold the ollowing conditions: n i= 0 Recursive Filtering Satial Filter β i, lim β n n = 0 The convolution oeration o a two-dimensional satial ilter is comutationall eensive unless the ilter is small. A satial ilter with size M N is convolved, the number o calculations is M N at a iel. Equations 8 and 9 show it is ossible that the ilter g,h can be searated to -directional one-dimensional ilter and - directional one-dimensional ilter, and the can be convolved searatel. This can reduce the calculation to M + N at a iel. However, this convolution oeration is still comutationall eensive. This section describes how the recursive iltering method can reduce the calculation more. 6 Let i denote the outut o a ilter hi and i denote the inut to the ilter. N k= 0 i = h k i k Equation is the transer unction, or Z transormation, o equation. N k= 0 k H z = h k z The net equation is an aroimation o this transer unction. H a m k= 0 + k= k bk z z = 3 n k a z k The net equation is the inverse Z transormation o 3. This equation shows that iltering can be erormed recursivel. m k= 0 n i = b i k a i k 4 k k= k Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 3

14 The combination o equations 6 and 8 construct the twodimensional ilter g and h. In the case o ilter g, -directional ilter is dierential gaussian ilter 8 and -directional ilter is gaussian ilter 6. Gaussian Filter The net equation is an aroimation o the Gaussian ilter. α n S n = k α n + e 5 where k e = + αe α α e α and α decides the variance σ o Gaussian. This aroimate ilter can be reresented in a recursive orm based on equations through 4 as ollows: n = n + n n =,.., M 6 n = k[n + e -α α - n - ]+ e -α n - - e -α n - n=,..,m n = k[e -α α - n + - e -α n + ] + e -α n + - e -α n + n = M,.., Dierential Gaussian Filter Equation 7 is an aroimation o the dierential Gaussian ilter. Dn = kne --α n 7 where, k e = e α α α is constant, which decides ilter size. Equation 7 is reresented in recursive orm based on equations through 4 as ollows: α n = ke [ n n] n =,, M 8 + n = ke = ke α α n = ke = ke [ n + e n α α n =,, M [ n + + e n α n = M,, + n n e α n + e α α + n ] n + ] 4 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

15 Temoral Filter The temoral ilter shown in equation 0 is also reresented in recursive orm. Equation 9 is a recursive temoral ilter. t = β t + β t 9 where t is an inut image. i is the outut o the ilter. β determines the ilter size. Result o Etracting Otical Flows Figure shows the etracted otical lows rom the inut images shown in Figure. The constant α in equations 6 and 8 and β in equation 9 are ied, as shown in this eeriment: g α = 0.3, h α = 0.7, β = 0.5 g α = 0.7 h α 0.3 where subscrits and mean the directions o the ilters. suerscrits g and h mean the satial ilters g and h, resectivel. I the absolute value o the denominator in equations 6 and 7 is less than a threshold, the otical low is not etracted in such iels because the etracted otical low is not reliable. In this eeriment, the threshold is.0. Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 5

16 Figure. Inut Images Figure. Otical Flows time t time t+ 6 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

17 Person Tracking Algorithm Figure 3. Tracking Algorithm Detection o a erson is accomlished b circumscribing the erson with a rectangle, or erson window, in the image. The tracking determines the size and location o the erson window over a sequence o images. It is imortant to control the active camera head to kee the erson in the ield o view o the camera. Person detection is erormed based on the otical lows. The region o the erson is segmented based on the dierences o the velocities between the erson and the background. When the velocit o the erson is similar to the background velocit, it is diicult to segment the region o the erson. When art o the bod is occluded, the whole bod can not be segmented. In this case, the knowledge o the shae o the erson is used to enable to track the erson. When a erson is walking, his shae in the image does not change greatl; that is, the ratio o the height to the width o the erson window does not change much. The erson window is determined to kee this ratio. This method can overcome the diiculties stated below. Figure 3 shows the tracking algorithm. Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 7

18 Person-Window Prediction Let u t-, v t- denote the velocit o the erson, and W t- denote the erson window at revious rame time t. W t- has two arameters: location and size. The location is reresented b the center oint o the window t-, t-, and the size is reresented b the height and the width o the erson window height t-, width t-. The interval between rames is short enough that we can assume the erson maintains velocit u t-, v t- as he moves. The redicted redict t erson window W at current rame time t is obtained b redict shiting W t- b the velocit u t-, v t-. The location o W t is t- + u t-, t- + v t- and the size is height t-, width t-. Etraction o Velocit and Object Window The mean velocit u t-, v t- is calculated in the redicted erson redict window W t and is the velocit o the erson at time t. The region that has the velocit similar to the mean velocit u t-, v t- is redict etracted near the redicted erson window W t. Velocities u and v, satising the ollowing conditions 0 and, are the similar velocities to the mean velocit u t, v t. The window circumscribing this region is called object window. u min < u < u ma 0 v min < v < v ma where u min u t + c u t - c u ma = u t + c u t +c v min v t + c v t - c v ma = v t + c v t +c c and c are redetermined constants. Determination o Person Window The size o the erson window is equal to the object window i the ratio o the height to the width o the object window is similar to the reviousl remembered ratio. I the ratio o the object window is dierent rom the remembered ratio, the erson window is determined b modiing the object window based on the remembered ratio. The size o the erson window is determined b the ollowing equations: width = width o 8 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

19 height = height o height = height 0 width = height /ratio where subscrit is the erson window and o is the object window. Equation is used when the dierence o the width between the object window and the redicted erson window is less than the dierence o the height. Otherwise, equation 3 is used. The location o the erson window is equal to the object window. This case means that a art o the bod o the erson is occluded, or that the velocit o the art o the bod is dierent rom the velocit o the other arts o the bod. I the velocit u t,v t is similar to the background velocit u t, vt, the erson window is not changed and is determined b the redicted erson window. In this case, the object window cannot be etracted because there is onl a small dierence between the background and the erson on the velocit ield; thus, the erson stos against the camera motion. b b Camera Control The active camera head is controlled to kee the erson in the ield o view o the camera. Let, and u, v denote the center oint and the velocit o the erson window, resectivel. For eamle, i the erson is in the let hal o the image L and moves to the let b the velocit u, the camera is turned to the let b u iels. The active camera head is controlled as ollows: I, L and u > 0, turn let b u iels. I, R and u < 0, turn right b u iels. I, U and v > 0, turn uward b v iels. I, D and v < 0, turn downward b v iels. where L is the let hal o the image. R is the right hal o the image. U is the uer hal o the image. D is the lower hal o the image. Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 9

20 Imlementation Hardware Architecture Figure 4 shows the overview o the real-time erson tracking sstem. The image rocessor has a video board and ive DSP boards connected in tandem. The image rocessor is ieline architecture. The video board is the interace with the active camera head, the host comuter, and the TV monitor. The DSP board can rocess the inut image in arallel because it has two indeendent DSPs. We call these DSP A and DSP B. DSP A in the ourth DSP board sends the control signal to the active camera head controller. The controller turns the active camera head based on the signal rom the DSP. The image rocessor includes the ollowing secial eatures. As the ieline architecture is used, a burden is disersed b increasing the DSP boards accordingl. Various image rocessings are available b changing the rograms downloaded to the DSPs. Parallel image rocessing is ossible because each DSP board has two indeendent rocessing sections each section has a DSP. Two DSPs on the same DSP board communicate with each other through the communication orts. 0 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

21 Figure 4. Sstem Overview The video board erorms three unctions: AID transormation o the image rom the active camera head Interace to the host comuter Digital-to-analog D/A transormation o the image rocessed b DSP boards Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

22 The inut signal is searated to snchronous signals and image data. The image signals are digitized to 8 bit image data. The snchronous signals and the digitized image data are sent to the DSP boards through the video bus. The video board sends the interrut signal rom the host comuter to the DSP boards. The video board downloads the rograms rom the host comuter to the each DSP through the control bus. Figure 5 shows the block diagram o the DSP board. The inut data and outut data o the DSP board are 8 bit or 3 bit. The 8 bit data are used or the inut and outut or the video board. The 8bit inut data video board is written to both the image memor o DSP A and DSP B. Either DSP A or DSP B can outut 8 bit data. The 3 bit data are used or the inut or the outut or the DSP board. The inut data is written to the image memor. Each DSP rocesses a image stored in the image memor according to the rogram downloaded b the host comuter. The rogram memor stores the rogram and various data. The result o the rocessing is outut to the net board. This image memor has a double buer, and the DSP can not read data rom the buer receiving the data rom the revious board. The DSP board can receive data rom the revious board even i the DSP is erorming the image rocessing. Figure 5. TMS30C40 DSP Board Block Diagram Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

23 Imlementation o the Algorithm The rocessing described in the Filtering section is assigned to the DSPs. DSP board The recursive satial ilter is convolved to the inut image. DSP A convolves ilter g. Filter h is convolved b DSP B. DSP board DSP A and B erorm the same rocessing. First, the recursive temoral ilter is alied to the outut rom DSP board using the result o the revious rame. And net, the dierentiations with resect to the,, t are calculated. The outut o the temoral ilter is stored to use the net rame in the rogram memor. DSP board 3 DSP B transmits the data rom revious the DSP board to DSP A. DSP A calculates the otical lows based on equations 6 and 7. DSP board 4 DSP B determines the erson window based on the otical lows and the knowledge o the shae o the erson. DSP A is not used. DSP board 5 DSP B overwrites the erson window on inut image and send it to the video board. DSP A is not used. Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 3

24 Figure 6. Imlementation 4 Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP

25 Result o Tracking Figure 7. Tracking Result Figure 7 shows the result o the real-time erson tracking. A erson is walking rom let to right. The bo on the loor occludes his eet. Unless knowledge o the shae o the erson is used, the eet o the erson can not be tracked. This method allows the sstem to track his whole bod even though a bo occludes the eet. The active camera head is controlled to kee the erson in its ield o view. Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP 5

26 Summar We develo a real-time erson tracking sstem. The sstem comrises the high-seed image rocessor and the active camera head. The image rocessor rocesses images in real time using TI TMS30C40 DSPs. The region o the erson in the image is detected based on the otical lows and knowledge o the shae o the erson. The active camera head is controlled to track the erson. An eerimental result shows the eectiveness o this method. Reerences Y.Shirai, J.Miura, Y.Mae, M.Shiohara, H.Egawa, S.Sasaki: Moving Object Percetion and Tracking b Use o DSP, Prc. CAMP, J.L.Barron, D.J.Fleet, S.S.Beauchemin. Sstems and Eeriment Perormance o Otical Flow Techniques, International Journal o Comuter Vision, Vol., No., ~43-77, H.Inoue, T.Tachikawa, M.Inaba: Robot Vision Sstem with a Correlation Chi or Real-time Tracking, Otical Flow and Deth Ma Generation, Proc. IEEE International Conerence on Robotics and Automation, M.V.Srinivasan: Generalized Gradient Schemes or the Measurement o Two-Dimensional Image Motion, Biological Cbernetics, Vol.63, P.Sobe, M.V.Srinivasan: Measurement o Otical Flow Using a Generalized Gradient Scheme, Journal o the Otical Societ o America, R.Derishe: Fast Algorithms or Low-Level Vision, IEEE Trans. Pattern Anal. Mach. Intell., Vol., Develoing a Real-Time Person Tracking Sstem Using the TMS30C40 DSP