Computer Vision System for Tracking Players in Sports Games

Transcription

1 Computer Vision System for Tracking Players in Sports Games Abstract Janez Perš, Stanislav Kovacic Faculty of Electrical Engineering, University of Lublana Tržaška 5, 000 Lublana Te development of computer vision system for tracking players in indoor team games is presented. Several image processing and tracking metods are described, along wit camera calibration and lens distortion correction. Te output of te system consists of spatiotemporal traectories of te players, wic can be furter processed and analyzed by sport experts. In some critical situations te automatic tracking process must be manually interrupted. To correct miss-trackings, uman supervision is required. Some experimental results are presented as well. Keywords: obect tracking, uman motion analysis, camera calibration, andball. Introduction Human motion analysis is receiving increasing attention from computer vision researcers. Tis interest is motivated by applications over a wide spectrum of topics (Aggarval and Cai, 997). In tis paper we concentrate on tracking people in team sports based on computer vision. If te traectories are sufficiently accurately determined, a wealt of additional information, e.g. players speed, acceleration and players interactions can be obtained. For many years te analysis of a sport event as been based on observation seets filled-in during te matc. Later, wit a elp of video recordings, motion acquisition and analysis were performed manually, a time consuming and tedious task. In te past, progress in introducing te computer vision tecnology to te team sports domain was slow, due to inadequate video and computational facilities and complexity of te tracking problem itself. Te players strive to move rapidly, cange direction unpredictably and collide wit one anoter. Tey violate te smoot motion assumption, on wic many tracking algoritms are based. Players appear in te images as igly non-rigid forms, especially due to te movements of teir extremities. Many of te proposed approaces solved te motion acquisition and analysis problem only partially and were terefore unable to provide an adequate solution to te sports experts, i.e. tracking every player in te wole field in every instance of time (Erdmann, 994).. Image acquisition Careful planning of image acquisition could be crucial for te success of te wole proect, and inappropriately placed cameras can add a significant degree of difficulty to te tracking problem (Intille and Bobick, 995). To determine te traectories of te players from te beginning to te end of te game, te obects in question ave to be in te field of view for te duration of te wole matc. Two stationary cameras, mounted directly above te court, as sown in Fig. (a) were cosen as te most straigtforward solution. First Int l Worksop on Image and Signal Processing and Analysis, June 4-5, 000, Pula, Croatia

2 (a) (b) Fig. : (a) Camera placement. (b) Image obtained. Bot te andball matc and te test video sequence of players moving on predefined pats were recorded using two PAL S-VHS videorecorders. Te transfer to digital domain was carried out using te Motion-JPEG frame grabber at 5 frames per second wit image resolution of 384x88 pixels. Te result is sown in Fig. (b). 3. Camera calibration Due to significant radial distortion oterwise widely used calibration tecnique (Tsai, 987) failed to produce satisfactory results. We decided to take different approac by modeling te radial image distortion more accurately. Let us imagine an ideal pinole camera C, mounted on a pan-tilt device above te point 0, as illustrated in Fig. X is te observed point on te court plane, at distance from te point 0. is te distance from te camera to te court plane. Angle α is te angle of pan-tilt device wen observing te point X. Te differential d of radius is proected to te differential dr, wic is parallel to te camera image plane. Te image of dr appears on te image plane. elations between d, dr and α are given witin te triangle on te enlarged part of Fig. (a). C α dr α d X 0 X camera (a) (b) Fig. : (a) Model of radial distortion. (b) Te resulting images after correction. Tus, we can write te following relations: dr = cos( α ) d, α = arctg( ), () dr = cos(arctg( )) d. () Now, let us substitute te pan-tilt camera wit fixed camera, equipped wit wide-angle lens. Te wole area, wic would be covered by canging te angle α of a pan-tilt camera, is captured simultaneously to te single image of a stationary camera. Additionally, let us First Int l Worksop on Image and Signal Processing and Analysis, June 4-5, 000, Pula, Croatia

3 assume te scaling factor between te dr and te image of dr on te image plane to be. Terefore, we can obtain te lengt of te image of radius on te image plane by integrating left side of te eq. () over te interval (0, r ), and over te interval (0, ) on te rigt side. Wit being te distance from te observed point X to te point 0, and r being te distance from te image of point X to te image of point 0 on te image plane, we ave te solution of te inverse problem: ln. r = + + (3) By solving eq. (3) for we obtain te formula, wic can be used to correct te radial distortion: 4. Player tracking r ( e ) =. (4) r e We developed tree different algoritms for use in player tracker: motion detection, template tracking and color based tracking. We tested different combinations of tese algoritms; eac one as its own advantages and disadvantages. 4. Motion detection Te most straigtforward approac to motion detection is image subtraction. Eac frame C in te sequence was subtracted from te reference frame, i.e. an image of te empty court, yielding te difference image D, D = C + G GC + B BC, (5) were, G and B denote te red, green and blue components of te current frame and te reference frame, respectively. Te difference image D is ten tresolded using fixed tresold value and subsequently filtered to reduce image noise. Filtered image is sown in Fig. 3. esulting blobs in te filtered image are ten counted, labeled and teir centers of gravity calculated. Some of blobs correspond to te players, wile oters are caused by noise, sadows and oter distracting obects. Tis limits te efficiency of tis tecnique. Collisions of more tan two players and intense player sadows contacting oter players require almost always an intervention by uman operator. Typical problem is sown in Fig. 3 (b),(c). (a) (b) (c) Fig. 3: (a) Blobs detected in te motion detection stage. (b) Typical scene, wic confuses te motion detection algoritm. Four players are marked by wite arrows. (c) Corresponding difference image. First Int l Worksop on Image and Signal Processing and Analysis, June 4-5, 000, Pula, Croatia

4 4. Template tracking Visual differences between te players and te background obects were exploited to furter improve tracking process. Te feature set, wic could be used to successfully separate obects from background was needed. Tis is a difficult problem, as sape of te players varies over time and te image resolution is low. Players are represented by relatively small area of only 0-5 pixels in diameter. We ave defined a set of D functions, i.e. templates {K, =,..,4}, wic extract te very basic appearances of te player. Tey resemble te Wals functions, but tey are not ortogonal. Tese functions are sown in Fig. 4 (a). F (unknown obect) d GF d FH H (player) (a) G (background) Fig. 4: (a) Basic appearances of te player. Black areas represent zeros, wile grey areas represent values of. (b) Classification of an unknown obect. Eac cannel of GB color image is processed separately and te vector of 4 features F for eac cannel is obtained using te following expression: 6 6 = x= y= F K ( x, I( x,, ( =,.., 4), (6) were K is one of templates, and I is one of te tree GB cannels, obtained from te current image. Te result is 4-dimensional feature vector F. Additionally, similar vector of features G is calculated from te reference image of te empty playing court at te same coordinates. Te tird vector H is obtained by averaging te last n vectors of features F, wic allows certain adaptivity, as te player appearance canges over time. Te simplified, two-dimensional case of classification is sown in Fig. 4 (b). (b) (a) (b) (c) Fig. 5: Locating te player wearing te yellow dress using te set of templates. (a) Player is sown in te center of image. (b) Te difference image, as defined in (5) for tat particular case. c) Calculated similarity measure s as defined in (7). Te wite region (sowing ig similarit marks te player location more accurately tan te difference image, sown in (b). 0. First Int l Worksop on Image and Signal Processing and Analysis, June 4-5, 000, Pula, Croatia

5 Te bot distances (Fig. 4 (b)) and corresponding similarity measure s are defined as follows: 4 d = ( F G ) d = ( F H ), GF = Te results of player localization using tis approac are sown in Fig Color tracking FH 4 = s = (7) Color as an identifying feature can be also used for tracking players. Te tree-dimensional GB color representation was cosen, as some players wear dark dresses, wic would result in undefined values of ue. Color identification and localization, based on color istograms (Swain and Ballard, 99) is not appropriate, as tere are only a few (3-6) pixels tat closely resemble te reference color of te player s dress. To ensure reliable tracking, only te location of te pixel tat is most similar to te player s color becomes new player position. Te similarity measure used is defined as follows: S( x, = (( I ( x, C ) + ( I ( x, C ) + ( I ( x, C ) ), (8) were I is te image and C is te color of te player. Subscripts, G and B denote te red, green and blue cannel, respectively. Te algoritm searces for te pixel most similar to te recorded color of te player in limited area around te previous player position. Colors of te players are defined by te operator supervising te tracking process, before te actual tracking is started. 5. esults Te following tracking metods were tested: te motion detection algoritm (A), te color tracking algoritm (B) and te combination of color and template tracking algoritm (C). Te use of color tracking in C avoids drift in player position, caused by te template tracking. Tests were performed on two video sequences. Teir lengts were 30 and 50 seconds, respectively. First sequence was taken from te recording of andball matc and was used to evaluate reliability of te algoritms by counting te uman interventions required to maintain errorfree tracking process. Te second one was te test sequence, in wic players were standing still. Te measured distances terefore directly correspond to noise added by particular tracking metod. Additionally, te processing time per frame on PC equipped wit 500 MHz Pentium III processor was measured. Te results are sown in Table. Table : Comparison results for tree different combinations of tracking metods. Metod Interventions Noise Time A meters 0.44 sec/frame B 49 meters 0.75 sec/frame C 4 55 meters 0.9 sec/frame Te advantages and disadvantages for eac of te metods can be clearly seen. Te motion detection (A) introduced little noise to te traectories. On te oter and, more tan one intervention per one second of playing time is required, wic puts enormous pressure on G G B d d + d B First Int l Worksop on Image and Signal Processing and Analysis, June 4-5, 000, Pula, Croatia

6 uman operator supervising te tracking process. Tis approac is te slowest. Figure 6 sows te traectories obtained using tis metod. Te metod B, te color tracking, required small number of interventions, as expected, it is fast, but te traectories contain extremely ig amount of noise. Te combination C is nearly as fast as color tracking alone, contains low amount of noise and requires little intervention from operator. We can conclude tat te combination C is te most suitable for use in an automated player tracker. Fig. 6: Traectories of several players for te first few minutes of te matc, obtained by motion detection algoritm (A) and post-filtered using te Gaussian filter. 6. Conclusions We presented te development of te automated tracking system for tracking players in te sports games. Altoug te system was only tested in te andball matc, we can assume tat it would be equally useful in oter team sports, provided tat te cameras could be placed directly above te playing court. Te system is capable of outputting te positions of every player in court coordinates in every instance of time. It was used to track players in te andball matc to perform analysis of teir movement. It forms a base of te complete sports analysis system, wic will include advanced data processing, statistical analysis and presentation capabilities. Acknowledgements We wis to tank Marta Bon and Marko Šibila from te Faculty of Sports at te University of Lublana for teir valuable contribution. eferences. Aggarval, J. K. and Cai, Q. (997), Human Motion Analysis: A review, IEEE Nonrigid and Articulated Motion Worksop, pp Erdmann, W. S. (99), Gatering of kinematic data of Sport Event by Televising te Wole Pitc and Track, Proc. of 0t ISBS symposium, pp Intille, S. S. and Bobick, A. F. (995), Visual Tracking Using Closed- Worlds, ICCV Swain, M. J. and Ballard, D. H. (99), Color Indexing, International Journal of Computer Vision, Vol. 7, No., November Tsai,. Y. (987), A Versatile Camera Calibration Tecnique for Hig-Accuracy 3D Macine Vision Metrology Using Off-te-Self TV Cameras and Lenses, IEEE Journal of obotics and Automation, Vol. A-3, No. 4. First Int l Worksop on Image and Signal Processing and Analysis, June 4-5, 000, Pula, Croatia