Raport końcowy Zadanie nr 8:

Transcription

1 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Raport końcow adae r 8: Przeprowadzee badań opracowae algortmów do projektu: adae 4 Idetfkacja zachowaa terakcj Opracowal: dam Gudś Marek Kulback Jakub Roser Jakub Sege Vtal Talaow Kaml Wereszczńsk atwerdzł: Herk Josńsk Podstawa opracowaa: Projekt rozwojow O R00 00 : astosowae sstemów adzoru wzjego do detfkacj zachowań osób oraz detekcj stuacj ebezpeczch prz pomoc techk bometrczch ferecj postac w 3D z wdeo.

2 . Objectve ad scope of the task Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu The objectve of ths task s the developmet of methods that do ot rel o a skeleto based bod model for represetg the moto of artculated fgures mage streams, ad usg such skeleto-free represetato of moto for detfcato of behavor ad teracto of the observed fgures. Ths goal arses from the followg two observatos: )whle a skeleto-based represetato gves good results uder a correct mappg of the mage to the skeletal model, a correct mappg gves ver poor recogto results, )a good mappg s dffcult to obta a ucotrolled magg evromet wthout hgh cotrast betwee the fgure ad the backgroud, whch s tpcal outdoor scees. The premse beg followed s that, such evromets, better results ma be obtaed wthout relg o a skeletal model tha f usg t.. Report of the task Secto. descrbes the proposed skeleto free represetato for moto of artculated objects, based o paths of local features, that serves as a gudele for the developmet of methods ths task. Secto. ad.3 address usupervsed learg the spaces of feature paths ad feature path segmets: Secto. descrbes a approach to clusterg of local feature paths based o smlart of feature moto ad Secto.3 descrbes clusterg of feature path segmets that leads to a smbol strg represetato of a feature path. Secto.4 descrbes developed methods ad programs for costructo of feature paths, based o SIFT ad SURF algorthms. Secto.5 descrbes developed methods ad programs for costructg feature paths based o domat pots of cotours. Secto.6 descrbes developed methods ad programs for clusterg local feature paths ad feature path segmets, followg the approach preseted. ad.3, ad demostrates a automatc costructo of a represetato for artculated moto the form of a collecto of smbol strgs, as proposed Secto... Skeleto free represetato of movg artculated objects local feature s a localzed characterzato of a mage, more precsel a fucto of a eghborhood of a pel. Itall, cosder ol local features wth a oretato parameter wll be cosdered. Features wthout oretato or wth a partal oretato formato ma be cluded later. local feature s characterzed b the followg parameters: - tpe, detfg the detector or fucto used to etract t - posto, ) - oretato agle - attrbute, a scalar or vector of values that are depedet of posto ad oretato

3 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu The tall used local feature wll be a corer, or a crtcal pot of the curvature of the cotour of a mage segmet. Its attrbute wll be a scalar related to the curvature ad versel related to the corer's agle. SIFT lke local features ma be cluded later. local feature tmele LFT) s a sequece of correspodg local features computed at cosecutve vdeo frames. It descrbes the evoluto of a local feature tme. LFT s descrbed b a sequece of the local feature values, augmeted b the tme derved parameters: veloct ad accelerato vectors or the posto, agular veloct ad agular accelerato for the oretato, ad the dervatve of the attrbute. The magtude of the vector dervatves ad the scalar dervatves eted the attrbute part of local features. LTF has a start ad a ed tme. fasccle s a budle of LFTs whch are grouped together accordg to a gve clusterg crtero. fasccle usuall follows a movg object or perso a sequece of mages. descrpto of a fasccle cossts of ts compoet LTFs, ts ow posto ad oretato values whch are derved from the LTFs, ad the postoal ad oretato dervatves estmated from these values. To facltate the use of fasccle represetato for classfcato ad learg ts basc descrpto wll be eteded wth sequeces that are derved from the LFTs. derved sequece s called a derved feature tmele DTF). DTFs wll be created usg operatos of parwse composto ad clusterg. The composto appled to a par of LFTs wll create a DFT whose elemets wll be represeted smlarl to the LFT usg parameters of tpe, posto, oretato, attrbute ad ther dervatves. The fasccle wll be also treated as LFT ad wll partcpate the composto. The ew DFT wll be defed o a tme segmet whch ts compoets overlap. Clusterg wll be appled to the attrbute porto of LFT or DFT, ad t wll result replacg the attrbute wth a smbol or the cluster de. structureless represetato of the fasccle or a movg object wll be a set of smbol sequeces over a commo alphabet, each wth a dvdual start ad ed tme, Fg. Damc tme warpg, Markov model ad HMM formulatos ca be appled to ths represetato, as well as smple hstogram based classfers. aghacvbsokmklsmvosa usghbsmkslsmbvcs kmsdoosdsomwvsadmsjsgswqrjwu9wkm mohwrtas6ksbwwhk sdkosdwk8sgbwed9sake Fg. Structureless represetato of a movg object 3

4 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu structural represetato adds a drected acclc graph whose edges lk a DTF wth ts compoets. Such represetato wll make possble to use graph based learg methods.. Clusterg local feature paths.. Feature paths path represets the moto of a local feature that begs the past ad eds at or before the curret feature. The path structure cotas, ad tme coordates for each feature pot alog the path as well as a de to the cotour object for each pot. esdes, t has the tme whe the path bega, the legth of the path, the boudg spatal rectagle cotag the path ad status bts to dcate whether the path structure s free ad f the path has eded. path at tme t s represeted b P k, t) where k s a de to the path whch vares from to p t where pt s the total umber of vald paths at tme t ). P has actve, assged, l, c, c,,,,,,,...,,, ) compoets t t l l tl where actve [0/] dcates f the path s actve or has eded, assged [0/] set to meas that t s assged to a cluster, l s the legth of the path ad,, t ) are the spato-temporal coordates of each pot o the path. The values c, c) gve the mea dsplacemet of the path from ts assocated cluster, whch s eplaed later the cluster updatg step. efore the path assged to a cluster c ad c are 0... Clusterg feature paths s a perso moves the evromet, feature pots are detfed each frame ad tracked across frames. The result s a umber of paths correspodg to the perso that are tpcall short-lved ad partall overlappg tme. The paths are lke short threads that dcate the moto of dfferet pots o the perso at dfferet tmes. To track the moto of each perso the evromet, these paths are grouped to clusters, such that each clusters whch s meat to represet the moto of oe perso. cluster has a represetato that s smlar to that of a path, wth addtoal parameters eeded to descrbe group propertes. cluster at a tme t s represeted b k, t) where k s a de to the cluster ad vares from z to t z where t s the total umber of vald clusters at tme t. cluster has compoets l, b, b, b, b, bk,,, t,,,, t,,...,,, t, ) where l s the legth of the l l l l cluster,,, t ) represet the spato-temporal coordates of each pot o the cluster ad represets the umber of feature pots cotrbutg to the th cluster pot. The par b, b) dcates the mea dsplacemet of the posto of a path cotrbutg to a cluster from the coordates, ) of the cluster. The par b, b) dcates the varace the dsplacemet of the posto of a path cotrbutg to the cluster from the cluster coordates. The value bk represets the total umber of paths cotrbutg to a cluster. 4

5 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Whe the path legth eceeds a mmum, t s checked to see f t has a cluster assocated wth t. If so, the cluster s updated. The overlap of the path ad the cluster s checked to see f the legth of the cluster eeds to be eteded. If so, the legth of the cluster s cremeted ad the last pot of the path s added as the last pot of the cluster. Else, the last pot of the path s used to update the last pot of the cluster. I the followg we use the otato Comp to mea the compoet Comp of the cluster, ad P Comp for the compoet Comp of the path P, for eample p value of coordate at tme p ) of path P. P meas the compoet p If a path P of legth p has a cluster of legth q assocated wth t, the cluster update proceeds as follows. If ad P = the cluster s updated as t p t q P Pc P P p q q p c ' ' q q, ) =, ) ) q q q q ' = ) q q where ' ', q q ' ad represet the updated values. P q c, Pc are calculated whe the path s frst assocated wth the cluster as dscussed later ths secto. If P the cluster s updated as t p t q q ' = q 3) q ' = P P 4) p c q ' = P P 5) p c = 6) ' q s see from equatos through 6, the cluster update volves computg ew mea values for ad coordates of the cluster based o the curret path ad the estg mea ad cout values. If the path legth eceeds a mmum ad the path does ot have a cluster assocated wth t, the curret clusters are searched to fd the cluster closest to the path. To fd the closest 5

6 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 6 cluster to a path, we proceed as follows. Frst, the overlap legth l of the path ad a cluster are computed. The, the sum d of the mea squared dffrece betwee the ad values of the path ad the cluster s computed over ther overlap legth. =0 =0 )) )) = b r a l b r a l P l P l d 7) where r t a t P = represets the frst stat of overlap of the path ad cluster. Net, aother dstace dt s computed as the ormalzed sum of the mea squared dffereces of the ad compoets of the taget vectors alog the path ad the cluster over ther overlap legths. l P P M P P dt r r l a a l r a r a l )) ), ] ) ) [ = =0 =0 =0 8) Thus, dt s ormalzed for the values of the tagetal compoets ad for the overlap legth. The taget at each pot alog the path or cluster s estmated as a dfferece betwee the outputs of two lear predctve flters appled to the ad compoets of the path or cluster trajector opposte drectos as show below. Here, we dcate the computato ol for the compoet of a path. p P P P =,,..., = 9) where p s the legth of the path ad ) = P P P P 0) p p P = P ) ad ), = P P P P ) = P P 3) If the dstaces d ad dt le wth certa bouds, the fal dstace betwee the path ad the cluster s computed as

7 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu D = Ma ' b ', d ) Ma b, d ) dt 4) where,, ad are weghtg factors for the, ad tagetal dstaces. ad are chose based o tpcal dmesos of a perso the mage. The values d ad d are computed as follows. l = P ) 5) l a r =0 l = P ) 6) l a r d =0 = b) 7) d = b) 8) The par, ) gves the mea dsplacemet of the path P from the cluster over the overlap legth. The values d ad d are squared dffereces betwee the mea dsplacemet, ), of the group of paths cotrbutg to the cluster, ad the mea b b ' ' dsplacemet, ) of path P. The values b, ad b gve the ew varace of path dsplacemets from the cluster f P s merged wth. The are computed usg equatos 6 ad 7 below. Thus, D equato 4 measures the dsplacemet of the path from the cluster, the spread of the paths cotrbutg to the cluster ad weghts these wth respect to the tpcal dmesos of a perso. esdes the dfferece the drecto of moto alog the path ad the cluster are measured equato 4. The cluster wth the smallest value for the dstace D from the path provded the dstace s wth a boud) s chose as the earest cluster to the path. The path s the assocated wth ths cluster ad the cluster parameters are updated based o the path parameters as follows. Over the overlap legth,.e., wth varg from 0 to l, the updated values are gve b Pc Pc = 9) = 0) P P c P P ' ' r r a r r a c, ) =, ) r ) r r r 7

8 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu ad ' = ) r r ' bbk Pc b = 3) bk bbk P ' c b = 4) bk ' bk = 5) bk ' b = b b bk ) bk ' b ) 6) ' b = b b bk ) bk ' b ) 7) where, are gve as equatos 5 ad 6. If the path legth p eceeds the overlap legth l of the path ad cluster, the cluster s eteded as follows wth varg from l to p r, r ) = P a P, P P ), = 8) c a c r If o estg cluster ca be assocated wth the path P, a ew cluster s created ad assocated wth the path. The tal parameters of are r, r ) = P a, P a ) 9) = 30) r = 3) bk 0, = 0 3) b = b 0, = 0 33) b = b fter assgg paths, the clusters are checked for possble merges. For each cluster, all the remag clusters are searched to fd the earest cluster. Ths search s smlar to the search for the earest cluster to a path, descrbed above. If there s a cluster close eough, the two clusters are merged. The process s repeated tll o more merges are possble. If ad are clusters, the followg equatos are used for ther tal comparg. 8

9 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 9 =0 =0 )) )) = ), b r b a l b r b a l l l d 34) where r t a t = represets the frst stat of overlap betwee these two clusters. The tagetal dstace ), dt s computed usg the taget sequeces of the clusters. l M dt r r l a a l r a r a l )) ), ] ) ) [ = ), =0 =0 =0 35) If the dstaces ), d ad ), dt le wth certa bouds, the fal dstace betwee the the clusters s computed as ), )),, )),, = ), dt d Ma d Ma D b b 36) where s the cluster resultg from the opererato of mergg clusters ad whch s descrbed the et subsecto,,, ad are the weghtg factors defed before, ad ), d ad ), d are computed as: =0 ))) = ), b r b a l l d 37) =0 ))) = ), b r b a l l d 38)..3 Mergg clusters If for a cluster there are clusters whose dstaces d ad dt to cluster pass the gve thresholds, the the cluster whch s earest dstace D to, amog these clusters, s merged wth, resultg a cluster whose parameters are computed as follows. r a r b r a b a r ) ) = 39) r a r b r a b a r ) ) = 40)

10 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu a = 4) a r b b = = b b b) bk b b) b bk bk bk bk ) ) bk b b bk bk 4) 43) bk = 44) bk bk = 0 45) b = b Notce, that the cluster resultg from mergg has the posto sequece at ts ceter, ad the bas values, are 0. b b..4 Idetfcato of vald clusters Whe a cluster has o updates for a suffcetl log tme, t s safe to assume that the perso assocated wth the cluster has moved out of the vew of the camera. t ths stage, the cluster ca be destroed. efore a cluster s destroed, t s checked to see f t s vald. cluster s vald f t has a legth greater tha a mmum ad has at least a mmum umber of feature pots cotrbutg to t. The trajector or track correspodg to each vald cluster s passed o to the approprate applcato program. The trajector cossts of a sequece of, ad tme coordates correspodg to the cluster..3 Clusterg Paths Segmets.3. Path segmets ad ther clusters path at tme t s represeted b P k, t), k s a de to the path whch vares from to N where N s the umber of paths a database used to buld clusters, ad t vares from 0 to, where l k s the legth of the path k. Spatal coordates of the path pot P k, ) are l k, ). path segmet of legth T s a T -log subpath take from a path whose legth s T. dstace D, ) betwee two path segmets ad of equal legth T s D, ) = T T =0 [[ m ) m )] [ m ) m )] ] ) 0

11 where m m Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu,,, are the spatal coordates of pots of ad, ad ),,, are the mea values of the coordates of ad. For eample m computed as: m m s T = ) m T = 0 If there are pots wth mssg values a path segmet the the mea coordates are computed over the vald pots, usg, stead of, where T v ) s the umber of vald T T v ) pots, ad the dstace D, ) s computed over tme dces where both ad have a vald pot, ad usg usg the umber of such dces T v, ) stead of T. cluster of path segmets s a group of path segmets P, P,... of detcal legth T. The path of the cluster s the mea of the paths of the cluster members, after subtractg ther mea, that s, f C s the path of a cluster ad C sze s the umber of path segmets ths cluster, the spatal coordates of the pots of C are computed as C C = C sze = C sze C sze j j P Pm ) 3) j= C sze j j P Pm ) 4) j= Mssg values are hadled as t was descrbed above. dstace betwee a cluster whose mea path s C, ad a path segmet P s smpl the dstace D C, P). dstace betwee two clusters whose mea paths are C ad C s D C, C )..3. Clusterg path segmets smple cremetal clusterg method takes path segmets oe b oe ad ether adds t to oe of the curret clusters or creates a ew cluster. - compare a ew path segmet wth all curret clusters - select the cluster C wth the smallest dstace to. - f D C, ) < DThreshold add to C ad update the path of C otherwse create a ew cluster whose sgle member s

12 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Updatg of the cluster path ca be doe cremetall..3.3 uldg path segmet clusters from a path database For a chose segmet legth T start at the frst path ad pass to the clusterg fucto the path segmet whch begs at the de 0, the the segmet begg at the de, ad repeat ths step up to the de l T. The process the same wa the remag paths the database Mergg clusters Cluster mergg s a clusterg operato appled to a set of clusters stead of path segmets, usg the cluster dstace fucto D C, C ). Ru cluster mergg o the set of the curret clusters ether at selected pots of the process, for eample ever Nmerge paths, or at the ed. The clusterg process should be repeated usg segmet legths T to ma, T s, T s... up m m m T. The values T, T, s DThreshold ad Nmerge are parameters of the method, to be set b the user. m ma,.4 uldg feature paths wth SIFT ad SURF Ths secto descrbes developmet of methods ad software mplemetato for buldg feature paths from mage streams, usg SIFT ad SURF for detectg ad characterzg local features. To evaluate these methods a test evromet has bee costructed wth the followg objectves:. Eablg to buld test applcatos for aalss of dfferet feature detectors ad descrptor etractors vdeo streams.. llowg the user to choose SIFT or SURF as a detector algorthms. 3. llowg the user to choose SIFT of SURF as a descrptor etractor algorthms. 4. llowg the user to set all the tal parameters a specal cofgurato fle 5. Create a mult-platform Wdows,Lu), CMake-based project cofgurato for the test applcato

13 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu.4. Test applcato for aalss of dfferet detector/descrptor algorthms Schematc represetato of the applcato I the course of ths task a test applcato has bee created wth the followg schematc:. Read the tal parameters from the cofgurato fle detectors.tt. Read the frst frame from the mage sequece, vdeo fle, or scramble sequece 3. Choose ol oe algorthm for selectg features ad descrptor etracto 4. Ope/create paths fle 5. For each cosecutve mage a. Read ew frame b. Detect ew kepots usg specfed detector algorthm c. Etract descrptors for kepots usg specfed descrptor etractor algorthm d. Create a cost matr betwee curret paths ad ew features e. Match features to paths f. pped matched features to paths g. Create ew paths from umatched features h. pped bad pots to umatched paths ad cremet the umber of mssed frames for ths path. Remove paths that have umber of mssed frames larger tha the user defed umber j. Draw paths o the source mage usg dfferet colors for dfferet path legths k. Show source mage wth paths a wdow 6. Release allocated data ad structures.4. Processg applcato for creatg paths for further aalss algorthms Schematc represetato of the applcato I the course of ths task a test applcato has bee created wth the followg schematc:. I ths mode user has a possblt of settg some specfc parameters through commad le argumets, otherwse all the parameters wll be read from cofgurato fle the et step. There should be eght commad le argumets: a. Path to the crumble fle alog wth the last backslash e.g e:\vdeos\crumbles\ b. Fleame wthout eteso ad evetual fle umber. Ths s the part commo for all the crumble fles to be processed. Eg c. Frst crumble fle umber e.g. 0 d. Last crumble fle umber e.g. 9 e. Eteso of crumble fles e.g..csq 3

14 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu f. Path to a place where the feature paths should be stored a fle e.g. e:\vdeos\crumbles\out\ g. Eteso of the paths fle e.g..tt h. f we wat to dspla wdows wth results.e. vdeo frames wth marked features) or 0 f we wat to see ol the cosole output.. Read the tal parameters from the cofgurato fle detectors.tt. Ulke the test applcato, the tal parameters processg applcato are costat throughout the whole rutme ad caot be chaged. 3. Read the frst frame from the scramble sequece 4. For each cosecutve mage a. Read ew frame alog wth ts Regos Of Iterest ROIs). Feature pots wll ol be detected ad matched sde those ROIs. b. For each ROI: a. Detect ew kepots usg specfed detector algorthm b. Etract descrptors for kepots usg specfed descrptor etractor algorthm c. Create a cost matr betwee curret paths ad ew features d. Match features to paths e. pped matched features to paths c. Create ew paths from umatched features d. pped bad pots to umatched paths ad cremet the umber of mssed frames for ths path e. Remove paths that have umber of mssed frames larger tha the user defed umber f. Draw paths o the source mage usg dfferet colors for dfferet path legths g. Show source mage wth paths a wdow f eeded h. pped approprate le to the paths fle 5. Close the paths fle 6. Release allocated data ad structures Cofgurato fle Ths paragraph cotas the cofgurato fle where each parameter s shortl descrbed: ############################################################ # efore rug the applcato: # set approprate paths to both ths fle detectors.tt) # fucto test_detectors). It s best to # provde full paths. The appl # a chages You eed to ths fle most mportatl # Vdeo I/O parameters wth fle paths!!) ############################################################ #!!IMPORTNT: SLIDER_MOD=/N meas that the value set # a slder for ths parameter wll be dvded # b N. # e.g. f SLIDER_MOD=/000 ad we set slder 4

15 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu # to posto 943, the actual parameter wll # be equal 943/000 = 9.43 # Ths s due to the fact that a slder # value ca ol be usged ad teger. ############################################################ # Vdeo I/O parameters ############################################################ # Set ths parameter to: # 0 f You wat to use vdeo fle # for mage sequece # for crumble sequece mage_mode = ; # INT Def:0, M:0, Ma: ########################################################### # VI FILE # Note t s best to specf the full path alog wth drve letter # avpath = ########################################################### # IMGE SEQUENCE # Image path alog wth the frst part of the fle ame all before the umbers) # mseqfrst = /frame_ # Number of dgts the frame umber represetato # mseqnumdgts = 6 # INT Def:6, M: Ma:0 # Last part of the fle ame alog wth the eteso all after the umbers) # mseqlast =.pg # IMPORTNT!!! # Set the approprate umber of mages ths sequece # Note: Frame umbers should be umbered from 0 to at least $vdeoframecout vdeoframecout = 500 # INT Def:500, M: ########################################################### # CRUMLE SEQUENCE # crumbleseqpath = e:\vdeos\crumbles\ csq ########################################### # OUTPUT # Mode for drawg paths # 0 - pat both les ad pots # - draw ol les # - draw ol pot sequece pathdrawmode = ; # INT Def:0, M:0, Ma: # Magfcato of the output wdow outmagffactor = 0.5f; # FLOT Def:.0, M:0.,Ma:0 # set to f You wat to save the vsual results saveresults = 0; # set to f You wat to save the paths for each frame 5

16 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu savepaths =; # used for storg the output frames f saveresults== # mseqfrstout = /frame_sift_ ########################################################### # Processg oe frame over ad over. The purpose of ths mode s to aalse the effects of certa parameter chages # o the whole detecto/matchg/vsualzato process. # To go to the et frame whle beg ths mode use the "et" butto. PlaceMode = 0; # I detector mode there are o descrptors computed ad therefore there are o paths # The kepots are ol chose ad show per each frame detectormode =0; # f =, the kepots wll be detected ol per-frame ad/or user-draw regos of mage/frame # f = 0, the kepots wll be detected the whole mage/frame useroifordetecto = ; # Set the tal userdefedroi # settg to - meas there wll be o userdefedroi specfed userdefedroi. = -; userdefedroi. = 400; userdefedroi.wdth = 350; userdefedroi.heght = 50; # Mamum umber of Frames Per secod mafps = 00; # INT Def:60, M:0, Ma:0 # Mamum cost allowed for matchg SIFT descrptors macostthresh = 50; # INT Def:50-00, Ma:000 # Mamum path legth mapathlegth = 0; # INT Def:0, M: # Mamum umber of cosecutve mssed frames path largest gap betwee two vald pots) # Note ths umber has to be equal or lower tha mapathlegth!!! mamssedframesipath = 5; # INT Def:5, M:0 Ma:$maPathLegth)- # Mamum dstace from ew pot to predcted et pot path epressed percetage of the vdeo frame's dagoal) # e.g. for vdeo sze of the dagoal s ~03 # f set to 0 the ma dstace wll be equal 0.3 percmadstfrompredctednetpot = ; # INT Def:5, M:0, Ma:00 # Mamum dstace from ew pot to predcted et pot path epressed meters) currmadstfrompredctednetworldpot = 9.5; # Def: 0.5 SLIDER_MOD=/0 # Those are the weghts for all three compoets for computg the fal cost matr value # NOTE: Usuall You wat to keep desccostmodfer oe order of magtude hgher tha the rest desccostmodfer =50; 6

17 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu dstacemodfer=; aglemodfer=3; # Ital marker postos for cached sequece markerfrst = 00; markerlast = 40; # Detector algorthms # 0 - SIFT default) # - SURF detectorid = 0; # Descrptor algorthms # 0 - SIFT default) # - SURF # Note: case of SURF descrptor, the cost matr values are multpled b 00 to be smlar rage to SIFT descrptorid = 0; ########################################################## # SIFT parameters ########################################################## # commo SIFTOctaveLaers = 3; # Def: 3 # detector SIFTThreshold = 0.04; SIFTEdgeThreshold = 0.0; #descrptor SIFTSgma =.6; # Def: 0.04 SLIDER_MOD=/000 # Def: 0.0 SLIDER_MOD=/000 # Def:.6 SLIDER_MOD=/000 ########################################################## # SURF parameters ########################################################## # commo SURFOctaves = 4; # Def: 3 SURFOctaveLaers = 4; # Def: 4 # detector SURFThreshold = 750; # Def: Feature pot matchg efore matchg ca beg we eed to create a cost matr betwee each path ad each ew feature usg three compoets the followg formula: 7

18 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu ) ) ) ), where desccm, dstcm ad aglecm are costat modfers through whch the user ca specf the relatve fluece of each compoet o the cost matr. The three compoets are as follows:. Descrptor cost: ) ) )), where P path vector, - path de, F ew feature vector, ew feature de, descrptor drecto. Note that f the ew feature s too far from predcted posto path the cost s set to ft.. Dstace cost: ) ) )), where P path vector, - path de, F ew feature vector, ew feature de, L,) L eucldea) dstace betwee pots ad 3. gle cost: { ) ) )) ) ) )) ) )) ) ) )), where P path vector, - path de, F ew feature vector, ew feature de, agledff,) dfferece of agles betwee pots ad For matchg features we have chose the Hugara algorthm that mmzes the global cost whle preservg oe to oe matchg coveto..4.4 ppedg matched features to paths ) Italze The umber of old paths ) Italze formato that the paths have ot receved a ew pot for ths frame 8

19 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 3) If matched, apped pots to approprate paths otherwse create ew paths from them 4) Check f paths have alread receved a pot for the curret frame. If ot, push a vald bad) pot to ths path. ddg a pot to a path wth ether or posto equal - meas that a matchg pot has't bee foud for that path the curret frame. If the path has't receved a vald pot last mamssedframes) remove ths path from the path vector 5) Compute the predcted posto of et pot based o the postos of two most recet pots usg blear etrapolato Drawg paths ) Draw paths usg oe of the three modes of drawg paths: a. Draw both, les betwee cosecutve pots path ad mark actual pots b. Draw ol les c. Mark ol pots.4.5 Results I Fgure below two tpes of ROIs ca be see: ) Gree rectagles Per frame ROIs defed the crumble fle ) Red rectagle User-defed ROI Note that ew features are foud ol the tersectos) of a gree ROI ad the userdefed ROI, thus o feature has bee foud the upper gree ROI. I the mage below the perso, alog wth ts gree ROI, moves left whch ca be easl spotted b aalzg the relatos betwee paths multcolor les), ew features dots colors correspodg to approprate paths) ad the gree, movable ROI. 9

20 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fgure Eample of usg Regos Of Iterest ROIs) Fgure 3 presets applcato s GUI wdows alog wth slders used to chage certa parameters. Note that each slder ame correspod to a cofgurato fle parameter wth the same ame for clart alog wth ts modfer see cofgurato fle secto). 0

21 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fgure 3. pplcato GUI wth slders

22 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fgure 4. Cosole wdow Fgure 4 presets the cosole wdow that cotas addtoal useful formato about each frame lke processg tme splt to several self-eplaator steps, total processg tme for that frame, umber of kepots features) foud ad umber of paths. Furthermore t shows the user-defed ROI s coordates. Note that f the frst ROI coordate equals - t meas there s o user-defed ROI set.

23 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fgure 5 shows some eamples of trackg Fgure 6. Perso trackg eamples 3

24 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu.5 uldg local feature paths b trackg domat pots.5. Itroducto Trackg objects o vdeo s oe of the mportat attrbutes of the comple problem of object/huma behavour aalss ad terpretato. There est a lot of algorthms of object/huma acto aalss whch have bee bult mostl as herarchcal algorthms of aalss of actos startg from ver smple oes to more complcated []. The most advaced of them use probablstc approaches such as Graphcal models aesa Networks N), Damcal aesa Networks DN) ad Radom Felds RF) that have a mportat subclass of Markov Radom Felds MRF)). ll such approaches do ot use the formato about object trackg ad ofte utlze full formato about object whch s avalable o vdeo frames. s kow Graphcal models usage could be ver epesve sese of computg whe a umber of radom hdde) varables s large [], [3]. O the other had ofte to compute a value cotag ever radom varable eeds use all formato from object ad case of Full HD vdeo wth hgh frame rate FR) s crtcal. other approach that refers to behavour aalss s based o aalss of tracks. I track buldg there est several prcple approaches. The most mportat are based o Kalma flter ad partcle flter as geeralzato ad eteso to Kalma flter []. oth of them are of probablstc character ad ca predct the damcs of a object. The dfferece betwee them cossts lmtatos of Kalma flter that eeds model be lear ad ose eeds be of Gauss character. Partcle flter uses samplg of posteror dstrbuto havg observatos ad to track objects wth ts applcato case of Full HD vdeo ad hgh FR s problematc. Oe more group of trackg algorthms do ot use a probablstc o dog trackg. The algorthms from these group geerate a umber of kepots ever of them to be assged to separate path f the assgmet s vald. the path we uderstad a sequece of pots matched o two cosecutve frames ad all these pots defe a trajector of a object. mog algorthms producg local feature pots we ca meto SURF Speeded up Robust Feature), SIFT Scale-Ivarat Feature Trasform) [] ad IPN algorthms [6]. ecause IPN algorthm geerall fds much less pots ad each pot could be much more formatve average) tha pot geerated b SIFT ad SURF we use t as feature geerato algorthm for trackg. IPN algorthm fds kepots o the cotour of a object ad belogs to geometrcal approach to feature geeratg. Frst of all we separate backgroud from mage usg Gaussa Mture Model GMM) [4]. Ths allows us to fd zoes of moto,.e. objects that move ad whch wll be used for further processg. To obta cotours of objects we do ther segmetato before. Ths guarates that we are gog to have ol eteral cotours of objects ad ot teral oes that could be case of Ca detector applcato [4], [5]. Ca flter uses gradets ad fds all cotours of a object. Kepots are foud place where curvature of the cotour segmet s more tha some gve value. fter fdg crtcal pots we use Hugara algorthm for assgmet of correspodg pots. Ths 4

25 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu algorthm refers to the damc programmg problem. I the et sectos we show how to use the algorthm whch cossts of dfferet stages ad some optmzato trcks also. For a computg we use C++ for Wdows MS VS 00) as well as for Lu Ubutu wth lbrares Ope CV, Ope GL, T, QT, OOST, Ope Threads ad others..5. Vdeo preprocessg To perform object motorg o mages t s ver mportat to have vdeo preprocessg stage realzed ad optmzed. Vde preprocessg gves the possblt to etract useful ad terestg formato from mages. vdeo preprocessg we mea the followg operatos o vdeo: backgroud subtracto, object segmetato, cotour detecto, cotour flterg ad domat pot detecto. For backgroud subtracto we use three operatos. Frst operato computes the foregroud mask after we compute backgroud mage. The we subtract ths backgroud from the grascale mage to obta a mage wth motos o the black backgroud. I more detals the task of backgroud subtracto could be preseted b the followg wa. The fal purpose s to evaluate the value of the bar label w {0, }, whch shows that th pot the mage s a part of the ukow backgroud w 0) or belogs to the object of the foregroud w ) that occludes some part of the backgroud mage. Ever pot of the mage geeral case could be descrbed color space RG. For metoed algorthm t s ver mportat to have a trag set of data I N { }, composed from empt scees where all pots belog to backgroud ad ths s true. O the other had t s ot commo to have eamples of objects from foregroud that are ver varable. Thus f a pot belogs to backgroud w 0) the the data are geerated from the ormal dstrbuto wth kow mea 0 ad varace 0. Whe some pot belogs to the object of foregroud w ) the t s assumed that the data are dstrbuted accordg to ut dstrbuto: Pr Pr w 0) N w ) k, [, ]; 0.) where k s some costat. fter that we do operatos o obtaed mage usg fuctos from Ope CV lbrar to make segmetato of objects ad the we fd cotours for segmeted objects. Segmetato of the scee of the mage s performed b the threshold level. If for eample we use oe threshold the we obta bar mage. fter that for bar mage oe apples the operator of the cotour detecto. We detect ol eteral cotours of objects that characterze the shape of 5

26 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu these objects ulke the gradet Ca operator [5] whch detects all eteral ad teral) cotours o the mage. For tall foud cotours we appl crcular averagg flter to smooth cotours. Ths s eeded because of possble local sharpess of cotours where the IPN algorthm could fd umerous of local domats what are ot formatve at all. Such a flter realzes the averagg operato both o two coordates a wdow of a gve sze. To solve the feature etracto problem o the vdeo or mages the cvfddomatpots fucto has bee take from the Ope CV lbrar Vsual Studo 00. I ths fucto the IPN algorthm s used that was developed b Dmtr Chetverkov & al. [6]. The dea of ths algorthm s how to buld the tragle sde of the partcular cotour. So the algorthm cossts descrbg the curvature b some kepots amed domat pots. ll cotours are gathered the sequece that has structure of a tree. ecause of ths we have 4 parameters 3 sdes ad the agel of our tragle) for the tug the fucto the best wa to solve our problem. s we have the law of tragle a metrc space two sdes together could ot be less tha the thrd oe) we have to select the parameters of the fucto a ver proper wa. lso we have to remember that agel could ot be more tha p 80 degree). The domat pots cota formato about the shape of the cotour ad are more formatve the others. The other mportat thg s to select the most stable domats to have the possblt the to buld the tracks of the target a vdeo. Havg such domats we ca coect them b les of dfferet orders sples) thus havg appromate cotours. Ths could be useful whe we would lke to realze the compresso of formato preseted cotours. The total umber of dscrete pots cotours s much more tha the umber of domats, so the compresso rate could be ver hgh some stuatos. It s ver mportat to f the parameters of IPN algorthm a wa that produces domats the approprate place. Normall the should be place wth hgh local curvature ad the dstace betwee pots should be large eough to smplf matchg. If the dstace betwee a of domats some feature space) s essetall more tha the mamal shft of some pot for several cosecutve frames) the same feature space ths should satsf approprate assgmet. We cosder the possble assgmets durg several cosecutve frames because some domats could dsappear for some tme due to cotour chages. It should be otced that tal IPN algorthm s ot varat to scale,.e t s ot varat to the sze of object a vdeo. To have domats approprate place the sdes of a tragle that should be placed the teral segmet of the cotour deped o the sze of ths segmet. Kowg ths depedece we ca make the re-scalg of the tragle. Fg.7 shows how to buld the tragle sde the segmet of a cotour ad all parameters used the IPN algorthm. 6

27 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fg.7. Geometrcal vsualzato of IPN algorthm s we ca see ever tragle s characterzed b coordates of vertces p, p ad p. s coordate features of a domat pot p we use, ) ad two agles ad oretato p p agle betwee sde b ad the mea le of the agle of the tragle. Fall we have 4 features: two coordates ad two agles. The Eucldea dstace betwee two domats from the prevous frame ad p j from the et frame could be foud as follows: p N k d p, ) ) p j fk f jk ) 0.) where N k have N 4. f f k ad N f j f jk -- are sets of features of p ad k p j domats. Here we Havg d p, p ) we buld so-called cost matr that wll be utlzed for assgmet usg j Hugara algorthm. ctuall p s the last pot of each actve track see). However the predctg procedure could be mproved b the followg wa. Let estmated posto of the et pot s vector. So we ca decompose p e j p v, where v v ), v )) s the prevous veloct e e p o p ) p ) v ) ad p ) p ) v ). It e j j should be otced that f we have oe pot some track tha veloct of ths pot s equal to zero: v 0,0). The we use the followg Eucldea dstace to calculate the cost matr: j 7

28 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu d p, p j ) w d w p j ) p j ) e j )) w p j ) p j ) e j ) / w )) d w ) w ), 0.3) where coordates. w d, w ad w - are the weghts o locato coordates of domats ad ts agle Fall we ca predct the et speed value as e j v v p p ), 0.4) j where [0, ]. Predcto 0.4) could be wrtte for separate coordates ad as well. dog recurso to compute the et value of the track veloct we have averaged veloct value of the track at each momet of tme because of recurso. Such averagg works good f the perso goes slghtl wth the appromatel costat veloct. However f the perso chages drecto ad the speed ver ofte ad fast ths could lead us sometmes to sgfcat errors. ut geeral stuatos ths veloct predcto works suffcetl good ad allows us to receve better tracks tha stadard wa of cost matr buldg. cost matr has bee bult o the bass of two vectors of domat pots take from the prevous ad the curret frame. Domats from the prevous frame we call predctg pots sese of assgmet of pots o the curret frame. Ths s because pots from the prevous frame have alread bee assged ad are the last pots of each track we assume that assgmet has bee take place at prevous stage). So f we have two sets of domats s prev s from the prevous frame) ad et from the curret frame) we ca costruct the cost matr C of sze m, where s prev ad j m s et : C, j d p, p j ), p p where s the th pot from the prevous set of domats ad j s the jth pot from the curret et relatvel to prevous frame) set of domats. For fdg assgmets we use the Hugara algorthm whch works wth cost matr C. fter applcato of ths algorthm we obta the bar matr X composed wth zeros ad oes. Sze of the cost matr C should be f m or m m f m. The realzato of Hugara algorthm C++ works stuato whe m. Ths s doe { C } { } j b addg colums or rows wth zeros to acheve a cost matr j or { X } matr j of the same sze that a cost matr s flled as follows: C m m. bar 8

29 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu X j for successful assgmet; 0 otherwse. The propertes of ths bar matr are as follows: N j N N X,..., N; j X j,..., N; N j j CX j j m, where N ma, m). The dees ad j of each oe the bar matr correspod to the domats the prevous ad the et sets that are caddates for assgmet. The fal decso about assgmet wll be made f the followg codto s satsfed: Dstace d p, p ) d. j ma d ma should be chose as the mamal shft of a object huma) o the vdeo wth respect to FR. Ths could be doe kowg mamal veloct of a object of terest wth respect to the calbrato parameters of vdeo camera. It should be otced that algorthm that bulds the paths should have the followg fuctoalt: The assgmet should be eecuted ol for actve paths. actve path we uderstad some path whch had o assgmet ot more tha durg several last frames ths parameter could be optmzed durg trag process). Otherwse the path becomes actve. The ed of the track s the momet whe the track s actve ad the last pots of ths track wll ot be checked for assgmet. The pots whch have ot bee assged are the beggs of ew tracks. The optmzed algorthm bulds paths a wa that the legth of such paths s as log as possble ad there are ot too ma paths for each separate object. 9

30 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu.5.3 Epermets ad tests The results of IPN algorthm applcato for the statc mages have bee offered below. To fd the optmal parameters of cvfddomatpots fucto we take some statc bar prmtves that are used to test the IPN algorthm. I statc bar mages t s eas to fd the optmal parameters because t s eactl kow where the domats should be. d for the bar mages the two thresholds are ot mportat. So ths case we have 5 parameters for regulato: the sze of the wdow of averagg crcular flter ad four parameters of cvfddomatpots fucto. So we put all the parameters o the slders. The order of the parameters s as follows -st parameter s the sze of averagg wdow; d- 4th s sze of the tragle; 5th s the agle of tragle. The base set of optmal parameters f we use averagg flter s 0, 30, 40, 35, 75). Other sets could be take as follows 0,30,35,35,75), 0,30,35,40,75) ad 0,35,40,35,75). If usg a flter the sze of the wdow mght be chaged ot more tha from 0 to 5. There are the eamples for dfferet bar mages wth the base optmal set ad others. I ths fgures the orgal cotour s marked b red color ad fltered as blue. Domats are marked b gree crcles. There was created the database of mages wth domat pots for dfferet sets of cotrol parameters for 8 test bar mages, take as oes whch are used to test IPN algorthm. 0,30,40,35,75) 0,30,35,35,75) 30

31 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fg.8. The results of domat pots search o statc mages havg dfferet forms a) b) Fg. 9. Image wth relatve degree of ose equal to 0.0 a) ad deosed oe wth 3

32 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu cotour detectg ad smoothg b) а) б) Fg. 0. Image wth relatve degree of ose equal to 0. a) ad deosed oe wth cotour detectg ad smoothg b) Some tmes scees o mages could be occluded b small objects or have some addtve ose due to atural factors ra, sow, mst etc.). I Fgs. 9, 0 the results of algorthm work whch detects ad smoothes cotours o mages wth dfferet degrees of addtve ose are preseted. For mages wth relatvel hgh degree of ose operators of eroso object coectvt search) should be appled. Ths s due to segmetato b thresholdg that creates a lot of os objects all almost smaller tha targets our case two mltar plaes). 3

33 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Nos objects are ot coected to each other so the could be deleted ver effectvel leavg ol bg eough coected objects. Degree of ose determes the sze of the eroso operator wdow that deletes objects of approprate sze. fter applg algorthms of domat pots detecto drectl to cotours as case of IPN algorthm) or to segmeted objects as case of Harrs corer detector) we obta a collecto of such pots located o the borders of object or cotours. s t was metoed before, we ca jo them b les of dfferet order to have appromated cotours. Ths allows us to have formato about cotours compressed form. Let us deote umber of coected pots the cotour ad c the d the umber of domats. The the compresso rate of formato preseted cotours ca be wrtte as follows R c. d a) 33

34 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu b) Fg.. Cotour appromato usg domats: a) relatve degree of ose equal to 0.0 ad b) relatve degree of ose equal to 0. I Fg. we have two mages wth appromated cotours. For the frst mage we have compresso rate R 9 ad for the secod oe R 7. s see we have good eough appromato of cotours. Such cotour appromatos pursue the followg two objects. Oe of them meas storg less formato ad the secod oe s to have kepots places where the cotour s more stable to ose fluece. fter jog them we ofte have cotours closer to the real oe tha after smoothg. I our test we do ot use pots obtaed b Harrs corer detector marked b ellow color Fg.5) for cotour appromato. We use ol domats from IPN algorthm marked b gree color). I geeral we could combe them to acheve the best appromato wth mmal umber of kepots. 34

35 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fg.. Complete set of tracks plotted o the tal vdeo of the Market Square Fg. 3. Fltered set of tracks wth the track legth threshold equal to 5 The results of the secod part of the vdeo trackg problem testg are as follows. Ths problem s coected wth track buldg. We tested our algorthm o a vdeo clp of 50 sec. wth FR=5 take from the vdeo stream recorded from the Market Square of tom where a umber of dfferet actvt actos ca be regstered. I Fgs. -4 the ma wdow wth 35

36 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu vew of the Market Square from a sgle vdeo camera s show. ll tracks have bee bult these wdows. Fg. 4. Fltered set of tracks wth the track legth threshold equal to 50 Fg. 5. Ital scee Moto Capture lab) 36

37 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu I Fgs.5,6 trackg results from Moto Capture lab at Polsh-Japaese Isttute of Iformato Techologes PJIIT) have bee preseted. Fg. 5 shows the tal scee ad Fg. 6 shows the results of buldg of tracks. I Fg. 7 we ca see domats foud the cotours of two actors. Fg. 6. Complete set of tracks Moto Capture lab) Fg. 7. Domat pots plotted o fltered cotours Moto Capture lab) 37

38 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu s see from Fg. 6 whe perso rus t s dffcult to have good' cotours. Ths s because of lmted frame rate. The smplest decso could be made b movg dow the value of threshold of segmetato algorthm to have more clear cotours but t ca leads to appearace of shadows o the floor of the lab. lso ths results havg post-cotours whe dog backgroud separato. Ths s because of small movemets of perso or camera. Ths effect could be removed partall b settg the threshold to approprate value. Fg. 8. Domat pots ad ther oretato o cotours of actors ad ther shadows Fg. 9. Vsualzato of domat pots assgmet: red pots take from the prevous frame ad the blue oes are those that have to be predcted from the et frame the vdeo stream 38

39 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu I Fgs. 8, 9 the epermets wth vdeo of aother scee have bee gve. O Fg. 8 we ca see domat pots wth ther oretato o detected cotours. Fg. 9 shows the results of domat pots assgmet two cosecutve frames. I geeral case tracks ad cotours of ever perso should be rescaled o the bass of trsc ad etrsc parameters of a vdeo camera. s see from Fgs. -4 the track buldg s suffcetl good. To that ed we use flter o the track legth. Such a flterg gves a possblt to see f the legth of a gve value s acheved for each perso separatel. s see eve for the threshold of the track legth equal to 50 we ca see from oe to several local tracks for some persos. For threshold equal to 5 we ca see tracks for all persos. For threshold equal to 0 we ca see all the tracks. The legth of a track as oe of the ma characterstcs of the track bulder depeds o the umber of parameters. ll of them we put o slders trackbars) to have the possblt to cotrol them b a huma. The slders o the Cotrol Pael Fg. 0) cotrol the followg parameters of the geeralzed algorthm: thresholds for the segmetato algorthm; sze of a wdow for crcular averagg flter; 4 parameters for the IPN algorthm; weghts o agles characterzg domats from the IPN algorthm; dma ; assgmet gap the umber of frames where tracks are actve f there s o assgmet); wdow sze to calculate average speed o the track ed; flter for the track legth; scale for the parameters of IPN algorthm ad d ma depedg o the dstace from the vdeo camera. 39

40 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fg. 0. Cotrol pael.5.4 Summar The frst teratve step to solve the vdeo trackg problem has bee made. Ths method gves us the possblt to process the vdeo ad obta tracks real tme. Yes, for Full HD vdeos t s dffcult for the momet, but for vdeo wth less resoluto t s qute possble. Such a algorthm mght geerate small eough amout of pots o the movg objects mamum) that could be easl matched from frame to frame. The ma dsadvatage of the algorthm s that the domat pots are ot stable: betwee the assged frames, we ca have relatvel ver dfferet umber of pots f the total umber of pots s ot ver large 5-0 pots)) that cause the loss of formato o the oe had ad buldg useless assgmets o the other. Istablt of domats also leads to some addtoal tracks that should be avoded. Usg slders for lmtato of dstaces solves ths problem partall. fter such dstace flterg we lose a formato about some assgmets that mght be formatve. O the other had we mght use the algorthms for predcto of domats the et frame usg some predctg algorthms. Ths could be doe usg the hstor of the bult paths ad some addtoal parameters lke a veloct of the target, ts accelerato, etc. If the predcto wll ot be satsfactor we ca use such a predctor for damcs as Kalma flter. Such a good tegrato of the geometrcal approach wth the advaced probablstc methods wth 40

41 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu the optmzato of such a tegrato gves the possblt to solve the problem of processg vdeo real-tme wth eough relablt of tracks costructo. To the ed let us sum up what s doe ad what s left to solve vdeo trackg problem: Doe: Trackg at 5 fps o full HD vdeos; Fuctoalt of the track buldg has bee developed; Usg formato about domats veloct for mprovg the pot assgmet to approprate track; Optmzato for preprocessg, track buldg ad tests. Left: Trackg hghl teracted objects; Trackg objects wth hgh level of veloct; Rescalg tracks based o camera calbrato; uldg of 3D model of tracks..5.5 Refereces [] Gog, Sh. ad Xag T., Vsual alss of ehavour: From Pels to Sematcs, Sprger, Lodo, 0 [] Maggo, E. ad Cavallaro., Vdeo Trackg: Theor ad Practce, Wle, 0 [3] Marslad, S., Mache Learg: lgorthmc Perspectve, ChapmaHall/ CRC, oca Rato, Florda, 009. [4] Prce, S., Computer Vso : Models, Learg ad Iferece, Cambrdge Uverst Press, 0 [5] radsk G. ad Kaehler., Learg OpeCV, O Rell, Sebastopol, C, 008 [6] Chetverkov, D. ad Szabo, s., Smple ad Effcet lgorthm for Detecto of Hgh Curvature Pots Plaar Curves, Robust Vso for Idustral pplcatos 999, Vol. 8,999, p

42 .6 Path Clusterg Lbrar Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu The am of the task s to provde a comprehesve lbrar for clusterg paths ad path segmets. Path ca be defed as a track of some partcular scee pot o cosecutve vdeo frames. The am of path clusterg procedure s to fd objects o the vdeo, thus the algorthm eeds to take to accout tme ol paths estg same frames ca be clustered) ad space paths belogg to same cluster must fulfl some localt codtos). Segmet s a sequece of cosecutve path pots wth partcular startg pot ad legth. Procedure of segmet clusterg ams at fdg set of some uversal moto patters. Therefore, t operates dfferetl tha path clusterg procedure. It does ot take to accout tme ad space e.g. sequece of movemets characterstc for huma walk s smlar for dfferet people dfferet vdeo phases). The ol thg that matters s a segmet shape..6. Theoretcal backgroud Paths clusterg procedure s doe accordg to the Clusterg Paths secto. Segmets clusterg procedure s doe accordg to the Clusterg Path Segmets secto. Clusterg space trasformato The ma dsadvatage of clusterg paths ad segmets drectl the mage space s fact that perspectve trasformato does ot preserve dstaces,.e. objects that further to the camera appear smaller. Ths meas that all parameters of clusterg algorthms epressed pels could ot be appled globall to the whole mage. To overcome ths, co called clusterg space trasformato has bee troduced. It cossts trasformg objects from mage plae to the world referece frame ad the, projectg t back to the mage plae wth orthographc projecto stead of perspectve oe. Trasformato from two dmesoal mage frame to the three dmesoal world space s ambguous, thus t was assumed that all objects see b camera le o the groud ther z coordate world referece frame s equal to 0). s orthographc projecto s dstace preservg, obtaed space s sutable for performg clusterg pel sze of detcal objects, e.g. huma slhouettes do ot deped o mage rego). Of course, order to calculate clusterg space trasformato the camera eed to be calbrated meag that both trsc ad etrsc parameters must be kow. If the codto s ot fulflled, trasformato caot be performed ad mage space s used for clusterg paths ad segmets. 4

43 .6. Lbrar usage.6.. Overvew Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Path clusterg lbrar has bee mplemeted C++ laguage wth some C++0 etesos) as a statc lbrar amed PathClusterg.lb. It requres some addtoal lbrares to work properl. These are: OpeCV verso.3, oost verso.47. The fuctoaltes offered b the lbrar are: path clusterg, segmets clusterg, The followg paragraphs descrbe brefl how to use abovemetoed features. Detaled documetato of all classes ad terfaces eported b the lbrar ca be foud a fle PathClustergDocumetato.pdf. To use the lbrar oe ol eed to clude Motoalss.h header fle. ll fuctoaltes provded b the lbrar are ecapsulated a amespace called moto_aalss..6.. ular data structures I order to facltate paths clusterg ad segmets clusterg procedures some aular tpes ad classes have bee troduced. The are descrbed the table below. Name Descrpto Pot Class represetg sgle pot of a path or other pot collecto. It stores spatal ad temporal coordates of pot. PathPostoTpe Tpe of pot spatal coordates float b default). PathTmeTpe Tpe of pot temporal coordates t b default). PotCollectoase bstract base class for all ettes storg collecto of pots. It declares some methods ad operators that must be mplemeted b derved classes. PotCollecto asc collecto of pots. It s derved from PotCollectoase class ad a base class for Path, Cluster, SegmetCluster classes Paths clusterg ular classes utlzed b paths clusterg procedure are descrbed the table below. Path Cluster Dstace Class represetg path. Class represetg cluster of paths. Class storg dstace betwee path ad cluster or dstace betwee 43

44 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu two clusters. The base class for performg path clusterg s called Clusterg. It allows oe to perform clusterg the real-tme,.e. at ever frame clusterg state ca be updated accordg to curretl estg paths. I order to create ew clusterg object oe eed to voke oparametrc costructor. Clusterg clusterg; mportat elemet of clusterg object state s curret tme. It s b default set to 0 ad ca be mapulated b two methods: cost PathTmeTpe gettme) cost; vod settme PathTmeTpe v ) The curret tme s used as a temporal coordate for pots ad should be usuall set to the umber of curret frame. There are two methods updatepath) ad removepath) desged to pass formato about curret path state to the clusterg object. Method updatepathd, categor,, ) appeds to the path of gve d a pot wth spatal coordates,) ad temporal coordate equal to curret tme value. Id must be uque wth a set of curretl estg paths f some path has fshed ts d ca be reused later). Categor s a aular varable allowg oe to dstgush few tpes of paths that should be clustered separatel ol paths wth same categor ca be clustered). If path wth gve d does ot ests clusterg object, t s created. Method removepathd) removes a path wth gve d from collecto. Clusterg state s updated b vokg update) method. It should be eecuted at the ed of curret frame after all path formato has bee updated. To obta lst of curretl estg path clusters oe eed to use getclusters) accessor. There s also possblt to get partcular path b ts d usg getpaths) method. Method reset) allows oe to reset clusterg to ts tal state Segmets clusterg ular classes utlzed b segmet clusterg procedure are descrbed the table below. Segmet Path segmet sequece of cosecutve path pots wth partcular startg pot ad legth) SegmetCluster Cluster of path segmets. SegmetDstace Dstace betwee segmet ad cluster of segmets or two clusters of segmets. The base class performg clusterg of path segmets s called SegmetClusterg. alogousl to Clusterg class, t allows oe to perform procedure real-tme updates at ever frame). New object of SegmetClusterg class s created b vokg costructor: SegmetClustergPathTmeTpe legth) 44

45 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Costructor parameter dcates legth of cluster segmets to be aalsed. Thus, f user s terested clusterg segmets of dfferet legths he eed to create several SegmetClusterg objects each for dfferet segmet legth. Segmet clusterg procedure ca be cotrolled wth a use of same methods as path clusterg,.e. gettme),settme),updatepath), removepath) ad reset). Segmets of desred legths are automatcall created o the bass of gve path formato. SegmetClusterg class offers set of methods facltatg access to ts members. The are descrbed a table below. getpaths) Gets collecto of curretl estg paths that ca be accessed b ther detfers. getsegmets) Gets lst of all segmets created sce last reset. getclusters) Get collecto of all segmet clusters detected sce last reset. getpathtosegmetmappgs) Gets collecto that maps gve path to lst of ts cosecutve segmets. getssgmets) Gets collecto mappg segmets to clusters of segmets..6.3 Clusterg cosole PathalssCosole.ee s a cosole applcato that allows user to perform ad vsualse path ad segmet clusterg procedures. It should be eecuted wth followg sta: USGE: PathalssCosole.ee mode cofg_fle Optos: mode - clusterg mode -p for path clusterg ol, -c for complete clusterg), cofg_fle - fle wth program cofgurato. s oe ca see, the applcato ca operate two modes. I the frst oe -p), ol path clusterg procedure s carred out. I the secod mode -c) both path ad segmet procedures are eecuted ad vsualsed Path clusterg mode Cotet of the cofgurato fle used b path clusterg mode s gve below. [Geeral] cameracout=0 *mapfle= [Camera0] pathsfle= vdeofle= beg= ed= *trscfle= *posefle= [Camera] 45

46 <camera cofgurato> Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu [Camera9] <camera cofgurato> Star smbol dcates optoal parameters. s oe ca see, path clusterg cosole allows user to aalse smultaeousl paths from several dfferet cameras the are clustered depedetl). Number of cameras to be processed ad optoall path to the map fle are gve the geeral secto. If map fle s specfed, paths ad clusters from cameras are addtoall vsualsed o the map gve that camera f calbrated). Each camera s descrbed b some parameters. These are oblgator oes: vdeo fle alteratvel t ca be a drector wth mage sequece), fle wth paths defto, dces of startg ad edg frames. Optoall, user ca specf fles wth trsc ad etrsc camera parameters. These parameters are used to trasform mage to so called clusterg space see correspodg chapter Secto ) ad trasformato to map space. If o trsc ad etrsc parameters are specfed, clusterg space s same as mage space. fter rug path clusterg cosole, a set of wdows s show. For each camera specfed cofgurato fle oe ca see followg wdows: Cofgurato wdow allows user to adjust parameters of clusterg procedure see Fgure ), Stream vew vsualses path clusterg drectl o a camera vew see Fgure ), Projecto vew vsualses path clusterg a clusterg space see Fgure 3). ddtoall, f camera s calbrated trsc ad etrsc fles are gve) ad map fle s specfed cofgurato fle, Map vew also appears. It vsualses clusterg procedures from all cameras a commo map space see Fgure 4). Fgure. Cofgurato wdow allows all 46

47 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu clusterg parameters to be adjusted. Fgure. Stream vew vsualses clusterg procedure drectl the mage space. Fgure 3. Projecto vew vsualses clusterg procedure the clusterg space. If camera s ot calbrated clusterg s doe the mage space, thus Projecto vew s the same as Stream vew. 47

48 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fgure 4. Map vew vsualses clusterg procedures from all cameras the commo map space. fter pressg smbol at the top of whatsoever wdow, a addtoal bar wth vsualsato settgs appears: elow oe ca fd the meag of all vsualsato settgs: Show paths turs o/off vsualsato of paths. Each path s represeted as a sequece of red pots. If path fshes t has ot bee updated for assumed umber of frames) t s removed from vsualsato as well. Show path clusters turs o/off vsualsato of paths clusters. Each cluster s represeted as a set of whte squares, whch sze s scaled accordg to umber of paths assged to the cluster. 48

49 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Show edg pots turs o/off vsualsato of clusters edg pots last pots of all paths assged to the cluster). Pots are draw as dots wth a cluster-characterstc colour. Show cluster cotours shows rectagles eclosg all cluster edg pots wth a cluster-characterstc colour. Show trval clusters turs o/off vsualsato of trval clusters those havg ol sgle path assged). Step b step turs o/off step b step mode whch user eeds to cofrm each frame before aalsg et oe..6.4 Testg procedure I order to evaluate mplemeted methods several tests have bee performed both o sthetc as well as o real datasets. The ma am of the epermets was to check how algorthm parameters flueces clusterg results Sthetc epermets Sthetc vdeo sequece cossted of several crcles movg o a two dmesoal plae. It was assumed that each crcle correspods to a sgle path attached ts mddle see Fgure 4.). Thus, fle wth path formato was also geerated sthetcall o path detecto was carred out). 49

50 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fgure 4.. Sthetc dataset cosstg of movg crcles. Each crcle has a path attached to ts cetre dcated b red le. Fgure 4.. Clusterg wth all threshold parameters set to 0. s oe ca see, each path falls to separate cluster each dcated b colour rectagle). There were four ma algorthm parameters to be tested the procedure: Postoal dstace threshold path assgmet, Tagetal dstace threshold path assgmet, Postoal dstace threshold cluster jog, Tagetal dstace threshold cluster jog. t the begg of the epermetal phase all these parameters were set to 0, whch are most restrctve values ol detcal paths could be clustered together ths scearo). ccordg to the epectatos, for each path separate cluster was created ad o merges were observed throughout the whole vdeo sequece see Fgure 4.). I the secod test I creased value of postoal dstace threshold whch allowed parallel paths to be clustered together. The hgher value of threshold parameter, the more dstat paths fell to same cluster. However, as tagetal threshold was 0, o-parallel paths were alwas assged to separate clusters Fgure 4.3). I the et step, postoal dstace threshold for path assgmet was set to ft, so ol tagetal compoet was sgfcat. The I started to crease tagetal dstace threshold observg that agle betwee paths allowg them to fall to sgle cluster also creases, thus procedure operates properl Fgure 4.4). The last sthetc epermets proved that creasg values of dstace thresholds cluster jog procedure allowed clusters to be merged further steps of procedure please compare Fgures 4.5 ad 4.6). To coclude, sthetc epermets showed that algorthm works accordg to epectatos. 50

51 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Fgure 4.3. Icreasg postoal dstace threshold path assgmet procedure allows parallel paths to be clustered together. Fgure 4.4. No-zero value of tagetal dstace threshold path assgmet procedure allows o-parallel paths to be clustered. Fgure 4.5. If thresholds cluster jog procedure are set to 0, o cluster jog procedure s performed. Fgure 4.6. No-zero values of thresholds cluster jog procedure allows oe to merge estg clusters further vdeo frames. 5

52 .6.4. Real-lfe epermets Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu I the secod epermetal part I checked how clusterg algorthm performs o real lfe data. For ths purpose I used a short vdeo sequece captured o tom Market square. Paths were acqured wth a method based o domat pots Paths clusterg Clusterg parameters were tued to best detect huma slhouettes at tested sequece. O the fgures below oe ca see fal results of clusterg epermets. s oe ca see, algorthm works accordg to epectatos. It properl detects people walkg o the market square each perso s represeted b a separate cluster). 5

53 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 53

54 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 54

55 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 55

56 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu Segmets clusterg Ths epermetal part amed at checkg whether segmets clusterg procedure properl detfes some uversal moto patters. Hgh repeatablt of smbols moto sequeces dcates that algorthm works properl. 56

57 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 57

58 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 58

59 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 59

60 Opracowae: Polsko- Japońska Wższa Szkoła Techk Komputerowch Wdzał amejscow Iformatk w tomu 3. The results of the task The result of the task has bee developmet of a ovel approach, methods ad programs for costructo of a skeleto-free represetato of artculated moto based o a collecto of smbol strgs. The results also clude methods ad software for buldg local feature paths usg SIFT ad SURF algorthms ad detectors of domat pots of a cotour, clusterg feature paths ad clusterg path segmets. The feature paths ad path clusters costructed tests o outdoor mage sequeces satsf optmstc epectatos, judged b ther stablt, cosstec ad repeatablt. Further plaed results wll be joural or coferece publcatos, that wll follow tests of detfcato of behavor ad teracto based o the developed represetato of moto. 4. Coclusos Methods for costructg skeleto-free represetato of artculated moto have bee developed. I tests o outdoor mage sequeces these methods, ad the resultg represetatos ehbt the desred characterstcs of stablt, cosstec ad repeatablt. Quattatve evaluato of the recogto of behavor, based o the developed represetato, has ot bee coducted ad t s plaed for curret quarter. ssue of the curret method that requres mprovemet s the large umber of parameters that eed to be had tued, whch s tme cosumg ad mperfect. Therefore oe of the drectos of future work wll be automatc settg of the parameters, usg mache learg methods. 60