Motion Estimation. 5LIN0 Video processing. Video course: Motion Estimation. G. de Haan. Schedule lectures 5P530. Picture delay

Transcription

1 Video course: Motio Estimatio 1 2 Schedule lectures 5P530 5LIN0 Video processig G. de Haa Week 1 Week 2 Week 3 Week 4 Basics h 2 3 Video isplays h 9 Filterig h 4 PR & eiterlacig h 78 Week 5 Week 6 Week 7 Week 8 Image Ehacemet h 5 Motio Estimatio h 10 Object etectio h Motio Estimatio Motio Estimatio Is there ay motio? How fast? Ito which directio? y 5 Applicatio depedecy of ME 6 Motio estimatio ad codig Sca rate coversio true-motio vectors e-iterlacig Picture rate coversio Video compressio low predictio error MPEG H.2.63 True-motio vectors are usually more cosistet tha codig vectors. osistecy has some but o domiat relevace for codig efficiecy ME Iput + Predictio - error Motio compesatio Image compressio: accuracy demads decrease with icreasig frequecy T-trasform + quatizatio Picture delay Output

2 Video course: Motio Estimatio Basic assumptios ad cosequece Gradiet ME methods optical flow ostat brightess assumptio Local liear lumiace assumptio Image -1 F -1 F Image F -1 F 9 Basic assumptios ad cosequece 10 Basic assumptios ad cosequece ostat brightess assumptio Local liear lumiace assumptio ostat brightess assumptio Local liear lumiace assumptio F -1 F F -1 F F -1 F F -1 F Iterative optical flow to deal with o-liear brightess 12 Iterative optical flow F 2 df 2 d i I+1 I+2 I+3 Algorithm: etermie gradiet of displaced frame differece F ad update vector i directio of decreasig F i i1 u F u d 1 i d F i 1 F F i 1 1 u F 1 d i F i 1 d isplacemet

3 Video course: Motio Estimatio 3 13 Pel-recursive ME; The use of predictios 14 Not popular for video format coversio Spatial causal predictio Temporal predictio urret piel Iitially due to compleity Real-time applicatios: Iitially codig later also format coversio For codig oe vector per piel is ot attractive There are much simpler block-based methods For format coversio true-motio requiremet problem Artifacts whe assumptios are ivalid Time The temporal cadidate may also be motio compesated 15 Block-matchig ME methods: Full-search 16 Block-matchig; fid correspodig block i image -1 orrespodig block Search area urret block -1 Image umber 17 Fidig block similarity y urret block 18 Formal defiitios Lumiace value i previous picture shifted over cadidate vector : F 1 Search area A block matcher optimizes a fuctio ost varyig : ost F F 1 B Ad the resultig cadidate vector for which the error is miimal is assumed to be the displacemet vector:

4 4 Video course: Motio Estimatio B B B F F F F Normalised cross-correlatio favourable performace rather high operatios cout B F F Summed Square Error good performace acceptable operatios cout 21 1 B F F Summed Absolute ifferece still good performace favourable operatios cout : 1 threshold a threshold a a T with F F T B Sigificatly differetly piels Rather poor performace Favourable operatios cout reduced register size compared to SA 23 orrelatio NF of piels i the two blocks Mea Square Error MSE betwee piels i the blocks Mea Absolute ifferece MA betwee piels i the blocks Number of sigificatly differet piels NS i the two blocks ompleity Alterative match criteria 24 ompariso of match criteria MSE SA NSP

5 Video course: Motio Estimatio 5 25 Operatios cout of full search block matchig IR sigal piels/s Search widow for realistic velocities 6448 HV i piels = 3000 possible vectors assumig iteger vector accuracy Matchig error SA calculatio oly: approimately: ops/s NB: Full H requires eve more tha 4 times as may computatios! 5LIN0 Video processig G. de Haa 27 Block-matchig efficiet search techiques 28 Fidig block similarity y Search area urret block 29 Sub-sampled search urret block 30 Sub-sampled full search y y 2 1 Search area Search area

6 Video course: Motio Estimatio step search Koga et al Oe-at-a-time search Sriivasa & Rao 1985 y y Search area 33 Successive approimatio may become ecessary 34 Prevetio of trap i local miimum 3 mi y y 0 b y mi b mi a 0 a i 1j otour plot of error plae mi c 0 c otour plot of error plae mi d 0 d 35 Itermediate coclusio 36 Reality is eve more complicated Efficiet search techiques ca highly reduce the operatios cout of a block matchig motio estimator but icrease the risk of gettig trapped i a local miimum of the error fuctio Methods to prevet the disadvatages of efficiet search icrease compleity agai.

7 Video course: Motio Estimatio 7 37 Ad sometimes there is o uique solutio 38 ompariso of search techiques FS LogS OTS 39 Piel subsamplig i match fuctio 40 Piel sub-samplig of match error criterio y Search area urret block 41 Piel sub-samplig i match error criterio Block subsamplig

8 Video course: Motio Estimatio 8 43 Block sub-samplig 44 Iterpolate missig motio vectors V-positio Search area adidate vector Up urret block Le curret Ri -1 Lo Picture umber H-positio urret = media{le Up +Lo /2 Ri } 1: urret y = media{le y Up y +Lo y /2 Ri y } 2: Use the vector-media to prevet ew vectors 45 Vector media: geeralizatio of scalar media 46 Summary cost reductio block matchers Scalar media Vector media Simple match criterio Efficiet search strategy Vector that has smallest Euclidea distace to al other vectors Piel sub-samplig i match criterio a factor of four is usually feasible with little ifluece o the performace Block sub-samplig oly valid if motio field is smooth Full search block matchig motio vectors Vectors ad object velocity

9 Video course: Motio Estimatio 9 49 True motio versus best match Number 7 Arm Scarf Seve: Arm: 1 clear o Scarf: mi clear multiple mi mi Poor relatio vectors & velocities 2 1 SA : 3 B F F 1 is motio vector F image grey value B 88 block piel positio picture r 50 Block-matchig true-motio estimatio 51 What is wrog with block matchig? 52 Blocks are ot uique Optimizatio is ill-posed problem Testig for best match gives too may solutios Solutios: Itroduce bias e.g. towards cosistet vectors test better Post-processig e.g. elimiatig outliers test agai Pre-selectio of likely cadidates test less Itroduce bias Test better 53 Itroduce bias Test better 54 Miimal match error gives o uique solutio B A improved criterio takes ito accout that vectors are cosistet withi objects ad over time: B F F 1 F F 1 P P Ps ad Ps are pealties depedig o spatial ad temporal cosistecy of the cadidate vector PROBLEM: osistetly oly kow after completio Typically solved usig a iterative approach s t Post-processig Test agai..

10 Video course: Motio Estimatio Post processig to improve vector cosistecy Reuter 1988 V-Pos 56 The effect of post-filterig 53 blocks y-2y y-y y y+y y+2y y+3y F k k Neighbourhood o p H-Pos Origial Average Media Hierarchical block matchig Thoma & Bierlig 1989 Pre-selectio Test less ow-sampled picture at itermediate level oarse estimatio Iitialise Iitialise ow-sampled picture at highest level Medium size update vectors Small size update vectors Origial picture 59 Hierarchical block matchig 60 Pre-selectio i Fourier domai- Phase Plae orrelatio Hierarchical Full search PP is a two-step hierarchical motio estimator 1 Select up to 10 largest correlatio peaks i the Fourier domai usig blocks larger tha Test SA oly for these vectors o small block here 88 i the spatial domai Algorithm origially proposed by Graham Thomas ad applied i professioal studio format coverters

11 Video course: Motio Estimatio Time recursive block matchig Niomya y Test SA oly for these vectors cetred aroud result vector previous picture 62 ST-recursive cadidate selectio et time 5LIN0 Video processig G. de Haa Recursive Search blockmatchig 65 3-imesioal Recursive Search 3RS RS: How to start? Sigle radom update sufficiet! Assumptios: 1. Objects are LARGER tha blocks 2. Objects have INERTIA y Noise vector update Spatial predictio cadidates Temporal predictio cadidate adidate set Spatial cadidates Temporal cadidates Updated cadidates??

12 Video course: Motio Estimatio hose cadidates 68 Spatial Performace Temporal Update 69 Operatios out 70 Smoothess of vector field FS: H3: Pel-Rec: PP 4-St 3-St OTS H2 3- RS ompute differece with all eighbourig vectors Average over all blocks i vector field This gives vector icosistecy Smoothess is the iverse of vector icosistecy 71 Vector field smoothess 72 Performace testig of true-motio estimator: M2SE ME M F mc 2 MMSE F Fmc 1 F 1 F St 3-St FS OTS H2 PP 3- RS picture r.

13 Video course: Motio Estimatio M2SE score of ME-methods 74 ompariso of best vector fields Phase Plae orrelatio motio vectors 3- Recursive Search BM motio vectors St OTS 3-St H2 FS H3 PP 3-RS 75 M up-coversio; Relevace of true-motio vectors 76 Iterpolated images usig full search motio vectors Iterpolated image usig 3-RS motio vectors Simplificatios 1 Reduced cadidate set I cotrast with codig for sca rate coversio true-motio is a absolute must. RATHER SMOOTH THAN AURATE!! 77 With 8 predictio ad 1 update: 9 cadidates 78 3RS 4 cadidates are eough icludig 1 update V-pos urret block Block i curret field Block i previous field V-pos urret block Block i curret field Block i previous field y-y S a S b S c y-y S b y S d Ta y S a y+y T b T c Td y+y y+2y y+2y T H-pos H-pos

14 Video course: Motio Estimatio Y-estimator advatage for pipe-liig V-pos urret block Block i curret field Block i previous field 80 Effect of cadidate reductio M2SE: 21.5 S: 2.8 M2SE: 26.0 S: 1.7 M2SE: 23.3 S: 2.6 y-y y S a S b y+y y+2y T H-pos 81 Block diagram of Y-estimator; Simple hardware 82 Mod p cout Look Up Table Update Geerator Predictio memory N bl 0 U Update urret Best vector selectio Previous Simplificatios 1 Reduced resolutio for ME picture picture 83 ME with reduced resolutio compared to applicatio 84 iput Applicatio like e-iterlacig PR etc. output Block-hoppig ow-scale video sigal Motio estimatio o reduced video Up-scale motio vectors

15 Video course: Motio Estimatio hose cadidates 86 Block-hoppig Spatial Temporal Update I may cases the spatial predictio SP is good. Save calculatios o the average by checkig the other cadidates oly if SP error is above Th alculate all SAs grey blocks are skipped 87 Block hoppig; optimal resource usage Vector memory alc. all alc. SA MU SAs of SP compare s Th Adapt threshold alculate Resource Usage/field Assig best Assig SP MU s 5LIN0 Video processig G. de Haa Iteratig more tha oce o a image pair Sophisticatios Effect of iteratios M2SE 100 smoothess Oce 1 st image Remark 1: If estimatig i the output domai 100Hz: 2 iteratios o video ad 4 iteratios o film material! 10 times Remark 2: Effect maily shows i 1 st image after scee chage: 1 iteratio 10 th frame: M2SE: 29 Smoothess: iteratios 10 th frame: M2SE: 28 Smoothess: 3.5

16 Video course: Motio Estimatio Block diagram of Y-estimator; Simple hardware Predictio memory Block-erosio N bl Mod p cout Look Up Table U 0 Update Best vector selectio Block erosio Update Geerator urret picture Previous picture 93 Block erosio No BE 1 step BE 94 The effect of block erosio U U L R Media L V 1 V 2 R V 3 V 4 2 step BE 3 step BE U U L R Media L V 1 V 2 R V 3 V 4 U U U U L R Media L V 1 V 2 R V 3 V 4 L R Media L V 1 V 2 R V 3 V imesioal Recursive Search 3RS Normal sca Meaderig sca Reverse sca Advaced scaig

17 Sample vectors Video course: Motio Estimatio Parametric motio models 98 Global motio estimatio Simple parametric motio model: p1 p3 p5 y... y p2 p4 y p6... p 1 ad p 2 describe pa ad tilt p 3 ad p 4 describe zoom p 5 ad p 6 describe rotatio 99 Sample vector field to calculate model parameters 100 erive robust backgroud model from sample vectors Take media of all estimated parameters to elimiate outliers: p 1 = media{p 1 1 p 2 1 p 3 1 p 18 1 } p 2 = media{p 1 2 p 2 2 p 3 2 p 18 2 } p 3 = media{p 1 3 p 2 3 p 3 3 p 18 3 } p 4 = media{p 1 4 p 2 4 p 3 4 p 18 4 } Motio model with 4 parameters ca be calculated from ay 2 idepedet sample vectors So i total from these 9 vectors 18 models ca be estimated 101 Etra cadidate from parametric motio model SAA Effect of etra cadidates from parametric model N bl > Mod p couter Look up table Update vector geerator Predictio memory U 0 update calculate local cadidates micro processor calculates parameters P 1 P 2.. urret picture Best vector selectio Block erosio Previous picture Without parametric cadidate With parametric cadidate learly the effect depeds o the settigs of the cadidate s pealty!

18 Positio Video course: Motio Estimatio The basic block matchig cocept Motio estimatio ad occlusio -1 V-positio Search area adidate vector Referece block 8 8 piels H-positio Picture umber 105 How to estimate motio estimatio i occlusio areas? 106 Ambiguities due to ucoverig Iformatio ot available i previous picture -1? Preferece for FG-vector i ucovered areas -1 Time 107 How to estimate motio estimatio i occlusio areas? Iformatio ot available i et picture Iformatio ot available i previous picture 108 Motio estimatio problem i occlusio areas Observatios: Foregroud: Matches always i.e. i previous ad i et picture Backgroud: I case of coverig all backgroud will match i previous picture I case of ucoverig all backgroud will match i et picture -1 oclusio: Switch betwee forward ad backward motio estimatio to prevet ambiguities

19 Positio Video course: Motio Estimatio Solutio: I coverig areas forward estimatio 110 Solutio: I ucoverig areas backward estimatio V-positio Search area adidate vector V-positio Referece block 8 8 piels -1 Referece block 8 8 piels -1 Search area adidate vector H-positio H-positio Picture umber Picture umber 111 Uambiguous motio vectors for origial images 112 ompariso 2 frame ad 3 frame motio estimatio Look for correspodeces i BOTH eighbourig images select predictio with the highest correlatio 2 frame ME 3 frame ME forward backward Time Projectio based global motio estimatio Global motio estimatio Algorithm: Accumulate lumiace over all lies Accumulate lumiace over all collums etermie global H- ad V- motio based o these projectios emo Samsug ME

20 Video course: Motio Estimatio Projectio based global motio estimatio 116 Success ad failure of the projectio based global ME Global motio: Miimum SA of projectio curret ad previous image Fik Fik+1 i EMO Global ME 2v i 117 oclusios 118 oclusios Motio estimators for sca rate coversio differ from estimators for codig due to additioal true-motio costrait True motio results from costraits like spatial ad temporal cosistecy 3 optios: better criterio post-processig pre-selectio Pre-selectio optios Hierarchical approach e.g. Phase Plae orrelatio. Recursive approach 3- RS Picture rate coversio requires very cosistet but ot ecessarily very accurate motio vectors iteger resolutio sufficiet the rage should be at least +/-16 piels e-iterlacig requires very accurate motio vectors at least 1/4 piel. For larger vectors the accuracy is less importat 119 Prepare yourself for the eam Last week: hapter 8 Today: hapter 10 SKIP: Object-based ME I recommed you read the tet Book available at Pt9:24 Ad try the eercises i the book: hapter hapter 10 skip 10.6 You have to dowload VidProc Sed me for password [email protected]