A HYBRID GRAMMAR-BASED APPROACH FOR LEARNING AND RECOGNIZING NATURAL ICONIC GESTURES. Amir Sadeghipour

Transcription

1 A HYBRID GRAMMAR-BASED APPROACH FOR LEARNING AND RECOGNIZING NATURAL ICONIC GESTURES Amir Sadeghipour

2 ICONIC GESTURES Different iconic gestures for the same sphere:.

3 3DIG DATASET 10 simple & 10 complex objects 3D Iconic Gesture Dataset: 29 participants (20m/9f) 3 gestures per object in a random order In total 1739 recorded gestures (~87 gestures/object) Recorded formats: color video, depth video, 20 joints skeleton MS Kinect

4 GESTURE ANALYSIS Representational Techniques 100% 80% Used technique per participant Drawing Shaping 60% 40% 20% Used technique per object Posturing Enactment 100% 1 80% % % 20% Apple Bottle Bulb Chair Cone Cube Cylinder Dumbbell Egg Ellipsoid House Parallel trapezoid Pyramid Rectangular box Sphere Table Teapot Trapezoid Umbrella Vase Simple objects Complex objects C

5 GESTURE ANALYSIS Spatiotemporal feature variabilities Movement direction Movement velocity Size of gesture space Location of gesture space Depicted orientation Depicted form Structural variabilities Handedness Degree of simplification Ordering Repetition

6 GESTURE ANALYSIS Spatiotemporal feature variabilities Drawing fast or slow Drawing in front of head or chest Making curved or straight movements Movement direction Movement velocity Size of gesture space Location of gesture space Depicted orientation Depicted form upward or downward while drawing Drawing a small or a big circle Drawing a horizontal or vertical projection Structural variabilities Handedness Degree of simplification Ordering Repetition

7 GESTURE ANALYSIS 1.2 Handedness per participant Spatiotemporal 100% feature variabilities 80% 60% 40% 20% Movement direction Movement velocity Size of gesture space Handedness per object Handedness per object Location of gesture space 100% 80% 60% 40% 20% 1 0 Depicted orientation Apple Bottle Bulb 1.2 Depicted form 1 Chair Cone Cube Cylinder Dumbbell Handedness per object set Both hands symmetric Both hands asymmetric Only right hand Only left hand Egg Ellipsoid House Parallel trapezoid Pyramid Rectangular box Sphere Table Teapot Trapezoid Umbrella Vase Handedness per object set Simple objects Both hands symmetric Complex objects Both hands asymmetric Only right hand Only left hand Structural variabilities Handedness Degree of simplification Chair Ellipsoid Dumbbell House Cylinder Egg Parallel trapezoid Cube Pyramid Rectangular Cone box Bulb Sphere Table Teapot Trapezoid Umbrella Vase Ordering Repetition

8 GESTURE ANALYSIS 3D gestures per participant Spatiotemporal feature variabilities 100% Movement direction 80% 60% 40% 20% Movement velocity Size of gesture space 3D gestures per object Location of space 3D per object 80% Depicted orientation 100% 60% 40% 20% 0 Apple Depicted form Bottle Bulb Chair Cone Cube Cylinder Dumbbell Egg Ellipsoid House Parallel trapezoid Pyramid Rectangular box Sphere Table Teapot Trapezoid Umbrella Vase Simple objects Complex objects C Structural variabilities Handedness Degree of simplification Ordering Repetition

9 GESTURE ANALYSIS Spatiotemporal feature variabilities Structural variabilities Movement direction Handedness Movement velocity Degree of simplification Size of gesture space Location of gesture space Depicted orientation Depicted form Drawing first the circular base of cone then the triangle, or vice versa. Ordering Repetition

10 GESTURE ANALYSIS Gestures with repetition per participant Spatiotemporal 100% feature variabilities 80% 60% 40% 20% Movement direction Movement velocity Size of gesture space Repetition per object Location 40% of gesture space 30% 20% 10% 0 Depicted orientation Apple Gestures with repetition per object Depicted form Bottle Bulb Chair Cone Cube Cylinder Dumbbell Egg Ellipsoid House Parallel trapezoid Pyramid Rectangular box Sphere Table Teapot Trapezoid Umbrella Vase Simple objects Complex objects C Structural variabilities Handedness Degree of simplification Ordering Repetition

11 COGNITIVE MODEL Three levels of motor knowledge Sensory system visual stimuli of others' gestures Motor system gesture performance 1. Motor Commands: single movement segments Perception strokes Prediction strokes surprise signal Realization Control strokes 2. Motor Programs: specific gesture performances 3. Motor Schemas: group of different gestures with the same iconic mapping motor knowledge Recognition Learning Shared Hierarchical Motor Knowledge Hierarchical Motor Knowledge Motor Schema (MS) Level MS1 MS2... Motor Program (MP) Level motor knowledge MP1 MP2 MP3... Motor Command (MC) Level MC1 MC2 MC3 MC4 MC5...

12 RELATED WORK Tsai & Fu (1980) proposed the idea to use attributed grammars to integrate statistical consideration into syntactic pattern analysis Tsai and Fu (1980). Attributed Grammar - A Tool for Combining Syntactic and Statistical Approaches to Pattern Recognition. IEEE Transactions on Systems Man and Cybernetics. Stolcke (1994) proposed a a Stochastic Context-Free Grammar (SCFG) and probabilistic framework for parsing and learning. Stolcke, Andreas (1994). Bayesian Learning of Probabilistic Language Models. PhD thesis. University of California, Berkeley SCFG represenation S NT1 NT2 NT3 [1] NT1 NT11 NT12 [0.5] NT1 t1 [0.5] NT11 t21 [1] NT12 t22 [1] NT2 t3 [1] NT3 t4 [1]

13 RELATED WORK Ivanov & Bobick (2000) proposed using a pre-defined SCFG for recognizing simple hand gestures (through Earley-Stolcke parsing) Ivanov and Bobick (2000). Recognition of visual activities and interactions by stochastic parsing. IEEE TPAMI. Kitani, Sato & Sugimoto (2006) applied SCFG for learning models of very simple activities. Kitani, Sato and Sugimoto (2006). An MDL Approach to Learning Activity Grammars. IEICE.

14 RELATED WORK Zhang, Tan & Huang (2011) extended SCFG by temporal logic and handling uncertain inputs for both learning and recognizing different visual events. Zhang, Tan and Huang (2011). An extended grammar system for learning and recognizing complex visual events. IEEE TPAMI. Pastra & Aloimonos (2012) proposed a biologically inspired approach to learn hierarchical syntax of human actions. Pastra and Aloimonos (2012). The minimalist grammar of action. Phil. Trans. R. Soc. B.

15 RELATED WORK HMM is the mostly used method for gesture recognition in HCI. Rautaray and Agrawal (2011). Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review. Natural gestures, different dann pre-defined HCI gestures, are more variable and complex. Thus, for recognition either extensions of HMM are applied Kruger et al. (2010). Learning Actions from observations Primitive-Based Modeling and Grammar. IEEE Robotics & Automation. or HMMs are applied for low-level symbolization as a basis for high level syntactic learning Ivanov and Bobick (2000). Recognition of visual activities and interactions by stochastic parsing. IEEE TPAMI.

16 FSCFG Feature-based Stochastic Context-Free Grammar (FSCFG) S Statistical Syntactic SCFG represenation S NT1 NT2 NT3 [1] NT1 NT11 NT12 [0.5] NT1 t1 [0.5] NT11 t11 [1] NT12 t12 [1] NT2 t2 [1] NT3 t3 [1] Feature sets: Features: w f1 F 1 w f2 w fn f 1 f 2... f n NT1 NT2 NT3 t1 NT11 NT12 t2 t3 t11 t12 w F1 w F2 w Fl w f1 F 2... w f2 w fn w f1 w f2 w fn f n f 1 f 2... f n f 1 f 2... F l

17 FSCFG Preprocessing: Symbolization of hand trajectories 3D wrist trajectories 2.2 Left wrist trajectory 2.1 Right wrist trajectory Z 1.8 Z 1.7 Y X X Z 1.8 Z 1.7 X X Y Y Extracted features per stroke (dim.) size (2) normal vector (3) concavity (3) average speed (1) start time (1) start-end vector (2) samples heights (5) In total 17 features 0.14 Guiding Strokes Segmenting on velocity dorps Smoothing spline data1 data1 data2 0.5Gesture #208 for bulb 0.14 data velocity data3 2 2 data3 data1 data4 Left wrist velocity data data2 data5 data5 Right wrist velocity 0.1 data6 data3 data6 Smoothed left wrist v. data7 0.1 data4 data7 data8 Smoothed right wrist v. data5 0.3 data data9 Left wrist's local min. data data9 data10 Right wrist's local min. 3 data7 data10 data data8 data data9 4 data data10 data data data2 0 data data1 data4 Left wrist traj data2 Right data5 wrist traj. 0 data3 Smoothed data6 0.2 l. traj. data4 0 data7 0.2 Smoothed r. traj. 1 1 data data8 0.1 frame data data data data data data data10

18 FSCFG Movie: FSCFG.mov

19 FSCFG FSCFG Representation Terminals Hand rules Start rules or FSCFGs FSCFG Processes Structure and parameter learning Parsing strings Generating strings vs. vs. Cognitive Representation Motor commands Motor programs Motor schemas Cognitive Processes Learning gestural motor knowledge Recognizing familiar gestures Performing gestures

20 FSCFG Parsing with FSCFGs Based on Earley-Stolcke parser for SCFG Statistical Syntactic SCFG represenation S NT1 NT2 NT3 [1] NT1 NT11 NT12 [0.5] NT1 t1 [0.5] NT11 t21 [1] NT12 t22 [1] NT2 t2 [1] NT3 t3 [1] Feature sets: Features: w f NT1 NT2 NT3 t1 NT11 NT12 t2 t3 w F F 1 f 1 f 2... t11 f n w f S t12 F 2... w f F l Extending the scanning step of parsing an input symbol by a terminal t : x i 2{x 1,...,x m } Feature-set parsing Feature parsing lx p(x i t)= w Fj g(x i F j ) g(x i F j )= j=1 ny k=1 w fk gauss(f k,xi µ fk, f k )

21 FSCFG Learning FSCFGs Structure learning Structure modification operators to minimize DL U XYXY chunk(x,y)=z U ZZ X a Y b merge(x,y)=z merge(a,b)=c Z c Using Bayesian loss measure: loss = α parsing probability description length Parameter learning Learning rule probabilities: normalized counter Learning feature sets weights: normalized counter Learning feature weights (of terminal t): 1 w f (t)= std({8f 2 F i F i 2 t; i =1,...,l})+1

22 FSCFG Handling uncertain input Substitution error: handled implicitly through feature-based representation Insertion error Deletion error S X Y [1] X a [1] Y b [1] Adding error handling rules S X Y [.99] S X Y SKIP [.01] X a [.98] X ε [.01] X SKIP a [.01] Y b [.98] Y ε [.01] Y SKIP b [.01] SKIP skip [1.0]

23 RESULTS Subsets of 3DIG dataset 1800 Number of gestures Simple synchronous drawing 2D drawing Drawing At least one drawing Drawing or Shaping No posturing All gestures subsets

24 RESULTS 1 ROC Analysis 0.9 Two-fold cross-validation Train and test with an FSCFG per class (object) Each ROC = mean of 20 FSCFGs for 20 classes Varying ROC parameter: classification threshold of parsing probability True positive rate Simple synchronous drawing (AUC=0.996) 2D drawing (AUC=0.979) Drawing (AUC=0.953) At least one drawing (AUC=0.851) Drawing or shaping (AUC=0.811) No posturing (AUC=0.824) All gestures (AUC=0.816) CChance level (AUC=0.5) False positive rate

25 RESULTS Target labels: Teapot 20 Dumbbell 19 Umbrella 18 Table 17 Chair 16 Bottle 15 Bulb 14 Vase 13 Apple 12 House 11 Par.trapezoid 10 Trapezoid 9 Pyramid 8 Cone 7 Cube 6 Rectang.box 5 Cylinder 4 Ellipsoid 3 Egg 2 Sphere 1 Confusion matrix of drawing gestures Predicted labels

26 RESULTS Simple synchronous drawing 2D drawing At least one drawing Drawing or shaping No posturing All gestures

27 RESULTS Comparison Human judgment: based on watching color, skeleton or hand movement videos F-measure / Huamns' correct guess rate FSCFG FSCFG norm. FSCFG optimized HMM SVM Humans color videos Humans skeleton videos Humans wrist videos Chance level SVM: ν-svm type (ν=0.01), radial basis kernel type (γ=0.01) HMM: 5 hidden states and 8 Gaussian mixtures Drawing 5 or No6 7 Simple synchronous drawing 2D drawing Drawing At least one drawing Shaping posturing All gestures

28 Classification performance RESULTS FSCFG FSCFG 0.9 FSCFG normalized 0.8 HMM SVM 0.7 Chance level norm. FSCFG Human judgment performance HMM SVM Color Humans videos color videos Skeleton videos Humans skeleton vide Wrist videos Humans wrist videos Chance level Chance level 0 Simple sync. drawing 2D drawing Drawing At least one drawing Drawing or shaping No All gestures posturing 0 Simple sync. drawing 2D drawing Drawing At least one drawing Drawing or shaping No All gestures posturing SVM: ν-svm type (ν=0.01), radial basis kernel type (γ=0.01) HMM: 5 hidden states and 8 Gaussian mixtures Human judgment: based on watching color, skeleton or hand movement videos

29 RESULTS Clustering of drawing gestures for simple objects Y X S=>L26 R27 S=>L31 R37 S=>L54 R54 S=>L70 R76 S=>L57 R61 S=>L83 R88 S=>L97 R102 Motor schemas Motor programs Motor commands Gesture performances Trapezoid Cone Cube Cylinder Sphere Ellipsoid Egg Rectangular box Parallel trapezoid Pyramid

30 RESULTS Simple synchronous drawing for simple objects Drawing or shaping for simple objects At least one drawing for simple objects Drawing for complex objects

31 RESULTS Learned FSCFG from drawing gestures for simple objects w f NT53 => gs45 y 1 y 2 y 3 y 4 y 5 n x n y n z c x c y c z l x l y w F w f NT54 => gs46 y 1 y 2 y 3 y 4 y 5 n x n y n z c x c y c z l x l y w F w f NT57 => gs w F h w t h w t y 1 y 2 y 3 y 4 y 5 n x n y n z c x c y c z l x l y h w t f v F f v F f v S => L26 R27 (30)[0.22] L26 => MG46 NT5 NT6 NT7 NT8 (17)[0.77] L26 => NT17 NT18 NT19 NT20 (1)[0.04] L26 => NT18 (1)[0.04] L26 => NT5 NT6 NT7 NT8 (3)[0.14] R27 => NT10 NT11 NT12 NT13 NT14 NT15(1)[0.04] R27 => R27 SK106 (13)[0.59] R27 => NT22 NT23 NT24 NT25 (6)[0.27] R27 => NT25 (1)[0.04] R27 => NT12 (1)[0.04] S => L31 R37 (17)[0.12] L31 => NT27 NT28 NT29 NT30 (16)[0.94] L31 => NT28 (1)[0.06] R37 => NT32 NT33 NT34 NT35 NT36 (15)[0.75] R37 => R37 SK39 (1)[0.05] R37 => NT32 NT33 NT35 NT36 (3)[0.15] R37 => R37 SK45 (1)[0.05] S => L54 R54 (3)[0.022] L54 => MG46 NT41 (1)[0.33] L54 => NT46 NT47 NT48 (2)[0.67] R54 => NT43 NT44 (1)[0.33] R54 => NT50 NT51 NT52 (2)[0.67] S => L57 R61 (46)[0.34] L57 => NT53 NT54 NT55 NT56 NT57 (2)[0.04] L57 => NT53 NT54 NT56 NT57 (6)[0.12] L57 => NT53 NT54 NT57 (40)[0.83] R61 => NT58 NT59 NT60 (46)[1.0] S => L70 R76 (14)[0.10] L70 => NT64 NT65 NT66 NT67 NT68 NT69(5)[0.31] L70 => NT64 NT65 NT66 NT67 NT69 (7)[4] L70 => NT64 NT65 NT66 NT69 (4)[0.25] R76 => NT71 NT72 NT73 NT74 NT75 (10)[0.37] R76 => R76 SK77 (12)[4] R76 => NT71 NT73 NT74 NT75 (5)[0.18] S => L83 R88 (10)[0.07] L83 => NT79 NT80 NT81 NT82 (10)[0.91] L83 => L83 SK91 (1)[0.091] R88 => NT84 NT85 NT86 NT87 (8)[0.88] R88 => NT87 (1)[0.11] S => L97 R102 (17)[0.12] L97 => NT93 NT94 NT95 NT96 (14)[0.93] L97 => NT94 (1)[0.07] R102 => NT98 NT99 NT100 NT101 (15)[0.94] R102 => NT99 (1)[0.06]

32 CONCLUSION FSCFG FSCFG integrates symbolization (emerging motor primitives) and learning the syntax of these symbols (hierarchical motor knowledge) within a homogenous framework. FSCFG considers statistical regularities between spatiotemporal features of gesture performances as well as the structural variabilities among them.

33 CONCLUSION Classification The classification performance of FSCFG outperformed other classifiers such as HMM and SVM. Comparing with human judgment performance, FSCFG was better at recognizing the well-specified drawing gestures.

34 CONCLUSION Clustering Unsupervised learning of iconic gesture performances as an FSCFG results in a three level hierarchical representation of motor knowledge (syntactic) based on a two level description of motor primitives (statistical). Gestures with similar iconic mapping were clustered within the same motor schema (e.g. a motor schema for round objects).