Diverse Particle Selection for Inference in Continuous Graphical Models Jason Pacheco

Transcription

1 Diverse Particle Selection for Inference in Continuous Graphical Models Jason Pacheco Collaborators: Erik Sudderth, Brown University Silvia Zuffi, Michael Black, MPI Tubingen

2 Reasoning Under Uncertainty Data Unknowns Estimate Probability Model Discrete Unknowns Continuous Unknowns Efficient inference based on dynamic programming Unrealistic approximations of model yield analytic updates

3 Reasoning Under Uncertainty Data Unknowns Estimate Probability Model Discrete Unknowns Continuous Unknowns Efficient inference based on dynamic programming Unrealistic approximations of model yield analytic updates.

4 Continuous Problems Human pose estimation & tracking Protein structure & side chain prediction Nonlinear time series & target tracking.

5 Continuous Problems Human pose estimation & tracking Protein structure & side chain prediction Nonlinear time series & target tracking.

6 Maximum a Posteriori (MAP) Data Unknowns Posterior MAP Estimate * Maximizer of the posterior probability: Issues with continuous models: Analytically intractable posterior density

7 Maximum a Posteriori (MAP) Data Unknowns Posterior MAP Estimate * * Maximizer of the posterior probability: Issues with continuous models: Analytically intractable posterior density Many local optima. These can be useful too

8 Goal Develop maximum a posteriori (MAP) inference algorithms for continuous probability models that: Does not assume tractable models Is black-box (e.g. no gradient calculations) Robust to multiple local optima

9 Pairwise Markov Random Field Likelihood Compatibility : Vertices, edges Graphical encoding of probability density Product of potential functions Facilitates efficient inference algorithms

10 Nonlinear time series Graphical Models Human pose estimation / tracking Frame t Frame t+1 Protein structure / side chain prediction

11 [ Zuffi et al., CVPR 2012 ] Human Pose Estimation Complicated Likelihood Non-Gaussian Compatibility Deformable Structures (DS): Continuous state for part shape, location, orientation and scale. PCA Shape

12 Message Passing Global MAP inference decomposes into local computations via graph structure

13 Message Passing Max-marginal distribution quantifies uncertainty over maxima.

14 Max-Product (MP) Belief Belief Propagation Passing messages in a graphical model Message Max-Marginal

15 Max-Product Belief Propagation Discrete Continuous???? Message Update: Message Update: Matrix-vector multiplication & discrete maximization Nonlinear optimization

16 Regular Discretization Approximate continuous max-product messages over regular grid of points Head Upper Arm Lower Arm Torso Example: Torso ~10 dimensions. 10 grid points per dimension 10 Million points! Location Shape Infeasible for high dimensional models.

17 Particle Filter T-1 T CONDENSATION algorithm [Isard & Blake, 1998] Particles degenerate over time Resampling step reduces effective number of particles Particle representation is too compact (e.g. entire trajectory)

18 Particle Representations Particle filter: Each particle is a full joint instantiation Max-Product: Each particle is a single random variable Efficiently enumerates all combinations

19 Particle Max-Product (PMP) Combine particle filter ideas with maxproduct more effectively. Particle approximation of continuous max-product (MP) messages.

20 Particle Max-Product (PMP) Head Upper Arms Lower Arms Torso 1 Augment Particles Proposal Augmented Set Sample new hypotheses at every node to grow particle set.

21 Particle Max-Product (PMP) Colors 1 Augment Particles 2 Max-Product Update Update MP messages on augmented particles.

22 Particle Max-Product (PMP) Colors 1 Augment Particles 2 Max-Product Update 3 Select Particles Given. particles Grow to. particles;. Reduce to. good particles Select subset of good particles and repeat Need a particle selection method

23 Synthetic Pose Estimation Binary image of 4 silhouettes. Model Truncated Gaussian pairwise potentials : Synthetic Image Colors PCA Shape Distance Likelihood Likelihood distance-map from silhouette contours. Random Initialization

24 Greedy PMP (G-PMP) Example Runs Colors Initial Particles Keep best particle Sample random walk Naïve proposal distribution Too greedy; Sensitive to initialization [ Peng et al., ICML 2011 ]

25 Top-N PMP (T-PMP) Example Runs Colors Keep N-best particles Sensitive to initialization Still too greedy; Selection reduces effective number of particles Maintain diversity in particles. [ Pacheco et al., ICML 2014 ]

26 Diverse Particle Selection Initial Particles Integer Program (IP) solved with efficient greedy approximation: Integer Program Diverse Selection LP : Linear Program relaxation IP: Optimal solution by brute force Greedy: Efficient approximation

27 Continuous Message Joint Distribution Message Bivariate Gaussian mixture:

28 Discrete Message Joint Distribution Message All Particles Regular grid of 50 states gives discretization:

29 Particle Selection Joint Distribution Selection vector Message All Particles Selected Indicator vector controls state selection: indicates selected states (red line)

30 Particle Selection Joint Distribution Message All Particles Selected Selecting more states reduces distortion between discrete message vectors.

31 Diverse Particle Selection Minimize total message distortion: NP-hard Submodular All Particles Selected Good approximation qualities.

32 Submodularity Set function iff diminishing marginal gains. is submodular Margin Diverse particle selection IP equivalent to submodular maximization. Efficient greedy approximation Within times optimal

33 Greedy Particle Selection Margin Maximum

34 Greedy Particle Selection Margin Maximum

35 Greedy Particle Selection Margin Maximum

36 Greedy Particle Selection Margin Maximum

37 Diverse Particle Selection Diversity: IP objective encourages diversity No explicit distance constraint Distance difficult to specify in general All Particles Selected Multiple neighbors: Two-node case generalizes: Concatenate message vectors: Still submodular, etc

38 Diverse Particle Max-Product (D-PMP) Example Runs Colors No explicit diversity constraint Objective encourages diversity Efficient Lazy greedy algorithm Bounds on optimality Avoids particle degeneracies by maintaining ensemble of diverse solutions near local modes. [ Pacheco et al., ICML 2014 ]

39 Synthetic Puppets True MAP Random Initialization Pose Error of MAP Estimate Log Probability of MAP Estimate G-PMP T-PMP D-PMP D/T-PMP G-PMP T-PMP D-PMP D/T-PMP Box plots summarize results from 11 random initializations.

40 Detection Real Images (Single Person) Top 3 arm hypotheses MAP estimate, 2 nd and 3 rd modes for upper arm (magenta, cyan), lower arm (green, ). Buffy dataset [Ferrari et al. 2008]. Detections versus number of ranked hypotheses. Baseline: Flexible Mixture of Parts (FMP) [Yang & Ramanan 2013; Park & Ramanan 2011] # Solutions [ Pacheco, Zuffi, Black & Sudderth, ICML 2014 ]

41 Real Images (Multiple People) Colors D-PMP Particles Mode Estimates Precision-Recall for multi-person frames: T-PMP : High precision, low recall, particles on one figure D-PMP : Outperforms FMP and other particle methods Note: G-PMP not reported due to poor performance. [ Pacheco, Zuffi, Black & Sudderth, ICML 2014 ]

42 Human Pose Tracking Prior work fails to show improvement with inference on temporal model We believe this is due to a failure of inference Joint work with S. Zuffi and M. Black

43 Human Pose Tracking Flowing Puppets [Zuffi et al., ICCV 2013] Input Sequence Hand Flow Likelihood Skin Color and contour likelihood terms as in Deformable Structures Input Dense Flow Hand Flow Map Sapp et al., Parsing Human Motion w/ Stretchable Models, CVPR11 Frame t Temporal Prior Motion: t t+1 Appearance Constancy: - Frame t+1

44 Loopy Max-Product BP Many interesting models exhibit cyclic dependency structure... Loopy Max-Product BP: Iteratively update until converged. Minimal guarantees.

45 MAP Probability Bound Spanning Tree Distribution Bound MAP via Jensen s Inequality: Dual Problem: [Wainwright et al., 2005]

46 Reweighted Max-Product (RMP) Edge Appearance Solve dual problem via reweighted message passing [Wainwright et al., 2005]

47 RMP Bound Tightness Pseudo-Max-Marginal distribution: Consistent maximizer: RMP bound tight and global MAP:

48 Loopy Particle Max-Product 1 Augment Particles 2 RMP Update 3 Select Diverse Colors Select diverse subset and repeat

49 Diverse Particle Selection Minimize reweighted message distortion: Accounts for spanning tree distribution Remains submodular Same greedy approximation

50 Pseudo-Max-Marginal Error Pseudo-max-marginal error bound: Pseudo-max-marginal definitions: Intuition: Selection IP objective upper bounds pseudo-max-marginal distortion.

51 VideoPose2 Experiments [Sapp et al. 2011] D-PMP T-PMP

52 Pose Tracking Particles T-PMP D-PMP Greater diversity in particles allows D-PMP to reason more globally Colors

53 Pose Tracking Oracle Solution Oracle solution quantifies diversity (30-pixel detection radius)

54 Protein Structure Prediction All information for predicting 3D structure encoded in amino acid sequence and physics

55 Protein Side Chains Sidechains Backbone Side chain prediction: Estimate side chains given fixed backbone. Sidechain 20 Amino Acid Types Backbone Leu (L) Val (V) Phe (F) Trp (W)

56 Dihedrals and Rotamers Dihedral Angles: Compact angular encoding 1D-4D continuous state Rotamer discretization based on marginal statistics fails to capture fine details Rotamers Truth Rotamers 60 o 180 o 300 o [Shapovalov & Dunbrack 2007]

57 Side Chain Prediction x 1 x 4 x 2 x 3 x 9 x 7 x 8 x 5 x 6 Edges between amino acids within distance threshold. [ Image: Harder et al., BMC Informatics 2010 ]

58 Side Chain Prediction Rotamer Likelihood Atomic Interaction Statistical and physical potential functions. [ Image: Harder et al., BMC Informatics 2010 ]

59 D-PMP for Side Chains 1 Augment Particles 2 RMP Update 3 Select Diverse Continuous optimization of side chains: Capture non-rotameric side chains Conformational diversity Likelihood-based proposals

60 Resolving Ties Particle diversity leads to more conflicts: Side Chain Particles T-PMP D-PMP

61 Resolving Ties Find consistent maximizer: x 1 x 4 x 2 x 3 x 9 x 7 x 8 x 5 x 6 1) Assign non-tied nodes

62 Resolving Ties Find consistent maximizer: x 3 x 7 x 5 x 6 1) Assign non-tied nodes 2) Create subgraph of tied nodes 3) Exact inference (e.g. via junction tree)

63 Rosetta 1 Rotamer Proposal 2 Gradient Optimization 3 Rosetta Energy 3 Accept / Reject Energy model used in FoldIt game Simulated annealing (SA) Monte Carlo Independent chains for multiple optima Replace SA with D-PMP. Use Rosetta as black-box energy method.

64 Protein Side Chain Prediction Log-probability of MAP estimate for 20 Proteins (11 Runs) 370 Proteins G-PMP, T-PMP, D-PMP, Rosetta simulated annealing [Rohl et al., 2004] [ Pacheco et al., ICML 2015 ]

65 Protein Side Chain Prediction Root mean square deviation (RMSD) from x-ray structure. Rosetta G-PMP T-PMP D-PMP Oracle selects best configuration in current particle set.

66 Non-Rotameric Side Chains Rosetta D-PMP Truth Rotamers Not all side chains obey standard rotamer discretization. Penicillin Acylase Complex, Trp154 [Shapovalov & Dunbrack 2007]

67 Protein Side Chain Prediction

68 Protein Side Chain Prediction

69 Summary Particle-based approximation for MAP inference in continuous graphical models. Many applications possible Diverse selection procedure avoids degenerate particles. Human Pose Estimation Human Pose Tracking Structure Prediction Code Available: cs.brown.edu/~pachecoj