Solving NP Hard problems in practice lessons from Computer Vision and Computational Biology

Transcription

1 Solving NP Hard problems in practice lessons from Computer Vision and Computational Biology Yair Weiss School of Computer Science and Engineering The Hebrew University of Jerusalem yweiss Collaborators: Talya Meltzer, Chen Yanover

2 Outline Energy minimization in computer vision and computational biology. Linear Programming relaxations. Using BP to solve LP. Experimental results.

3 Pairwise energy minimization E(x) = <ij> E ij (x i, x j ) + i E i (x i ) Stereo vision. Side-Chain Prediction. Protein Design.

4 Stereo Vision Left Right Disparity (Tsukuba University)

5 Stereo problem as discrete optimization x is disparity image. E(x i, x j ) is compatability cost and E(x i ) is local data cost. x = argmin x i,j E ij (x i, x j ) + i E i (x i ) Old formulation (Marr and Poggio 82) but how do we optimize?

6 Protein Folding Gene sequence Amino-acid sequence 3D structure.

7 Protein folding in two stages Backbone Side Chains

8 Backbone

9 Backbone plus sidechains

10 Side chain prediction as combinatorial optimization Search space: position of each side chain defined by 4 angles. Each angle is one of 3 possibilities. Search space is 81 n. Cost function: local energy and pairwise energies. Goal: find set of angles such that energy is minimal.

11 Protein design Given desired sequence - find Amino-acid sequence.

12 Protein design as combinatorial optimization Search space: amino-acids plus angles. Search space is (2 81) n. Cost function: local energy and pairwise energies. Goal: find set of amino-acids and angles such that energy is minimal.

13 Graphical models Variables = nodes, edges = pairwise energy term

14 MAP in graphical models x = argmin x i,j E ij (x i, x j ) + i E i (x i ) NP Hard for protein folding and stereo vision many approximate algorithms.

15 What s wrong with approximations? Best approximate minimizers: Stereo vision - unsatisfactory results. Side chain prediction - approx. 85% Protein design - good structure often in top 1. Better minimizer or better energy functions?

16 Linear Programming relaxations J(x) = i,j E ij (x i, x j ) + i E i (x i ) q i (x i ), q ij (x i, x j ) are indicator variables. J({q}) = i,j x i,x j q ij (x i, x j )E ij (x i, x j ) + i x i q i (x i )E i (x i )

17 Integer Programming formulation minimize: J({q}) = i,j x i,x j q ij (x i, x j )E ij (x i, x j ) + i x i q i (x i )E i (x i ) subject to: q ij (x i, x j ) {,1} x i,x j q ij (x i, x j ) = 1 x i q ij (x i, x j ) = q j (x j )

18 Linear Programming relaxation minimize: J({q}) = i,j x i,x j q ij (x i, x j )E ij (x i, x j ) + i x i q i (x i )E i (x i ) subject to: q ij (x i, x j ) [,1] x i,x j q ij (x i, x j ) = 1 x i q ij (x i, x j ) = q j (x j )

19 Guarantee of optimality If the LP solution is integer (q ij (x i, x j ) {,1} for all i, j) then we have found the global minimum of J.

20 LP relaxations for vision? State of the art LP solver can be applied to subimage in a machine with 4G memory (34 34 with 2G) #Variables #Constraints LOQO MATLAB, linprog MATLAB, bintprog

21 LP using BP Surprisingly, belief propagation can be used to solve for the linear programming relaxation (Wainwright, Jaakkola and Willsky 3; Weiss, Meltzer and Yanover 5)

22 Belief Propagation: A parallel message-passing algorithm. Every node sends a probability density to its neighbors.

23 From energy function to potentials Ψ ij (x i, x j ) = exp( 1 T E ij(x i, x j )) Ψ i (x i ) = exp( 1 T E i(x i )) Iterate: m ij (x j ) α Ψ ij (x i, x j )Ψ i (x i ) m ki (x i ) x i X k N(X i )\X j From the messages we calclate b i (x i ), b ij (x i, x j ).

24 BP and LP Fixed points of BP correspond to stationary points of the Bethe- Kikuchi free energy (Yedidia, Freeman, Weiss 1, Kabashima Saad 98). F β = i,j +T x i,x j b ij (x i, x j )E ij (x i, x j ) + i i,j x i,x j b ij (x i, x j )ln b ij (x i, x j ) i x i b i (x i )E i (x i ) (d i 1) x i b i (x i )ln b i (x i )

25 BP and LP minimize: F β = i,j +T x i,x j b ij (x i, x j )E ij (x i, x j ) + i i,j x i,x j b ij (x i, x j )ln b ij (x i, x j ) i x i b i (x i )E i (x i ) (d i 1) x i b i (x i )ln b i (x i ) subject to: b ij (x i, x j ) [,1] b ij (x i, x j ) = 1 x i,x j b ij (x i, x j ) = b j (x j ) x i

26 Visualizing BP and LP F β = i,j +T x i,x j b ij (x i, x j )E ij (x i, x j ) + i i,j x i,x j b ij (x i, x j )ln b ij (x i, x j ) i x i b i (x i )E i (x i ) (d i 1) x i b i (x i )ln b i (x i ) b ij (x i, x j ) = ( a b b 1 (a + 2b) )

27 Visualizing BP and LP.5 T=

28 Visualizing BP and LP.5 T=

29 Visualizing BP and LP.5 T=

30 Visualizing BP and LP.5 T=

31 Visualizing BP and LP.5 T=

32 Visualizing BP and LP.5 T=

33 Visualizing BP and LP.5 T=

34 Visualizing BP and LP.5 T=

35 Visualizing BP and LP.5 T=

36 Solving LP by running BP at zero temperature Ψ ij (x i, x j ) = exp( 1 T E ij(x i, x j )) Ψ i (x i ) = exp( 1 T E i(x i )) Iterate: m ij (x j ) αmax x i Ψ ij (x i, x j )Ψ i (x i ) m ki (x i ) X k N(X i )\X j Claim: If sharpened max-product beliefs are feasible they give zero-temperature sum-product beliefs.

37 Visualizing BP and LP.5 T=

38 Visualizing BP and LP.5 T=

39 Visualizing BP and LP.5 T=

40 Visualizing BP and LP.5 T=

41 Visualizing BP and LP.5 T=

42 Visualizing BP and LP.5 T=

43 Visualizing BP and LP.5 T=

44 Visualizing BP and LP.5 T=

45 Is F β convex? F β = i,j +T x i,x j b ij (x i, x j )E ij (x i, x j ) + i i,j x i,x j b ij (x i, x j )ln b ij (x i, x j ) i x i b i (x i )E i (x i ) (d i 1) x i b i (x i )ln b i (x i ) The entropy is always bounded but convexity depends on graph topology (through degree d i ) (Pakzad and Ananthram 3, Heskes 3).

46 Convexifying F F β = i,j +T x i,x j b ij (x i, x j )E ij (x i, x j ) + i i,j x i,x j b ij (x i, x j )ln b ij (x i, x j ) x i b i (x i )E i (x i ) (several convexification possible)

47 Visualizing BP and LP.5 T=

48 Visualizing BP and LP.5 T=

49 Visualizing BP and LP.5 T=

50 Visualizing BP and LP.5 T=

51 Visualizing BP and LP.5 T=

52 Visualizing BP and LP.5 T=

53 Visualizing BP and LP.5 T=

54 Visualizing BP and LP.5 T=

55 Minimizing Convexified F β n ji (x i) = X k N(X i )\X j m ki (x i ) m ij (x j) = α x i Ψ ij (x i, x j )Ψ i (x i )n ji (x i ) m ij (x j ) n ij (x j ) ( m ij (x j) ) γ j ( n ij (x j ) ) γ j 1 ( m ij (x j) ) γ j 1 ( n ij (x j) ) γ j γ j = 1 2 1/d j. two way GBP (Yedidia, Freeman, Weiss 2)

56 Minimizing Convexified F β at zero temperature n ji (x i) = m ij (x j) X k N(X i )\X j m ki (x i ) αmax x i Ψ ij (x i, x j )Ψ i (x i )n ji (x i ) m ij (x j ) = ( m ij (x j) ) γ j ( n ij (x j ) ) γ j 1 n ij (x j ) ( m ij (x j) ) γ j 1 ( n ij (x j) ) γ j A new LP solver?

57 Comparisons Compare run-times of: Generic BP code. CPLEX 9.. solver. Perhaps the most powerful commercial LP

58 Comparisons - stereo #Variables 1 5 #Constraints Tomlab Grid size BP can solve full sized images.

59 Comparisons - protein folding

60 side-chain prediction Variables Constraints Non zeros x x x Protein length [aa] Protein length [aa] Protein length [aa] Variables Constraints Non zeros design x Protein length [aa] x Protein length [aa] x Protein length [aa]

61 Run times - stereo BP TRBP PRIMAL BP TRBP DUAL BP TRBP NET time [min.] time [min.] time [min.] non zeros non zeros non zeros time [min.] BP TRBP BARRIER time [min.] BP TRBP SIFTING time [min.] BP TRBP CONCURRENT non zeros non zeros non zeros

62 Run times - side chain prediction

63 scwrl time [min.] BP TRBP PRIMAL time [min.] BP TRBP DUAL time [min.] BP TRBP NET non zeros x non zeros x non zeros x 1 5 time [min.] BP TRBP BARRIER time [min.] BP TRBP SIFTING time [min.] BP TRBP CONCURRENT non zeros x non zeros x non zeros x 1 5 rosetta time [min.] 1 5 BP TRBP PRIMAL time [min.] BP TRBP DUAL time [min.] BP TRBP NET non zeros x non zeros x non zeros x BP TRBP BARRIER 2 BP TRBP SIFTING 1 BP TRBP CONCURRENT time [min.] 4 2 time [min.] time [min.] non zeros x non zeros x non zeros x 1 5

64 Comparisons - protein design 25 BP TRBP BARRIER time [min.] non zeros x 1 7

65 Discussion BP almost always faster than all solvers in CPLEX. BP can be applied when CPLEX cannot. BP is not a general purpose LP solver.

66 Global Optimum? When LP/BP solution integer. Even when BP solution is partly fractional (Meltzer et al. 5) In about 9% of benchmark problems, we can find global optimum in about 1 minutes.

67 Does global optimum improve performance? Using global optimizers: Stereo vision - unsatisfactory results. Side chain prediction - approx ɛ% Protein design - good structure often in top 1. Better energy functions are needed!

68 Conclusions Energy minimization in computer vision and computional biology. Typically NP hard. Standard LP solvers cannot handle LPs arising from our applications. Convex BP gives a simple, efficient algorithm for solving LP relaxations. Globally optimal solutions can be obtained in minutes.