Matrix Product. Matrix-Multiply (A, B) Cost of multiplying these matrices is p q r.

Transcription

1 Dynamic Programming A powerful paradigm for algorithm design. Often leads to elegant and efficient algorithms when greedy or diide-and-conquer don t work. DP also breaks a problem into subproblems, but subproblems are not independent. DP tabulates solutions of subproblems to aoid soling them again.

2 Dynamic Programming Typically applied to optimization problems: many feasible solutions; find one of optimal alue. Key is the principle of optimality: solution composed of optimal subproblem solutions. Example: Matrix Chain Product. A sequence M 1, M 2,..., M n of n matrices to be multiplied. Adjacent matrices must agree on dim.

3 Matrix Product Matrix-Multiply (A, B) 1. Let A be p q; let B be q r. 2. If dim of A and B don t agree, error. 3. for i = 1 to p 4. for j = 1 to r 5. C[i, j] = 0 6. for k = 1 to q 7. C[i, j] + = A[i, k] B[k, j] 8. return C. Cost of multiplying these matrices is p q r.

4 Matrix Chain Consider 4 matrices: M 1, M 2, M 3, M 4. We can compute the product in many different ways, depending on how we parenthesize. (M 1 (M 2 (M 3 M 4 ))) (M 1 ((M 2 M 3 )M 4 )) ((M 1 M 2 )(M 3 M 4 )) (((M 1 M 2 )M 3 )M 4 ) Different multiplication orders can lead to ery different total costs.

5 Matrix Chain Example: M 1 = , M 2 = 100 5, M 3 = Parentheses order ((M 1 M 2 )M 3 ) has cost = Parentheses order (M 1 (M 2 M 3 )) has cost = 75, 000!

6 Matrix Chain Input: a chain M 1, M 2,..., M n of n matrices. Matrix M i has size p i 1 p i, where i = 1, 2,..., n. Find optimal parentheses order to minimize cost of chain multiplying M i s. Checking all possible ways of parenthesizing is infeasible. There are roughly ( 2n n ) ways to put parentheses, which is of the order of 4 n!

7 Principle of Optimality Consider computing M 1 M 2... M n. Compute M 1,k Compute M k+1,n order. = M 1... M k, in some order. = M k+1... M n, in some Finally, compute M 1,n = M 1,k M k+1,n. Principle of Optimality: To optimize M 1,n, we must optimize M 1,k and M k+1,n too.

8 Recursie Solution A subproblem is subchain M i, M i+1..., M j. m[i, j] = optimal cost to multiply M i,..., M j. Use principle of optimality to determine m[i, j] recursiely. Clearly, m[i, i] = 0, for all i. If an algorithm computes M i, M i+1..., M j as (M i,..., M k ) (M k+1,..., M j ), then m[i, j] = m[i, k] + m[k + 1, j] + p i 1 p k p j

9 Recursie Solution m[i, j] = m[i, k] + m[k + 1, j] + p i 1 p k p j We don t know which k the optimal algorithm will use. But k must be between i and j 1. Thus, we can write: m[i, j] = min {m[i, k] + m[k + 1, j] + p i 1p k p j } i k<j

10 The DP Approach Thus, we wish to sole: m[i, j] = min {m[i, k] + m[k + 1, j] + p i 1p k p j } i k<j A direct recursie solution is exponential: brute force checking of all parentheses orders. What is the recurrence? What does it sole to? DP s insight: only a small number of subproblems actually occur, one per choice of i, j.

11 The DP Approach Naie recursion is exponential because it soles the same subproblem oer and oer again in different branches of recursion. DP aoids this wasted computation by organizing the subproblems differently: bottom up. Start with m[i, i] = 0, for all i. Next, we determine m[i, i + 1], and then m[i, i + 2], and so on.

12 The Algorithm Input: [p 0, p 1,..., p n ] the dimension ector of the matrix chain. Output: m[i, j], the optimal cost of multiplying each subchain M i... M j. Array s[i, j] stores the optimal k for each subchain.

13 The Algorithm Matrix-Chain-Multiply (p) 1. Set m[i, i] = 0, for i = 1, 2,..., n. 2. Set d = For all i, j such that j i = d, compute m[i, j] = min {m[i, k] + m[k + 1, j] + p i 1p k p j } i k<j Set s[i, j] = k, where k is the choice that gies min alue in aboe expression. 4. If d < n, increment d and repeat Step 3.

14 Illustration m array M1 = 30 x 35 j ,125 11,875 10, i M2 = 35 x 15 M3 = 15 x 5 M4 = 5 x ,375 7,125 5,375 7,875 4,375 2,500 3, M5 = 10 x 20 M6 = 20 x ,750 2, M1 M2 M3 M4 M5 M6

15 Illustration 1 2 j m array ,125 11,875 10,500 9,375 7,125 5,375 7,875 4,375 2,500 3,500 15,750 2, i 5 6 M1 M2 M3 M4 M5 M6 Computing m[2, 5]. min m[2, 2] + m[3, 5] + p 1 p 2 p 5 = = m[2, 3] + m[4, 5] + p 1 p 3 p 5 = = 7125 m[2, 4] + m[5, 5] + p 1 p 4 p 5 = = 11375

16 Finishing Up The algorithm clearly takes O(n 3 ) time. The m matrix only outputs the cost. The parentheses order from the s matrix. Matrix-Chain (M, s, i, j) 1. if j > i then 2. X Matrix-Chain (A, s, i, s[i, j]) 3. Y Matrix-Chain (A, s, s[i, j] + 1, j) 4. return X Y

17 Longest Common Subsequence Consider a string of characters: X = ABCBDAB. A subsequence is obtained by deleting some (any) characters of X. E.g. ABBB is a subsequence of X, as is ABD. But AABB is not a subsequence. Let X = (x 1, x 2,..., x m ) be a sequence. Z = (z 1, z 2,..., z k ) is subseq. of X if there is an index sequence (i 1,..., i k ) s.t. z j = x ij, for j = 1,..., k. Index sequence for ABBB is (1, 2, 4, 7).

18 Longest Common Subsequence Gien two sequences X and Y, find their longest common subsequence. If X = (A, B, C, B, D, A, B) and Y = (B, D, C, A, B, A), then (B, C, A) is a common sequence, but not LCS. (B, D, A, B) is a LCS. How do we find an LCS? Can some form of Greedy work? Suggestions?

19 Trial Ideas Greedy-1: Scan X. Find the first letter matching y 1 ; take it and continue. Problem: only matches prefix substrings of Y. Greedy-2: Find the most frequent letters of X; or sort the letters by their frequency. Try to match in frequency order. Problem: Frequency can be irreleant. E.g. suppose all letters of X are distinct.

20 Properties 2 m subsequences of X. LCS obeys the principle of optimality. Let X i = (x 1, x 2,..., x i ) be the i-long prefix of X. Examples: if X = (A, B, C, B, D, A, B), then X 2 = (A, B); X 5 = (A, B, C, B, D).

21 LCS Structure Suppose Z = (z 1, z 2,..., z k ) is a LCS of X and Y. Then, 1. If x m = y n, then z k = x m = y n and Z k 1 = LCS(X m 1, Y n 1 ). 2. If x m y n, then z k x m implies Z = LCS(X m 1, Y ). 3. If x m y n, then z k y n implies Z = LCS(X, Y n 1 ).

22 Recursie Solution Let c[i, j] = LCS(X i, Y j ) be the optimal solution for X i, Y j. Obiously, c[i, j] = 0 if either i = 0 or j = 0. In general, we hae the recurrence: c[i, j] = 0 if i or j = 0 c[i 1, j 1] + 1 if x i = y j max{c[i, j 1], c[i 1, j]} if x i y j

23 Algorithm A direct recursie solution is exponential: T (n) = 2T (n 1) + 1, which soles to 2 n. DP builds a table of subproblem solutions, bottom up. Starting from c[i, 0] and c[0, j], we compute c[1, j], c[2, j], etc.

24 Algorithm LCS-Length (X, Y ) c[i, 0] 0, c[0, j] 0, for all i, j; for i = 1 to m do for j = 1 to n do if x i = y j then c[i, j] c[i 1, j 1] + 1; b[i, j] D else if c[i 1, j] c[i, j 1] then c[i, j] c[i 1, j]; b[i, j] U else c[i, j] c[i, j 1]; b[i, j] L return b, c

25 LCS Algorithm LCS-Length (X, Y ) only computes the length of the common subsequence. By keeping track of matches, x i = y j, the LCS itself can be constructed.

26 LCS Algorithm PRINT-LCS(b, X, i, j) if i = 0 or j = 0 then return if b[i, j] = D then PRINT-LCS(b, X, i 1, j 1) print x i elseif b[i, j] = U then PRINT-LCS(b, X, i 1, j) else PRINT-LCS(b, X, i, j 1) Initial call is PRINT-LCS(b, X, X, Y ). By inspection, the time complexity of the algorithm is O(nm).

27 Optimal Polygon Triangulation Polygon is a piecewise linear closed cure. Only consecutie edges intersect, and they do so at ertices. P is conex if line segment xy is inside P wheneer x, y are inside

28 Optimal Polygon Triangulation Vertices in counter-clockwise order: 0, 1,..., n 1. Edges are i i+1, where n = 0. A chord i j joins two non-adjacent ertices. A triangulation is a set of chords that diide P into non-oerlapping triangles

29 Triangulation Problem Gien a conex polygon P = ( 0,..., n 1 ), and a weight function w on triangles, find a triangulation minimizing the total weight. Eery triangulation of a n-gon has n 2 triangles and n 3 chords

30 Optimal Triangulation One possible weight: w( i j k ) = i j + j k + k i But problem well defined for any weight function.

31 Greedy Strategies Greedy 1: Ring Heuristic. Go around each time, skipping one ertex; after logn rounds, done. Motiation joining closeby ertices. Not always optmal. Consider a flat, pancake like conex polygon. The optimal will put mostly ertical diagonals. Greedy s cost is roughly O(log n) times the perimeter.

32 Greedy Strategies Greedy 2: Always add shortest diagonal, consistent with preious selections. Counter-example by Lloyd. P = (A, B, C, D, E), where A = (0, 0); B = (50, 25); C = (80, 30); D = (125, 25); E = (160, 0). Edge lengths are BD = 75; CE < 86; AC < 86; BE > 112; AD > 127. Greedy puts BD, then forced to use BE, for total weight = 187. Optimal uses AC, CE, with total weight = 172.

33 Greedy Strategies GT (S) is within a constant factor of MW T (S) for conex polygons. For arbitrary point set triangulation, the ratio is Ω(n 1/2 ).

34 The Algorithm m[i, j] be the optimal cost of triangulating the subpolygon ( i, i+1,..., j ). Consider the with one side i j. Suppose the 3rd ertex is k. Then, the total cost of the triangulation is: m[i, j] = m[i, k] + m[k, j] + w( i j k )

35 The Algorithm Since we don t know k, we choose the one that minimizes this cost: m[i, j] = j min {m[i, k] + m[k + 1, j] + w( i j k )} i<k<j j 1 m[k, j] k i i+1 m[i, k]

36 All-Pairs Shortest Paths Gien G = (V, E), compute shortest path distances between all pairs of nodes. Run single-source shortest path algorithm from each node as root. Total complexity is O(nS(n, m)), where S(n, m) is the time for one shortest path iteration. If non-negatie edges, use Dijkstra s algorithm: O(m log n) time per iteration. With negatie edges, need to use Bellman-Ford algorithm: O(nm) time per iteration.

37 Floyd-Warshall Algorithm G = (V, E) has ertices {1, 2,..., n}. W is cost matrix. D is output distance matrix. algorithm Floyd-Warshall 1. D = W ; 2. for k = 1 to n 3. for i = 1 to n 4. for j = 1 to n 5. d ij = min{d ij, d ik + d kj } 6. return D.

38 Correctness Pij k : shortest path whose intermediate nodes are in {1, 2,..., k}. Goal is to compute Pij n, for all i, j. i P1 k P2 j Use Dynamic Programming. Two cases: 1. Vertex k not on Pij k. Then, P ij k = P k 1 ij. 2. Vertex k is on Pij k. Then, neither P 1 nor P 2 uses k as an intermediate node. in its interior. (Simplicity of Pij k.) Thus, P k ij = P k 1 ik + P k 1 kj

39 Correctness Recursie formula for P k ij : 1. If k = 0, P k ij = c ij. 2. If k > 0, d k ij = min{dk 1 ij, d k 1 ik + d k 1 kj }

40 Example Matrices D 0 and D 1 :

41 Example Matrices D 2 and D 5 :