CS420/520 Algorithm Analysis Spring 2010 Lecture 04

Transcription

1 CS420/520 Algorithm Analysis Spring 2010 Lecture 04 I. Chapter 4 Recurrences A. Introduction 1. Three steps applied at each level of a recursion a) Divide into a number of subproblems that are smaller instances of the same problem b) Conquer solve the subproblems recursively unless the subproblems from a base case in which they are solved in some direct manner c) Combine through some mechanism combine the intermediate subproblem results into the results for the entire problem; sometimes involves solving subproblems that aren t quite the same as the original problem 2. Recurrences. a) a recurrence is an equation or inequality that describes a function in terms of its values on smaller inputs (1) MERGE-SORT worst case running time (2) (3) is the solution we saw in Chapter 2 b) Recurrences need not split problem into equal-sized subparts (1). c) Sometimes we just chip away at the problem. (1) 3. Text will cover a) Substitution method guess a bound and then use mathematical indication to prove our guess correct b) Recursion-tree method nodes represent costs incurred at various levels of the recursion and use techniques for bounding summations to solve the recurrence c) Master Method provides bounds for some recurrences of the form, where a 1, b 1 and f(n) is a given function 4. Notes from Baase & Van Gelder a) Common Recurrence Equations (page 133 ff; notation converted to match our text book) (1) Intro (a) subproblems refers to a smaller instance of the main problem to be solved by a recursive call a and b are constants in each problem (b) (2) Divide and Conquer (a) The sizes of subproblems are known to be n/2 or some other fixed fraction of n (b) In general, the main problem of size n can Spring

2 II. be divided into a subproblems (a 1) of size n/b (b > 1). There is also some nonrecursive cost f(n) (to split the problem into subproblems and/or combine the solutions of the subproblems into a solution of the main problem) (c) a is called the branching factor (d) T ( n) at n / b f ( n ) (3) Chip and Conquer (a) Main problem of size n can be chipped down to one subproblem of size n b, where b > 0, with nonrecursive cost f(n) to create the subproblem and/or extend the solution of the subproblem into a solution of the overall problem. (i) T(n) = T(n - b) + f(n) (4) Chip and Be Conquered (a) The main problem of size n can be chipped down to a subproblems (a 1) each of size n b, where b > 0, with nonrecursive cost f(n) to split up the problem into subproblems and/or to combine the solutions of the subproblems into a solution of the main problem. Call a the branching factor (i) T(n) = a T(n - b) + f(n) (b) If the subproblems have various sizes, all within some range n b max to n b min, then upper and lower bounds can be obtained using b max and b min, respectively in place of b in the equations 5. Sometimes have recurrences that are not equalities but rather inequalities such as a) Such a recurrence states only an upper bound on T(n) so we will use O-notation rather than -notation b) then the recurrence gives only a lower bound and we will use -notation 6. Technicalities. a) assumption of integer arguments to functions (floor & ceiling notation) b) MERGE-SORT is really (1). c) typically ignore boundary conditions (1) generally have T(n) (1) for sufficiently small n d) when we solve and state recurrences (1) omit floors, ceilings, and boundary conditions (2) they do sometimes matter 4.1 The maximum-subarray problem Spring

3 A. Problem setup 1. You are allowed to buy a single unit of stock once and sell it at a later date 2. Buying and selling are after the close of trading for the day 3. You are allowed to learn what the price of the stock will be in the future 4. Goal is to maximize profit B. You might not be able to buy at the lowest process then sell at the highest price within a given period 1. For example lowest price may occur in a future day after the highest price C. A brute force solution 1. Try every possible pair of buy and sell dates in which the buy date precedes the sell date 2. Combinatiorial number of pairs to check 3. Evaluating each combinations in at best constant time places a lower bound on the running time of the algorithm for a list of length n, D. A transformation 1. Might we design an algorithm so that? 2. Find a sequence of days over which the net change from the first day to the last day is maximized 3. Instead of looking at daily price look a daily change in price (first differences which approximate slope of first derivative) and place these differences in array A 4. Looking for non-empty, contiguous subarray of A whose values have the largest sum 5. Still looks like we have to check subarrays for a period of n days a) Can organize the computation so that each subarray takes constant time to compute, given the values of previously computed sub array sums b) So can bound the brute force solution above and below by n 2,. E. A solution using divide-and-conquer 1. Suppose we want a maximum-subarray of the subarray A[low high] a) Divide into two subarrays of as equal size as possible b) A[low mid] and A[mid+1 high] 2. Figure 4.4 (a) page 71 analysis a) Any contiguous subarray A[i j] of A[low high] must lie in exactly one of the following places (1) Entirely in the subarray A[low mid] so that low i j mid (2) Entirely in the subarray A[mid+1 high] so that mid i j high (3) Crossing the midpoint so that low i mid j Spring

4 high b) A maximum subarray of A[low high] must lie in exactly one of these places c) Need to find a maximum subarray that crosses the midpoint and take a subarray with the largest sum of the three d) Finding a maximum subarray crossing the midpoint in time linear in the size of the subarray A[low high] (1) Not a smaller instance of the original problem because it has the added restriction that the subarray it chooses must cross an endpoint (2) Any subarray crossing the midpointis itself made of two subarrays A[i mid] A[mid+1 j] where low i mid j high (3) We need to find the maximum subarrays of the form A[i mid] A[mid+1 j] and then combine them 3. FIND-MAX-CROSSING-SUBARRAY algorithm a) Analysis of linear runtime on page FIND-MAXIMUM-SUBARRAY algorithm a) Divide and conquer algorithm for the problem b) Note that lines 6-11 are the combine parts of the divide-andconquer algorithm 5. Analyzing the divide-and-conquer algorithm a) First assume that the original problem is a power of 2 in size b) c) d) Later we ll see that III. 4.2 Strassen s algorithm for matrix multiplication A. Standard method for multiplying two square matrices 1. SQUARE-MATRIX-MULTIPLY (A,B) a) Due to triple-nested for loops, each running n times and each operation is constant time, total time T(n) O(n 3 ) b) Input matrices A and B c) Output matrix C. B. What about the lower bound? 1. Is it T(n) (n 3 )? 2. No because we have a way to multiply square matrics in T(n) o(n 3 ) 3. [So ask class what is the difference between O(n 3 ) and o(n 3 )] 4. Strassen s method T(n) O(n lg7 ) C. A simple divide-and-conquer algorithm 1. Assume that n is a power of two (to avoid problems with floor/ceiling when doing divide steps) 2. In each divide step, divide n x n matrices into four n/2 x n/2 matrices 3. Partitions produced and method of combination a), Spring

5 D. Algorithm SQUARE-MATRIX-MULTIPLY-RECURSIVE(A,B) 1. How does line 5 actually work? a) We have to avoid a naïve approach to creating the 12 submatrices in the partition which could take at most O(n 2 ) time copying entries b) [ASK STUDENTS TO EXPLAIN WHY O(n 2 ) WOULD BE NEEDED]. 2. Use row/column indexes for submatrices a) Can execute line 5 in O(1) time b) [COMMENT that it makes no difference asymptotically to the overall runtime whether we copy or partition in place] 3. Runtime of SQUARE-MATRIX-MULTIPLY-RECURSIVE(A,B) a) When n = 1, T(n) (1) because just one scalar multiplication on line 4 b) When n > 5 (1) Line 5 T(n) (1) using index calculations (2) Lines 6-9 make eight recursive calls 8T(n/2) (3) Lines 6-9 Four matrix additions, each containing n 2 /4 so each adds (n 2 ) time (4) T(n) = (1) + 8T(n/2) + (n 2 ) (5) T(n) = 8T(n/2) + (n 2 ) c) [COMMENT now we see why it makes no difference asymptotically to the overall runtime whether we copy or partition in place] d) e) This recurrence has solution T(n) ( n 3 ) f) Analysis page 78 why we the asymptotic notation summarizes the constant terms in the work we just analyzed g) Though asymptotic notation subsumes constant multiplicative factors, the recursive notation at(n/b) does not. 4. So we haven t done any better asymptotically from our initial approach E. Strassen s Method 1. Key is to reduce the number of recursive multiplications from 8 to 7 a) This increases the number of additions required for a solution but reduces by a matrix multiplication 2. Steps a) Step 1: Divide the input matrices A and B and output matrix C into n/2 x n/2 matrices as Equation 4.9 Spring

6 (1) Step takes (1) using index calculations b) Step 2: Create 10 different matrices S 1, S 2, S 10 each of which is an n/2 x n/2 matrix and each is the sum or difference of two matrices created in Step 1 (1) This process takes (n 2 ) time c) Step 3: using the submatrices from Step 1 and the matrices from Step 10, recursively compute seven matrix products P 1, P 2, P 7 each of which is an n/2 x n/2 matrix d) Step 4: compute the result submatrices C 11, C 12, C 21, C 22 by adding and subtracting various combinations of the P i matrices (1) This process takes (n 2 ) time 3. Initial Analysis a) When n = 1, T(n) (1) because just one scalar multiplication on line 4 b) When n > 1 Steps 1, 2, and 4 take a total of (n 2 ) time and Step 3 takes seven multiplications of n/2 x n/2 matrices c). d) So have traded off one matrix multiplication for a constant number of matrix additions 4. Details of Step 2 Creation of S 1, S 2,, S 10 (page 80) 5. Details of Step 3 Creation of P 2,, P 7 (page 80) 6. Details of Step 4 Creation of C 11, C 12, C 21, C 22 (pages 80-81) a) We add or subtract n/2 x n/2 matrices eight times so the step takes (n 2 ) time F. Overall, Recurrence has solution T(n) ( n lg7 ) IV. 4.3 The substitution method for solving recurrences. A. The substitution method - guess a bound and then use mathematical induction to prove our guess correct 1. guess the form of the solution 2. use mathematical induction to find constants and prove that the solution works 3. can establish either upper (Big-Oh) or lower bounds (Big-Omega) on a recurrence 4. may have to extend boundary conditions to make the inductive assumption work for small n 5. (see hand notes pages 2 and 2A). B. making a good guess 1. heuristics a) when n is large, what is the behavior b) prove loose upper and lower bounds on the recurrence and then reduce the range of uncertainty 2. (see hand notes pages 3A). C. subtleties 1. inductive assumption may not be strong enough to prove the detailed bound Spring

7 a) page 85 example, the result of the substitution results in (cn + 1) which is not the exact form of the guessed asymptotic bound and is greater than it as well b) Off only by a constant of revise the guess by subtracting a lower-order term a) seeking <something you want> - <something greater than 0>. 3. can prove something stronger for a given value by assuming something stronger for smaller values D. avoiding pitfalls 1. we need the exact form of the hypothesis to result from the substitution method s computations E. (hand example page 3B, 4) F. changing variables 1. example pages V. The iteration method (not covered in the CLRS 3 rd Edition)- converts the recurrence into a summation and then relies on techniques for bounding summations to solve the recurrence A. doesn't require a guess B. may require more algebra than the substitution method C. solve by expanding the recurrence and express it as a summation of terms dependent only on n and the initial conditions D. the key is to focus on two parameters 1. the number of times the recurrence needs to be iterated to reach the boundary condition 2. the sum of the terms arising from each level of the iteration process 3. often assume recurrence defined only on exact powers of a number E. (see hand notes page 5 and 5a). VI. 4.4 The recursion tree method for solving recurrences A. Introduction 1. Each node represents the cost of a single subproblem somewhere in the set of recursive function invocations 2. Sum the costs within each level of the tree to obtain a set of perlevel costs 3. Sum all per-level costs to determine total cost of recursion 4. Best used to generate a good guess which is then verified by the substitution method 5. If careful, can use as a direct proof of solution B. Example pg ) 1. When do we hit the base case? 2. The subproblem size for a node at depth i is n/4 i, so solve n/4 i = 1 and solution i = log 4 n 3. Number of levels includes 1 for depth 0 to log 4 n 4. Add up costs over all levels in the tree; a decreasing geometric series 5. In this case, cost of the root dominates the total cost of the tree 6. Check guess via the substitution method C. Example pg ) 1. Longest simple path from root to a leaf for Upper Bound analysis 2. (check math on page 92). 3. We expect solution to the recurrence to be cost of each level * Spring

8 number of levels 4. In this case, consider the cost of the leaves which if the cost of each leaf is constant is which is but the tree is not a complete binary tree and has fewer than leaves 5. Check guess O(nlgn) with substitution method. D. Recursion Tree Rules (Baase & Van Gelder pp136 ff) 1. The work copy of the recurrence equation uses a different variable from the original copy; it is called the auxiliary variable. Let k be the auxiliary variable for the purposes of discussion. The left side of the original copy of the recurrence equation (assume it is T(n)) becomes the size field of the root node for the recursion tree 2. An incomplete node has a value for its size field, but not for its nonrecursive cost 3. The process of determining the nonrecursive cost field and the children of an incomplete node is called the expansion of that node. We take the size field in the node to be expanded and substitute it for the auxiliary variable k in our work copy of the recurrence equation. The resulting terms containing T on the right side of that equation become children of the node being expanded; all remaining terms become that node s nonrecursive cost 4. Expanding a base-case size gives the nonrecursive cost field and no children. (usually assume base case cost of 1) E. Recursion tree evaluation (Baase & Van Gelder pp136 ff) 1. Size of field of root = nonrecursive cost of expanded nodes + size fields of incomplete nodes 2. First sum the nonrecursive costs of all nodes at the same depth (the row-sum for that tree depth) 3. Sum over those row-sums over all depths 4. Usually need to know the maximum depth of the recursion tree (the depth at which the size parameter reduces to a base case) F. Divide-and-Conquer, General Case (Baase & Van Gelder pp137 ff) 1. The size parameter decreases by a factor of b each time the depth increases 2. Base case occurs when (n / b D ) = 1, where D is the node depth for base-case-nodes 3. D = lg(n) / lg (b) (log(n)) 4. Do not assume that row-sums are the same at all depths 5. How many leaves does a tree have? a) Branching factor a b) Number of nodes at depth D is L = a D c) Transform by lg(l) = D lg(a) = (lg(a))/(lg(b)) * lg(n) d) The critical exponent is E = lg(a) / lg(b) = log b a e) Therefore the number of leaves in a recursion tree is approximately L = n E G. Chip and Conquer, or Be conquered (Baase & Van Gelder pp139 ff) 1. If the branching factor, a > 1, then a) T(n) = a T(n - b) + f(n) b) The total for the tree is exponential in n even when the most favorable assumption f(n) = 1 is made Spring

9 c) T( n) n/ b n/ b d n/ b f ( bh) a f ( n bd) a h d 0 h 0 a d) in most practical cases the last sum is (1), then T(n) ( a n/b ), T(n) 1 1 T(n b) 1 T(n b) 1 2 T(n 2b) 1 T(n 2b) 1 T(n 2b) 1 T(n 2b) 1 4 e) 2. if the branching factor a = 1 then n/ b n/ b n d 1 T( n) a f ( n bd) f ( bh) f ( x) dx b d 0 h 0 0 a) if f(n) is a polynomial n α then T(n) ( n α+1 ), b) if f(n) is a polynomial log(n) then T(n) (n log(n)) H. recursion examples stress check by substitution method VII. 4.5 The master method for solving recurrences A. provides bounds for recurrences of a special (but often occurring form) T(n) = at(n/b) + f(n) 1. a 1, b>1 are constants 2. f(n) is an asymptotically positive function B. divide the problem of size n into a subproblems of size (n/b) 1. a subproblems solved recursively in time T(n/b) 2. cost of dividing and combining is described by f(n) C. Little Master Theorem (Baase & Van Gelder pp138 ff) 1. If the row-sums form an increasing geometric series (starting from row 0 at the top of the tree), then T(n) ( n E ), where E is the critical exponent. This means that the cost is proportional to the number of leaves in the recursion tree a) Geometric series D d n 1 ar a r r d If the row-sums remain about constant then T(n) ( f(n) log(n)) 3. If the row-sums form a decreasing geometric series, k 0 k x 1 1 x when r 1 then T(n) ( f(n) ), which is proportional to the cost of the root D. Theorem 4.1 (Master Theorem) 8 (2 2/b ) Spring

10 log b a n 1. in each case comparing f(n) to 2. solution to the recurrence is determined by the larger of the two functions 3. log in case 1, f(n) must be polynomially smaller than b a n a) log b a f(n) must be asymptotically smaller than n by a factor of n for some constant > case 3, f(n) must be polynomially larger... and satisfy the "regularity" condition that a(f(n/b) cf(n). 5. if f(n) falls in these gaps, the master method cannot be used log b a n a) gap between cases 1 and 2 when f(n) is smaller than but not polynomially smaller b) log gap between cases 2 and 3 when f(n) is larger than b a n but not polynomially larger E. Using the master method 1. (examples page 95-96) 2. (examples in hand notes). VIII. 4.6 Proof of the master theorem A The proof for exact powers 1. figure 4.7 assumes n is an exact power of b > 1, where b need not be an integer 2. generalizations page Restatement of theorem page 102 as Lemma 4.4 B Floors and ceilings 1. proof given for upper bounds only Spring