Chapters 1 and 5.2 of GT

Subject 2 Spring 2014 Growth of Functions and Recurrence Relations Handouts 3 and 4 contain more examples;the Formulae Collection handout might also help. Chapters 1 and 5.2 of GT Disclaimer: These abbreviated notes DO NOT substitute the textbook for this class. They should be used IN CONJUNCTION with the textbook and the material presented in class. If there is a discrepancy between these notes and the textbook, ALWAYS consider the textbook to be correct. Report such a discrepancy to the instructor so that he resolves it. These notes are only distributed to the students taking this class with A. Gerbessiotis in Spring 2014 ; distribution outside this group of students is NOT allowed. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 1

Growth of Functions Asymptotic Notation When we analyze the performance of algorithms for large input sizes we are concerned only with the rate of growth of running time. The most important term of the running time determines the rate of growth and dominates lower order terms or constants. For example if running time T (n) is T (n) = 2n 3 + 40n 2 + 1000n, then for large n the running time is cubic and the contributions of the quadratic and linear terms can be ignored. We are thus interested in how running time increases with problem size in the limit for large n. The best algorithm is the one with the smallest asymptotic growth. The rate of growth can be determined very informally as follows: (a) drop lower-order terms, and (b) turn remaining multiplicative constants into ones. Thus the original T (n) becomes T (n)?2n 3 after rule (a) is applied, and after we apply rule (b) we end up T (n)?n 3. The question that arises however is how do we describe the relationship between the left side and the right side. It would improper to say for example T (n) = n 3 since we know that T (n) is indeed T (n) = 2n 3 + 40n 2 + 1000n. We sometimes use T (n) Θ(n 3 ), or T (n) = Θ(n 3 ), or simply say the order of growth of T (n) is n 3, or just say T (n) is Θ(n 3 ). All of them are equivalent and correct. However the first two are the ones the textbook is more often using and these are the ones we will follow here. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 2

Growth of Functions Asymptotic notation: An introduction We shall introduce symbols to represent the asymptotic relationship of two functions f(n) and g(n). o (little oh). Think of < or asymptotically less. ω (little omega). Think of > or asymptotically greater than. Θ (big theta, or plain, theta). Think of = or asymptotically equal. O (big oh). Think of or asymptotically less than or equal. Ω (big omega). Think of or asymptoticall greater than or equal to. We shall express the relationship between two functions by writing. Notation 1: f(n) = Θ(g(n)), f(n) = o(g(n)), f(n) = O(g(n)), f(n) = ω(g(n)), f(n) = Ω(g(n)). Notation 2: f(n) Θ(g(n)), f(n) o(g(n)), f(n) O(g(n)), f(n) ω(g(n)), f(n) Ω(g(n)). = (Notation 1) is a misnomer for (Notation 2: the formal correct one). Θ(g(n)), o(g(n)), O(g(n)), ω(g(n)), Ω(g(n)), are SETS of functions. What does f(n) = O(g(n)) mean? It means that the growth of f(n) is slower or equal to the growth of g(n). IT DOES NOT NECESSARILY MEAN that f(n) g(n). For example f(n) = 10n and g(n) = n have the same growth i.e. 10n = O(n), even if 10n n. Of course as 100000n < n 2 for large enough n, 100000n = O(n 2 ), as well. What does f(n) = Ω(g(n)) mean? It means that the growth of f(n) is greater or equal to the growth of g(n). IT DOES NOT NECESSARILY MEAN that f(n) g(n). For example f(n) = n and g(n) = 10n have the same growth i.e. n = Ω(10n), even if n 10n. What does f(n) = Θ(g(n)) mean? It means that the two functions have the same growth, i.e. f(n) = n 2 and g(n) = 100n 2 + 10000n are such that n 2 = Θ(100n 2 + 10000n). What does f(n) = o(g(n)) mean? It means that the growth of f(n) is slower than the growth of g(n). What does f(n) = ω(g(n)) mean? It means the growth of f(n) is faster than the growth of g(n). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 3

Growth of Functions O, Ω, Θ notation Assumption: All functions are asymptotically positive, and c 1, c 2, n 0 are POSITIVE (and thus nonzero) constants 1. Θ Definition. For a function g(n) we denote by Θ(g(n)) the set of all functions f(n) that have the following property. positive constants c 1, c 2, n 0 : 0 c 1 g(n) f(n) c 2 g(n) n n 0. Equivalently we denote the relationship between f(n) and Θ(g(n)) by writing f(n) = Θ(g(n)). 2. Ω Definition. For a function g(n) we denote by Ω(g(n)) the set of all functions f(n) that have the following property. positive constants c 1, n 0 : 0 c 1 g(n) f(n) n n 0. Equivalently we denote the relationship between f(n) and Ω(g(n)) by writing f(n) = Ω(g(n)). 3. O Definition. For a function g(n) we denote by O(g(n)) the set of all functions f(n) that have the following property. positive constants c 2, n 0 : 0 f(n) c 2 g(n) n n 0. Equivalently we denote the relationship between f(n) and O(g(n)) by writing f(n) = O(g(n)). We say respectively that g(n) is an asymptotic tight bound, is an asymptotic lower bound, is an asymptotic upper bound of f(n). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 4

O, Ω, Θ notation Comments Note 1. As Θ(g(n)) is a set, we should have written f(n) Θ(g(n)). As mentioned before, we write f(n) = Θ(g(n)) to denote the membership of f(n). Note 2. Θ(g(n)) = f(n) as the left side is a set, the right side is an element of a set and = is in fact a. It is also meaningless to write f(n) Θ(g(n)), since the left-side is a function (element of a set) and the right side is NOT a function but a set of functions! Note 3. If we write Θ(f(n)) = Θ(g(n)), this is to mean that no matter how we pick a function from the left hand side set, there is a way to choose a function from the right hand side set to make equality to hold! Example 1. Show that n 2 2n = Θ(n 2 ). Proof. We determine POSITIVE CONSTANTS c 1, c 2, n 0 so that for all n n 0. c 1 n 2 n 2 2n c 2 n 2 As n 2 2n n 2 for all n > 0, we have shown that n 2 2n c 2 n 2, for c 2 = 1 and for all n n 1 = 1. This is equivalent to showing that n 2 2n = O(n 2 ) as well. As n 2 /2 n 2 2n for all n 4, we have shown that c 1 n 2 n 2 2n, for c 1 = 1/2 and n n 2 = 4. This is equivalent to showing that n 2 2n = Ω(n 2 ) as well. The n 0 in the definition is the largest of n 1 and n 2 i.e. n 0 = 4. Then, c 1 n 2 n 2 2n c 2 n 2 is true for c 1 = 1/2, c 2 = 1 and for all n n 0 = 4. Note 4. c 1 = 1/2, c 2 = 1 and n 0 = 1000 is also a correct answer to Example 1. OUR OBJECTIVE IS TO FIND A SET OF SATISFYING CONSTANTS not THE BEST SET OF SATISFYING CONSTANTS. Example 2. Since a constant is a degree 0 polynomial any constant c is such that c = Θ(1), c = O(1), and c = Ω(1). Example 3. Show that 1000 = O(1). Proof. There exists constant c = 2000 such that 1000 c 1 = 2000 1 = 2000, for all n 1 = n 0. c = 1000 works also. Be reminded: We don t need to find the best c or n 0. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 5

O, Ω, Θ notation Remarks Lemma 1. f(n) = Θ(g(n)) if and only if f(n) = O(g(n)) and g(n) = O(f(n)). Lemma 2. f(n) = O(g(n)) if and only if g(n) = Ω(f(n)). Lemma 3. Let lim n f(n)/g(n) exists. f(n) = O(g(n)) if and only if lim n f(n)/g(n) c for positive constant c. Lemma 4. Let lim n g(n)/f(n) exists. f(n) = Ω(g(n)) if and only if lim n g(n)/f(n) c for positive constant c. Remark 5. n = O(n), n = O(n 2 ), 1 = O(n). n = Ω(1), n = O(2 lg n ), n 3 = O(n2 n ), n = O(2 n ). Remark 6. Normally, the phrase the running time is O(n 2 ) is meaningless as the running time is expressed by a number and a unit of time (e.g. 6 seconds, 10 milliseconds). This expression when used this way in this class means that the worst-case running time (which is a function of n) is O(n 2 ), and by extension, no matter what particular input of size n is chosen for each value of n, the running time on that set of input instances is O(n 2 ). Remark 7. A running time of Ω(g(n)) for an algorithm means that no matter what the input of size n is, for each value of n, the running time of the algorithm will be at least cg(n), for some constant c for large enough n. For example the running time of insertion-sort is Ω(n). Remark 8. n = Ω(1), 2 3 lg n = Ω(n 2 ), 1000000 = Ω(1), n 3 = Ω(n 2 ), n = Ω(n/ lg n), n! = Ω(n 1000000 ), 1 = Ω(1000000). Remark 9. Transitive property: f(n) = O(g(n)) and g(n) = O(h(n)) then f(n) = O(h(n)) Remark 10. Reflexive property: f(n) = O(f(n)) Remark 11. Symmetric property: f(n) = Θ(f(n)) Remark 12. Antisymmetric property: f(n) = O(g(n)), then g(n) = Ω(f(n)) (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 6

Growth of Functions ω and o and... Θ (again) 1. o(n) For two functions f(n) and g(n) we write f(n) = o(g(n)), if and only if f(n) lim n g(n) = 0 2. ω(n) For two functions f(n) and g(n) we write f(n) = ω(g(n)), if and only if f(n) lim n g(n) = The latter is a consequence of the two definitions above and Lemmas 1,3,4. 3. Θ(n) For two functions f(n) and g(n) we write f(n) = Θ(g(n)), if and only if f(n) lim n g(n) = constant > 0 Lemma 13. If f(n) = o(g(n)), then f(n) = O(g(n)). Lemma 14. If f(n) = ω(g(n)), then f(n) = Ω(g(n)). Lemma 15. f(n) = o(g(n)) if and only if f(n) = O(g(n)), and f(n) Ω(g(n)). Lemma 16. f(n) = ω(g(n)) if and only if f(n) = Ω(g(n)), and f(n) O(g(n)). Lemma 17. f(n) = o(g(n)) if and only if g(n) = ω(f(n)). Lemma 18. Antisymmetric property: f(n) = o(g(n)), then g(n) = ω(f(n)) (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 7

Growth of Functions More Examples Example 4. Show that 2 n = Ω(n). Proof. Since lim 2 n /n = as n, we have 2 n = ω(n). By the Lemma of the previous page we have 2 n = Ω(n) as well. Example 5. Show that n = O(n 3 ). Proof. Since lim n/n 3 = 0 as n, we have n = o(n 3 ). By the Lemma of the previous side we have n = O(n 3 ) as well. Example 6. Show that n 3 = ω(n). Proof. Since lim n 3 /n = as n, we have n 3 = ω(n). The result can also be proved as a consequence of n = o(n 3 ). Example 7. Show that n 2/ lg n = o(n). Proof. Thus n 2/ lg n = o(n). 2/ lg n n n = (2lg n 2/ lg n ) n = 22 n = 4 n 0 Example 8. Show that n! = ω(n). Proof. From Stirling approximation formula, n! (n/e) n. Therefore, n! n. Thus n! = ω(n). Example 9. Show that n 3 = Ω(2 lg n ). Proof. We have that 2 lg n = n. By Example 3, we have n 3 = ω(n), and thus n 3 = Ω(n). Note: All limits in this page are for n. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 8

Recurrences Introduction For a recursive algorithm like mergesort running time is described by a recurrence like the one shown below. T (n) = T (n/2) + T (n/2) + Θ(n) = 2T (n/2) + Θ(n). In order to solve this recurrence we need a boundary condition for the base case of the recurrence. The base case for the running time of merge-sort is the running time for an input of size 1 or in general, of some small constant size. In this case, we may assume that T (1) = 1, or T (1) = Θ(1) i.e. the sorting of a one-key sequence takes one step or constant time. The solution for this recurrence was claimed to be T (n) = Θ(n lg n) (proof by induction or by using a recursion tree). Remark 1. In the recurrence above, we ignored floors and ceilings. When we solve recurrences, we first remove floors/ceilings, show the claimed bounds and then check whether the existence of floors/ceilings would or could have made a difference. For the remainder of this course we assume that values like n/2 are always integer, and so the effect of the removal of floors/ceilings will not be discussed. Remark 2. For boundary conditions we assume in general that T (k) = Θ(1), for some small constant k, as constant problem sizes can be solved in constant time. We use T (k) = Θ(1) instead of a more specific/explicit T (k) = l (l some constant), as such a choice affects only constants in the expression for T (n); i.e. growth rate is preserved. In general, we shall ignore statements about boundary conditions, because this simplifies the recurrence solution. If a boundary condition is explicitly given, the solution must be consistent with it. We introduce three methods to solve recurrences. Substitution or guess and check method: Guess solution and then verify/check it (proof by induction). RecursionTree/Iteration method: Unfold recurrence by turning it into a sum. Master method: Provides a solution formula for recurrences of certain form such as T (n) = at (n/b) + f(n), for constant a > 0 and b > 1 and asymptotically positive f(n). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 9

Substitution Method An example Problem. Solve the recurrence T (n) = 2T (n/2) + n using the substitution (a.k.a. guess-and-check) method. (Implicit assumption is that T (n) is nonnegative and defined for all positive n). Proof. By Induction. Since no boundary condition is given we can thus choose k and l constants greater than zero so that T (k) = l. We choose k, l in such a way to make the inductive proof as simple as possible. A. Guess Step. We guess T (n) = O(n lg n), i.e. constants c 2, n 0 > 0 : T (n) c 2 n lg n for all n n 0. B. Check Step. We are going to show that the guessed solution is the solution to the recurrence. We shall prove our claim by using induction. 1. Base Case of Induction. We choose T (1) = 1. However T (1) = 1 c 2 1 lg 1 = 0. Therefore n 0 cannot be 1, because for n 0 = 1 we can not show that T (1) c 2 1 lg 1. What about n 0 = 2, can we show that T (2) 2c 2 lg 2? We first note that c 2 2 lg 2 = 2c 2. But how much is T (2)? We find this from the recurrence T (2) = 2T (1)+2 = 2T (1)+1 = 2 1 + 2 = 4. Therefore, T (2) c 2 2 lg 2 = 2c 2 as long as 4 2c 2, which means that c 2 must be at least 2. Base case conclusion. The lowest value of n for which T (n) c 2 n lg n is true is n 0 = 2, as long as c 2 2. 2. Induction Hypothesis. We assume that the guessed solution T (n) c 2 n lg n is true for all integer values less than n i.e. that for all i n 1 we have that T (i) c 2 i lg i for all 2 = n 0 i n 1, Remark 1. Under ordinary induction we show that a statement is true for the base case, assume it is true for i = n, and we then show it for i = n + 1. This is, however, equivalent to showing that the statement is true for the base case, and assuming that it is true for i = n 1, then showing that it is true for i = n. This is what is known as ordinary induction. Here we show that the statement is true for the base case, assume that it is true for all values i < n, and show it for i = n. This is sometimes called strong induction. Remark 2. Why the n 1 n induction step, instead of the n n + 1? Answer: It is easier to show T (n) cn lg n rather than T (n + 1) c(n + 1) lg (n + 1). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 10

Substitution Method An example: continued 3. Inductive Step. In the inductive step we show the correctness of our guess for i = n i.e. given that it is true for n 0 i n 1, i.e. we should show that T (n) c 2 n lg n Step 1. As n/2 < n, the induction hypothesis applies to i = n/2. Therefore, by the Induction Hypothesis we have T (n/2) c 2 (n/2) lg (n/2) (1) Step 2. In order to prove the induction step we use the only piece of information that we have available for T (n). This is the recurrence relation, i.e. T (n) = 2T (n/2) + n. We can then combine the two (recurrence and induction hypothesis) as follows (the inequality is a result of Step 1). T (n) = 2T (n/2) + n 2(c 2 (n/2) lg (n/2)) + n = 2(c 2 (n/2)(lg n 1)) + n = c 2 n lg n c 2 n + n (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 11

Substitution Method An example: continued Step 3. In order to show that T (n) c 2 n lg n, the last expression c 2 n lg n c 2 n + n must be less than or equal to c 2 n lg n. This is so provided that c 2 1. (a condition on c 2 is thus established). Therefore for c 2 1 we have that T (n) = 2T (n/2) + n c 2 n lg n c 2 n + n c 2 n lg n Inductive Step Conclusion. We have shown that T (n) c 2 n lg n as long as c 2 1 and n 1. C. Combine Base Case and Induction Step restrictions on c 2 and n 0. For the base case we obtained that c 2 2. In the induction step we got c 2 1. The stronger of the two must be true i.e. c 2 2. For the base case we had n 0 = 2. In the induction step we got n 1 or equivalently n n 1 = 1. The stronger of the two cases is n 2 = max{n 0, n 1 } = 2. D. Conclusion. T (n) c 2 n lg n for any constant c 2 1.5 and for all n 2. We have thus shown the inductive step. This completes the induction. Note that no boundary condition was given. What if a boundary condition was given? (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 12

Substitution Method Boundary Conditions Question 1: What if we were given T (1) = 0? Answer: A solution to the recurrence T (n) = 2T (n/2) + n, T (1) = 0 is T (n) cn lg n for any c 1 and n 1. Question 2: What if we were given T (1) = 3? Answer: Since T (1) = 3, then 3 = T (1) c 1 lg 1 = 0 can not be true. This means that the base case cannot be established for n 0 = 1. We need to try n 0 = 2. From the recurrence we have that T (2) = 2T (1) + 2 = 2 3 + 2 = 8, as T (1) = 3. Then 8 = T (2) < c 2 lg 2 = 2c holds provided that 8 2c, i.e. c 4. A solution to the recurrence T (n) = 2T (n/2) + n, T (1) = 3 is T (n) cn lg n for any c 4 and n 2. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 13

Substitution Method Subtleties Remark 1: How do we guess right? Experience helps, as well as intuition. We try first O(n), O(n lg n), O(n 2 ) in simple cases. Remark 2. Say we guess incorrectly T (n) = O(n) i.e. T (n) cn for the solution of the merge-sort recurrence. In the induction step one should show that T (n) = 2T (n/2) + n cn + n, and then that the latter is at most cn, i.e. cn + n cn. The latter CAN NEVER BE TRUE for positive n. Remark 3. What if we try to solve T (n) = T (n/3) + T (2n/3) + 7? If we try to prove that T (n) cn we end up with proving T (n) cn/3 + c2n/3 + 7 cn + 7 not T (n) cn. How do we fix this tiny problem? By subtracting something smaller than the high order term from the solution i.e. try T (n) cn a. (cn a MUST be positive as well). We then get T (n) cn/3 a + 2cn/3 a + 7 cn a + (7 a). For the assumption to hold it suffices that a 7. Remark 4. What if we try to solve T (n) = T (n/2) + T (n/3) + 10n? Guess T (n) = Θ(n) (note that 1/2 of n/2 plus 1/3 of n/3 is 5/6 < 1). Remark 5. What if we try to solve T (n) = T (2n/3) + T (n/3) + 10n? Guess T (n) = O(n lg n) as 2/3 + 1/3 = 1. T (n) = Θ(n lg n) is the tightest possible bound. Remark 6. What if we try to solve T (n) = T (4n/3) + T (n/3) + 10n? Guess T (n) = O(n α ) or T (n) = Θ(n α ) for some constant α > 1. Remark 7. Useful Tricks: Change of variables might help. Remember to derive the correct boundary conditions when changing variables. T (n) = T ( n) + lg n. Change variables H(m) = T (2 m ). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 14

The Iteration/Recursion Tree Method Introduction If the method is done graphically it is sometimes called the recursion tree method instead of the traditional iteration method. Technique. Expand the recurrence and sum up the residuals. T (n) = 2T ( n/2 ) + n, T (1) = 1. In order to solve this recurrence we use the following identity. For all integers n, a, b n/a /b = n/(ab). In addition T (1) = 2T (0) + 1 = Θ(1). T (n) = n + 2T ( n/2 ) n + 2(n/2 + 2T ( n/4 )) = n + 2n/2 + 4T ( n/4 ) n + 2n/2 + 4n/4 + 8T ( n/8 ) n + 2n/2 + 4n/4 + 8n/8 + 16T ( n/8 ) n + 2n/2 + 4n/4 + 8n/8 +... + 2 i n/2 i + 2 i+1 T ( n/2 i+1 ) = n + 2n/2 + 4n/4 + 8n/8 +... + 2 lg n 1 n/2 lg n 1 + Θ(nT (1)) = n lg n + Θ(nT (1)) = n lg n + Θ(n) Inside T, n/2 j becomes 0 for j > lg n. We substitute for T (1) and then, T (n) can be derived. Things to remember about this method. Keep track of number of iterations/depth of recursion tree. Keep track of sum of terms per iteration/level of recursion tree. Sometimes, the two previous steps and experience allow us to guess the solution correctly. We can then stop the solution with this method and switch to the substitution method instead. Exercise Solve simple recurrence T (n) = 4T (n/2) + n, and T (n) = 2T (n/2) + n, T (8) = 2. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 15

The Iteration Method A detailed Example Problem. Solve the recurrence T (n) = 2T (n/2) + n, T (2) = 5 using the iteration/recursion tree method. Proof. We first rename variables in the recurrence relation substituting i for n to avoid confusion. Substituting n/2 for i we get. Substituting n/4 = n/2 2 for i we get. Substituting n/8 = n/2 3 for i we get. T (i) = 2T (i/2) + i T (n/2) = 2T ((n/2)/2) + n/2 = 2T (n/2 2 ) + n/2 T (n/2 2 ) = T (n/4) = 2T ((n/4)/2) + n/4 = 2T (n/2 3 ) + n/2 2 T (n/2 3 ) = T (n/8) = 2T (n/2 4 ) + n/2 3 Similarly we can substitute n/2 4,..., n/2 i,..., n/2 lg n for i and get similarly stated recurrences. We use all the derived recurrences to expand T (n) by observing that T (n) is 2T (n/2) + n and then by expressing T (n/2) in terms of the derived T (n/2 2 ), then T (n/2 2 ) in terms of T (n/2 3 ) and so on... in terms of T (n/2 i ), then T (n/2 i ) in terms of..., in terms of T (2) which is the base case and which forces a stop to this unfolding of T (n). We then sum the term that appear in the expression for T (n) by observing the existence in the expansion sequence of some symmetries and repetitions that allow the combination of as many terms as possible into a single term thus allowing a closed form solution for T (n) to be found. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 16

The Recursion Tree Method A detailed Example: continued T (n) = 2T (n/2) + n = 2(2T (n/2 2 ) + n/2) + n = 2 2 T (n/2 2 ) + 2 n/2 + n = 2 2 (2T (n/2 3 ) + n/2 2 ) + 2 n/2 + n = 2 3 T (n/2 3 ) + 2 2 (n/2 2 ) + 2 n/2 + n = 2 3 T (n/2 3 ) + n + n + n = 2 3 T (n/2 3 ) + 3n =... = 2 i T (n/2 i ) + i n From the boundary condition T (2) = 5, we decide when to stop the unfolding of T (n). The expansion ends at n/2 i = 2 since then T (n/2 i ) = T (2) = 5. We solve for i by taking logarithms base two of both sides. Then we get that lg n i = 1 ie i = lg n 1. Then, for i = lg n 1, we get that. T (n) = 2T (n/2) + n =... = 2 i T (n/2 i ) + i n = 2 lg n 1 T (n/2 lg n 1 ) + (lg n 1) n = (n/2)t (2) + (lg n 1) n = (n/2)5 + (lg n 1) n = (3n/2) + n lg n Thus we have obtained the following solution to the recurrence: T (n) = 3n/2+n lg n. Compare this with the recursion tree approach of the MergeSort recurrence in Subject 1. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 17

The Iteration Method A detailed Example: continued Question 1. Is the obtained solution the correct one? Answer: Yes, unless we missed something. Question 2. How can we be sure? Answer: Check the solution i.e. make sure that T (n) = 3n/2+n lg n is such that (a) T (2) = 5 (i.e. boundary condition can be verified) and (b) T (n) = 2T (n/2) + n (i.e. recurrence can be verified). Verify boundary condition. T (2) = 3 2/2 + 2 lg 2 = 3 + 2 = 5. Since T (2) = 5 indeed by the boundary condition, our solution satisfies the boundary condition. Verify recurrence. We obtained the solution Substituting n/2 for n we get that T (n) = 3n/2 + n lg n. T (n/2) = 3n/4 + (n/2) lg n/2 = 3n/4 + (n/2)(lg n 1) = n/4 + (n/2) lg n. We start from the right hand side of the recurrence using the preceding equality. 2T (n/2) + n = 2(n/4 + n/2 lg n) + n = 3n/2 + n lg n. The last term is T (n). We have thus proved that for T (n) = 3n/2+n lg n, we have that T (n) = 2T (n/2)+n, i.e. the recurrence is satisfied. By (a) and (b) the solution satisfies both the recurrence and the boundary condition, i.e. it is indeed a solution to the recurrence. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 18

The master method Introduction Technique. Memorization; The master method provides an easy solution method for a certain class of recurrences that are of the form T (n) = at (n/b) + f(n), where a > 0 and b > 1 are constant and f(n) is an asymptotically positive function. Note that n/b can be either a floor(n/b) or a ceiling(n/b). Theorem. For such a recurrence, T (n) is bounded as follows. 1 If f(n) = O(n lg b a ɛ ) for some constant ɛ > 0, then T (n) = Θ(n lg b a ). 2 If f(n) = Θ(n lg b a lg k n), then T (n) = Θ(n lg b a lg k+1 n), where k 0 is a non-negative constant. 3 If f(n) = Ω(n lg b a+ɛ ) for some constant ɛ > 0, and if af(n/b) cf(n) for some constant c < 1 and for large n, then T (n) = Θ(f(n)). 1. Note that a lg n/ lg b = 2 lg a lg n/ lg b = n lg a/ lg b. 2. Consider T (n) = 9T (n/3) + f(n). Then a = 9, b = 3 and lg b a = 2 i.e. n lg b a = Θ(n 2 ). If f(n) = n, then T (n) = Θ(n 2 ). If however f(n) = n 2, then T (n) = Θ(n 2 lg n). 3. T (n) = T (n/2) + f(n), where a = 1 and b = 2, and n lg b a = n 0 = 1. If f(n) = 1, then T (n) = Θ(lg n). If f(n) = n, then because af(n/b) = a n/b = 1 n/ 2 1/2 n = cf(n) for c = 1/2, we get that T (n) = Θ( n). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 19

Substitution vs Recursion Tree An Example Suppose T (n) = T (n/4) + T (3n/4) + n. Try to use a recursion tree instead of using substitution. Expand the T (n) by replacing T (n) with then non T (n) term on the right side (i.e. n) and make that node have two children the other two T (.) terms. T(n) ---> n ----> n ==SUM== n / \ / \ T(n/4) T(3n/4) n/4 3n/4 ==SUM== n 2/ \ 2 / 2 \ 2 2 (n/4 ) (3n/4) (3n/4) (3/4)n ==SUM== n Deepest subtree is the rightmost one of depth lgn /lg(4/3). Total time is thus O(n lgn) (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 20

Appendix Example 1: Asymptotic Notation Use the definitions to show that n 5 25n = Θ(n 5 ). We show first that n 5 25 = O(n 5 ) and then that n 5 25 = Ω(n 5 ). Case 1. Show that n 5 25 = O(n 5 ). For all n 1 we have that n 5 25 n 5 Therefore there exist constant n 2 = 1 and c 2 = 1 such that n 5 25 c 2 n 5 for all n n 2. This proves the claim. Technique 1. What we use in this proof is the fact that n 2 ±An±B is bounded above, for positive A, B, by n 2 +An 2 +Bn 2 (1 + A + B)n 2. Case 2. Show that n 5 25 = Ω(n 5 ) Technique 1 can not be used in this case. The next step is non-trivial. We bound n 5 25 from below by n 5 /2. This is so as long as n 5 25 n 5 /2 n 5 /2 25 n 5 50 n 3 We can do this as long as n is not zero; this is true since for all cases we assume that at least n 1. Therefore there exist constant n 1 = 3 and c 1 = 1/2 such that n 5 25 n 5 /2 for all n n 1 = 3. This proves the claim. Note that 3 is not the best possible constant. We don t need to find the best possible constant but THE EASIEST POSSIBLE! In order to show that n 5 25 = Θ(n 5 ) we need to establish c 1, c 2 and n 0. c 1 and c 2 are 1/2 and 1 respectively. n 0 = max(n 1, n 2 ) = 3. For these values the problem is thus shown. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 21

Appendix Example 2: Iteration method Use the iteration method to solve the recurrence, T (n) = 4T (n/2) + n, T (4) = 6. T (n) = 4T (n/2) + n ( ( n ) ( n )) = 4 4T 2 2 + + n 2 ( n ) = 4 2 T 2 2 + 41 n 2 1 + n ( ( n ) = 4 2 4T 2 3 + = 4 3 T ( n 2 2 )) + 2 1 n + 2 0 n ( n 2 3 ) + 2 2 n + 2 1 n + 2 0 n =... ( n ) = 4 i T 2 i + 2 i 1 n +... + 2 1 n + 2 0 n ( n ) = 4 i T 2 i + n ( 2 i 1 ) The base case is T (4) = 6. We set n/2 i = 4, i.e. n = 2 i+2. If we solve this for i, we get i = lg n 2. Then T (n/2 i ) = T (4) = 6. In addition, 2 i = n/4, 4 i = 2 i 2 i = n/4 n/4 = n 2 /16. Therfore the formula for T (n) becomes. ( ) n T (n) = 4 i T + n ( 2 i 1 ) 2 i = (n 2 /16)T (4) + n(n/4 1) = 3n 2 /8 + n 2 /4 n = 5n 2 /8 n Note. It is easy to verify for example that T (4) = 5 4 2 /8 4 = 6. This can serve as a checking mechanism that the solution that you have established satisfies at least the base case. (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 22

Appendix Example 3: Substitution Method Part 1 Solve the following recurrence using the substitution method: T (n) = T (n/3) + T (n/5) + 90n, T (1) = 45. Part b.1 For n = 1, T (n) is given by the base case and for all other natural values, T (n) is given by the recurrence. We observe that because T (n) = T (n/3) + T (n/5) + 90n, we have that T (n) = T (n/3) + T (n/5) + 90n 90n. Therefore T (n) 90n for all n > 1. Is this, however, true for n = 1 as well? For n = 1 we have 45 = T (1) given by the base case, not the recurrence. However 45 = T (1) 90 1 is false! Therefore we can not show that T (n) 90n for all n 1! This is easily fixable by observing that T (n) = T (n/3) + T (n/5) + 90n 45n. Therefore T (n) 45n for all n > 1, and this is true also for n = 1 since 45 = T (1) 45 1. Therefore T (n) 45n for all n 1, i.e. there exist positive constant c 1 = 45 > 0 and n 1 = 1 such that T (n) c 1 n for all n n 1. This shows that T (n) = Ω(n). Part b.2 We show that T (n) = O(n) using strong induction, i.e we are going to show that there exist positive constants n 2 and c 2 such that T (n) c 2 n for all n n 2. We call this (text inside ) P (n). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 23

Appendix Example 3: Substitution Method Part 2 Base Case. This is for n = 1. We are going to show P (1) is true. For n = 1 we have T (1) = 45. By P (1), 45 = T (1)? c 2 1 is true as long as c 2 45. Therefore P (1) is true for all n 1, as long as c 2 45. Our current values for c 2, n 2 are 45 and 1 respectively. (Strong) Inductive Hypothesis. We are going to use strong induction i.e. we are going to establish that if P(j) is true for all j = n 2,..., n 1, then P (n) will also be true. This is equivalent (since n 2 = 1) to If T (j) c 2 j for all j = 1,..., n 1, then T (n) c 2 n. (Strong) Inductive Step. We show the inductive step using the recurrence. By the Inductive Hypothesis since j = n/3 < n and also j = n/5 < n, we have that T (n/3) c 2 (n/3) and T (n/5) c 2 (n/5). We apply the inductive hypothesis twice in the first and second terms of the first equation equations, we do calculations to derive the fourth, whereas our objective is to establish the inequality on the fifth line. T (n) = T (n/3) + T (n/5) + 90n = c 2 (n/3) + T (n/5) + 90n = c 2 (n/3) + c 2 (n/5) + 90n = c 2 (8n/15) + 90n? c 2 n For the latter to be true we need to have c 2 48 as is shown in detail below. c 2 (8n/15) + 90n? c 2 n 90n? 7c 2 n/15 1350? 7c 2 200? c 2 Therefore the inductive step is true for all n and c 2 200. Again, 200 is not the best possible constant, 1350/7 is less than 200, more accurate but more cumbersome. This (c 2 200) supersedes the c 2 45 of the base case; the induction is valid for all n 1 and c 2 max(45, 200) = 200. Therefore there exist c 2 = 200 and n 2 = 1 such that T (n) c 2 n for all n n 2. Thus T (n) = O(n). We proved O and Ω i.e. T (n) = Θ(n). (c) Copyright A. Gerbessiotis. CS 610-102 : Spring 2014. All rights reserved. 24