5.2 The Master Theorem

170 CHAPTER 5. RECURSION AND RECURRENCES 5.2 The Master Theorem Master Theorem In the last setion, we saw three different kinds of behavior for reurrenes of the form at (n/2) + n These behaviors depended upon whether a<2, a =2,and a>2. Remember that a was the number of subproblems into whih our problem was divided. Dividing by 2 ut our problem size in half eah time, and the n term said that after we ompleted our reursive work, we had n additional units of work to do for a problem of size n. There is no reason that the amount of additional work required by eah subproblem needs to be the size of the subproblem. In many appliations it will be something else, and so in Theorem 5.1 we onsider a more general ase. Similarly, the sizes of the subproblems don t have to be 1/2 the size of the parent problem. We then get the following theorem, our first version of a theorem alled the Master Theorem. (Later on we will develop some stronger forms of this theorem.) Theorem 5.1 Let a be an integer greater than or equal to 1 and b be a real number greater than 1. Let be apositive real number and d a nonnegative real number. Given a reurrene of the form at (n/b)+n d if n =1 then for n apower of b, 1. if log b a<, Θ(n ), 2. if log b a =, Θ(n log n), 3. if log b a>, Θ(n log b a ). Proof: In this proof, we will set d =1,sothat the bottom level of the tree is equally well omputed by the reursive step as by the base ase. It is straightforward to extend the proof for the ase when d 1. Let s think about the reursion tree for this reurrene. There will be log b n levels. At eah level, the number of subproblems will be multiplied by a, and so the number of subproblems at level i will be a i. Eah subproblem at level i is a problem of size (n/b i ). A subproblem of size n/b i requires (n/b i ) additional work and sine there are a i problems on level i, the total number of units of work on level i is ( ) a a i (n/b i ) = n i ( ) a i = n b. b i Reall from above that the different ases for = 1 were when the work per level was dereasing, onstant, or inreasing. The same analysis applies here. From our formula for work on level i, wesee that the work per level is dereasing, onstant, or inreasing exatly when ( a b ) i

5.2. THE MASTER THEOREM 171 is dereasing, onstant, or inreasing. These three ases depend on whether ( a b )is1,less than 1, or greater than 1. Now observe that ( a b )=1 a = b log b a = log b b log b a = Thus we see where our three ases ome from. Now we proeed to show the bound on T (n) inthe different ases. In the following paragraphs, we will use the fats (whose proof is a straightforward appliation of the definition of 1ogartihms and rules of exponents) that for any x, y and z, eah greater than 1, x log y z = z log y x and that log x y = Θ(log 2 y). (See Problem 3 at the end of this setion and Problem 4 at the end of the previous setion.) In general, we have that the total work done is log b n ( ) a i log b n n b = n ( a b ) i In ase 1, (part 1 in the statement of the theorem) this is n times a geometri series with a ratio of less than 1. Theorem 4.4 tells us that log b n n ( ) a i b =Θ(n ). Exerise 5.2-1 Prove Case 2 of the Master Theorem. Exerise 5.2-2 Prove Case 3 of the Master Theorem. In Case 2 we have that a b =1and so log b n n ( a b ) i = n log b n 1 i = n (1 + log b n)=θ(n log n). In Case 3, we have that a b > 1. So in the series log b n ( ) a i log b n n b = n ( a b ) i, the largest term is the last one, so by Theorem 4.4,the sum is Θ ( n ( a b ) logb n ). But n ( a b ) logb n = n a log b n (b ) log b n = n nlog b a n log b b = n nlog b a n = n log b a.

172 CHAPTER 5. RECURSION AND RECURRENCES Thus the solution is Θ(n log b a ). We note that we may assume that a is a real number with a>1 and give a somewhat similar proof (replaing the reursion tree with an iteration of the reurrene), but we do not give the details here. Solving More General Kinds of Reurrenes So far, we have onsidered divide and onquer reurrenes for funtions T (n) defined on integers n whih are powers of b. Inorder to onsider a more realisti reurrene in the master theorem, namely at ( n/b )+n or at ( n/b )+n or even a T ( n/b )+(a a )T ( n/b )+n it turns out to be easiest to first extend the domain for our reurrenes to a muh bigger set than the nonnegative integers, either the real or rational numbers, and then to work bakwards. For example, we an write a reurrene of the form f(x)t(x/b)+g(x) if x b k(x) if 1 x<b for two (known) funtions f and g defined on the real [or rational] numbers greater than 1 and one (known) funtion k defined on the real [or rational] numbers x with 1 x<b. Then so long as b>1itispossible to prove that there is a unique funtion t defined on the real [or rational] numbers greater than or equal to 1 that satisfies the reurrene. We use the lower ase t in this situation as a signal that we are onsidering a reurrene whose domain is the real or rational numbers greater than or equal to 1. Exerise 5.2-3 How would we ompute t(x) inthe reurrene 3t(x/2) + x 2 if x 2 5x if 1 x<2 if x were 7? How would we show that there is one and only one funtion t that satisfies the reurrene? Exerise 5.2-4 Is it the ase that there is one and only one solution to the reurrene f(n)t ( n/b )+g(n) k if n =1 when f and g are (known) funtions defined on the positive integers, and k and b are (known) onstants with b an integer larger than or equal to 2? (Note that n/b denotes the eiling of n/b, the smallest integer greater than or equal to n/b. Similarly x denotes the floor of x, the largest integer less than or equal to x.)

5.2. THE MASTER THEOREM 173 To ompute t(7) in Exerise 5.2-3 we need to know t(7/2). To ompute t(7/2), we need to know t(7/4). Sine 1 < 7/4 < 2, we know that t(7/4) = 35/4. Then we may write Next we may write t(7/2) = 3 35 4 + 49 4 = 154 4 = 77 2. t(7) = 3t(7/2) + 7 2 = 3 77 2 +49 = 329 2. Clearly we an ompute t(x) inthis way for any x, though we are unlikely to enjoy the arithmeti. On the other hand suppose all we need to do is to show that there is a unique value of t(x) determined by the reurrene, for all real numbers x 1. If 1 x<2, then 5x, whih uniquely determines t(x). Given a number x 2, there is a smallest integer i suh that x/2 i < 2, and for this i, wehave 1 <x/2 i. We an now prove by indution on i that t(x) isuniquely determined by the reurrene relation. In Exerise 5.2-4 there is one and only one solution. Why? Clearly T (1) is determined by the reurrene. Now assume indutively that n > 1 and that T (m) isuniquely determined for positive integers m<n. We know that n 2, so that n/2 n 1. Sine b 2, we know that n/2 n/b, sothat n/b n 1. Therefore n/b <n,sothat we know by the indutive hypothesis that T ( n/b ) isuniquely determined by the reurrene. Then by the reurrene, ( ) n f(n)t + g(n), b whih uniquely determines T (n). Thus by the priniple of mathematial indution, T (n) is determined for all positive integers n. Forevery kind of reurrene we have dealt with, there is similarly one and only one solution. Beause we know solutions exist, we don t find formulas for solutions to demonstrate that solutions exist, but rather to help us understand properties of the solutions. In the last setion, for example, we were interested in how fast the solutions grew as n grew large. This is why we were finding Big-O and Big-Θ bounds for our solutions. Reurrenes for general n We will now show how reurrenes for arbitrary real numbers relate to reurrenes involving floors and eilings. We begin by showing that the onlusions of the Master Theorem apply to reurrenes for arbitrary real numbers when we replae the real numbers by nearby powers of b. Theorem 5.2 Let a and b be positive real numbers with b>1 and and d be real numbers. Let t(x) be the solution to the reurrene at(x/b)+x if x b d if 1 x<b.

174 CHAPTER 5. RECURSION AND RECURRENCES Let T (n) be the solution to the reurrene at (n/b)+n if n 0 where n is a nonnegative integer power of b. Letm(x) be the smallest integer power of b greater than or equal to x. Then Θ(T (m(x))) Proof: If iterate (or, in the ase that a is an integer, draw reursion trees for) the two reurrenes, we an see that the results of the iterations are nearly idential. This means the solutions to the reurrenes have the same big-θ behavior. See the Appendix to this Setion for details. Removing Floors and Ceilings We have also pointed out that a more realisti Master Theorem would apply to reurrenes of the form at ( n/b )+n,orat ( n/b )+n,orevena T ( n/b )+(a a )T ( n/b ) +n. For example, if we are applying mergesort to an array of size 101, we really break it into piees, one of size 50 and one of size 51. Thus the reurrene we want is not really 2T (n/2) + n, but rather T ( n/2 )+T ( n/2 )+n. We an show, however, that one an essentially ignore the floors and eilings in typial divide-and-onquer reurrenes. If we remove the floors and eilings from a reurrene relation, we onvert it from a reurrene relation defined on the integers to one defined on the rational numbers. However we have already seen that suh reurrenes are not diffiult to handle. The theorem below says that in reurrenes overed by the master theorem, if we remove eilings, our reurrenes still have the same Big-Θ bounds on their solutions. A similar proof shows that we may remove floors and still get the same Big-Θ bounds. The ondition that b>2 an be replaed by b>1, but the base ase for the reurrene will depend on b. Sine we may remove either floors or eilings, that means that we may deal with reurrenes of the form a T ( n/b )+(a a )T ( n/b )+n Theorem 5.3 Let a and b be positive real numbers with b 2 and let and d be real numbers. Let T (n) be the funtion defined on the integers by the reurrene at ( n/b )+n d n =1, and let t(x) be the funtion on the real numbers defined by the reurrene at(x/b)+x if x b d if 1 x<b. Then Θ(t(n)). The same statement applies with eilings replaed by floors. Proof: As in the previous theorem, we an onsider iterating the two reurrenes. It is straightforward (though dealing with the notation is diffiult) to show that for a given value of n, the iteration for omputing T (n) has at most two more levels than the iteration for omputing

5.2. THE MASTER THEOREM 175 t(n). The work per level also has the same Big-Θ bounds at eah level, and the work for the two additional levels of the iteration for T (n) has the same Big-Θ bounds as the work at the bottom level of the reursion tree for t(n). We give the details in the appendix at the end of this setion. Theorem 5.2 and Theorem 5.3 tell us that the Big-Θ behavior of solutions to our more realisti reurrenes at ( n/b )+n d n=1 is determined by their Big-Θ behavior on powers of the base b. A version of the Master Theorem for more general reurrenes. We showed that in our version of the master theorem, we ould ignore eilings and assume our variables were powers of b. In fat we an ignore them in irumstanes where the funtion telling us the work done at eah level of our reursion tree is Θ(x ) for some positive real number. This lets us apply the master theorem to a muh wider variety of funtions. Theorem 5.4 Theorems 5.3 and 5.2 apply to reurrenes in whih the x term is replaed by a funtion f in whih f(x) =Θ(x ). Proof: We iterate the reurrenes or onstrut reursion trees in the same way as in the proofs of the original theorems, and find that the ondition f(x) =Θ(x ) gives us enough information to again bound one of the solutions above and below with a multiple of the other solution. The details are similar to those in the original proofs. Exerise 5.2-5 If f(x) =x x +1,what an you say about the Big-Θ behavior of solutions to 2T ( n/3 )+f(n) where n an be any positive integer, and the solutions to where n is restrited to be a power of 3? 2T (n/3) + f(n) Sine f(x) =x x +1 x x = x 3/2,wehave that x 3/2 = O(f(x)). Sine x +1 x + x = 2x = 2 x for x>1, we have f(x) =x x +1 2x x = 2x 3/2 = O(x 3/2 ). Thus the big-θ behavior of the solutions to the two reurrenes will be the same.

176 CHAPTER 5. RECURSION AND RECURRENCES Extending the Master Theorem As Exerise 5.2-5 suggests, Theorem 5.4 opens up a whole range of interesting reurrenes to analyze. These reurrenes have the same kind of behavior predited by our original version of the Master Theorem, but the original version of the Master Theorem does not apply to them, just as it does not apply to the reurrenes of Exerise 5.2-5. We now state a seond version of the Master Theorem. A still stronger version of the theorem may be found in CLR, but the version here aptures muh of the interesting behavior of reurrenes that arise from the analysis of algorithms. The ondition that b 2 in this theorem an be replaed by b>1, but then the base ase depends on b and is not the ase with n =1. Theorem 5.5 Let a and b be positive real numbers with a 1 and b 2. LetT (n) be defined by Then at ( n/b )+f(n) 1. if f(n) =Θ(x ) where log b a<, then Θ(n )=Θ(f(n)). 2. if f(n) =Θ(n ), where log b a =, then Θ(n log b a log b n) 3. if f(n) =Θ(n ), where log b a>, then Θ(n log b a ) The same results apply with eilings replaed by floors. Proof: Sine we have assumed that f(n) =Θ(n ), we know by Theorem 5.4 that we may restrit our domain to exat powers of b. Wemimi the original proof of the Master theorem, substituting the appropriate Θ(n ) for f(n) inomputing the work done at eah level. But this means there are onstants 1 and 2, independent of the level, so that the work at eah level is between 1 n ( ) a i b and 2 n ( ) a i b so from this point on the proof is largely a translation of the original proof. Exerise 5.2-6 What does the Master Theorem tell us about the solutions to the reurrene 3T (n/2) + n n +1 ifn>1 1 if n =1? As we saw in our solution to Exerise 5.2-5 x x + 1=Θ(x 3/2 ). Sine 2 3/2 = 2 3 = 8 < 3, we have that log 2 3 > 3/2. Then by onlusion 3 of version 2 of the Master Theorem, Θ(n log 2 3 ).

5.2. THE MASTER THEOREM 177 Appendix: Proofs of Theorems For onveniene, we repeat the statements of the earlier theorems whose proofs we merely outlined. Theorem 5.6 Let a and b be positive real numbers with b>1 and and d be real numbers. Let t(x) be the solution to the reurrene at(x/b)+x if x b d if 1 x<b. Let T (n) be the solution to the reurrene at (n/b)+n if n 0 where n is a nonnegative integer power of b. Letm(x) be the smallest integer power of b greater than or equal to x. Then Θ(T (m(x))) Proof: By iterating eah reursion 4 times (or using a four level reursion tree in the ase that a is an integer), we see that ( x ) ( a ) 3x ( a ) 2x a 4 t b 4 + b + b + a b x and ( n ) ( a ) 3n ( a ) 2n a 4 T b 4 + b + b + a b n Thus ontinuing until we have a solution, in both ases we get a solution that starts with a raised to an exponent that we will denote as either e(x) ore(n) when we want to distinguish between them and e when it is unneessary to distinguish. The solution will be a e times T (x/b e ) plus x or n times a geometri series G(x) = ( ) e a i. b Inboth ases T (x/b e ) (or T (n/b e )) will ) be d. Inboth ases the geometri series will be Θ(1), Θ(e) orθ( a e, b depending on whether a b is less than 1, equal to 1, or greater than one. Clearly e(n) =log b n. Sine we must divide x by b an integer number greater than log b x 1 times in order to get a value in the range from 1tob, e(x) = log b x. Thus if m is the smallest integer power of b greater than or equal to x, then 0 e(m) e(x) < 1. Then for any real number r we have r 0 r e(m) e(x) <r,or r e(x) r e(m) r r e(x).thuswehave r e(x) =Θ(r e(m) ) for every real number r, inluding r = a and r = a b. Finally, x m b x, and so x =Θ(m ). Therefore, every term of t(x) isθof the orresponding term of T (m). Further, there are only a finite number of different onstants involved in our Big-Θ bounds. Therefore sine t(x) isomposed of sums and produts of these terms, Θ(T (m)). Theorem 5.7 Let a and b be positive real numbers with b 2 and let and d be real numbers. Let T (n) be the funtion defined on the integers by the reurrene at ( n/b )+n if n b d n =1,

178 CHAPTER 5. RECURSION AND RECURRENCES and let t(x) be the funtion on the real numbers defined by the reurrene at(x/b)+x if x b d if 1 x<b. Then Θ(t(n)). Proof: As in the previous proof, we an iterate both reurrenes. Let us ompare what the results will be of iterating the reurrene for t(n) and the reurrene for T (n) the same number of times. Note that n/b < n/b +1 n/b /b < n/b 2 +1/b < n/b 2 +1/b +1 n/b /b /b < n/b 3 +1/b 2 +1/b < n/b 3 +1/b 2 +1/b +1 This suggests that if we define n 0 = n, and n i = n i 1 /b, then it is straightforward to prove by indution that n i < n/b i +2. The number n i is the argument of T in the ith iteration of the reurrene for T. Wehave just seen it differs from the argument of t in the ith iteration of t by at most 2. In partiular, we might have to iterate the reurrene for T twie more than we iterate the reurrene for t to reah the base ase. When we iterate the reurrene for t, weget the same solution we got in the previous theorem, with n substituted for x. When we iterate the reurrene for T,weget for some integer j that j 1 a j d + a i n i, with n n b i i n +2. But, so long as n/b i 2, we have n/b i +2 n/b i 1. Sine the number of b i iterations of T is at most two more than the number of iterations of t, and sine the number of iterations of t is log b n, wehave that j is at most log b n +2. Therefore all but perhaps the last three values of n i are less than or equal to n/b i 1, and these last three values are at most b 2, b, and 1. Putting all these bounds together and using n 0 = n gives us or j 1 a i( n b i ) j 1 a i n i j 4 n + a i( n ) + a j 2 b i 1 (b 2 ) + a j 1 b + a j 1, i=1 j 1 a i( n b i ) j 1 a i n i j 4 ( ) n + b a i( n ) + a j 2 b j ( ) b b i b j 2 + a j 1 j ( b b j 1 + a j j b j i=1 As we shall see momentarily these last three extra terms and the b in front of the summation sign do not hange the Big-Θ behavior of the right-hand side. ).

5.2. THE MASTER THEOREM 179 As in the proof of the master theorem, the Big-Θ behavior of the left hand side depends on whether a/b is less than 1, in whih ase it is Θ(n ), equal to 1, in whih ase it is Θ(n log b n), or greater than one in whih ase it is Θ(n log b a ). But this is exatly the Big-Θ behavior of the right-hand side, beause n<b j <nb 2,sob j =Θ(n), whih means that ( b j b i ) =Θ (( n b i ) ), and the b in front of the summation sign does not hange its Big-Θ behavior. Adding a j d to the middle term of the inequality to get T (n) does not hange this behavior. But this modified middle term is exatly t(n). Important Conepts, Formulas, and Theorems 1. Master Theorem, simplified version. The simplified version of the Master Theorem states: Let a be an integer greater than or equal to 1 and b be a real number greater than 1. Let be apositive real number and d a nonnegative real number. Given a reurrene of the form then for n apowerofb, (a) if log b a<, Θ(n ), (b) if log b a =, Θ(n log n), () if log b a>, Θ(n log b a ). at (n/b)+n d if n =1 2. Properties of Logarithms. For any x, y and z, eah greater than 1, x log y z = z log y x. Also, log x y = Θ(log 2 y). 3. Important Reurrenes have Unique Solutions. The reurrene f(n)t ( n/b )+g(n) k if n =1 when f and g are (known) funtions defined on the positive integers, and k and b are (known) onstants with b an integer larger than 2 has a unique solution. 4. Reurrenes Defined on the Positive Real Numbers and Reurrenes Defined on the Positive Integers. Let a and b be positive real numbers with b>1 and and d be real numbers. Let t(x) be the solution to the reurrene Let T (n) be the solution to the reurrene at(x/b)+x if x b d if 1 x<b. at (n/b)+n if n 0 where n is a nonnegative integer power of b. Let m(x) bethe smallest integer power of b greater than or equal to x. Then Θ(T (m(x)))

180 CHAPTER 5. RECURSION AND RECURRENCES 5. Removing Floors and Ceilings from Reurrenes. Let a and b be positive real numbers with b 2 and let and d be real numbers. Let T (n) bethe funtion defined on the integers by the reurrene at ( n/b )+n, d n =1 and let t(x) bethe funtion on the real numbers defined by the reurrene at(x/b)+x if x b d if 1 x<b. Then Θ(t(n)). The same statement applies with eilings replaed by floors. 6. Solutions to Realisti Reurrenes. The theorems summarized in 4 and 5 tell us that the Big-Θ behavior of solutions to our more realisti reurrenes at ( n/b )+n d n=1 is determined by their Big-Θ behavior on powers of the base b. 7. Master Theorem, More General Version. Let a and b be positive real numbers with a 1 and b 2. Let T (n) bedefined by at ( n/b )+f(n) d if n =1 Then (a) if f(n) =Θ(x ) where log b a<, then Θ(n )=Θ(f(n)). (b) if f(n) =Θ(n ), where log b a =, then Θ(n log b a log b n) () if f(n) =Θ(n ), where log b a>, then Θ(n log b a ). The same results apply with eilings replaed by floors. A similar result with a base ase that depends on b holds when 1 <b<2. Problems 1. Use the master theorem to give Big-Θ bounds on the solutions to the following reurrenes. For all of these, assume that T (1) = 1 and n is a power of the appropriate integer. (a) 8T (n/2) + n (b) 8T (n/2) + n 3 () 3T (n/2) + n (d) T (n/4) + 1 (e) 3T (n/3) + n 2 2. Extend the proof of the Master Theorem, Theorem 5.1 to the ase T (1) = d. 3. Show that for any x, y and z, eah greater than 1, x log y z = z log y x.

5.2. THE MASTER THEOREM 181 4. Show that for eah real number x 0 there is one and only one value of T (x) given by the reurrene 7xT (x 1)+1 ifx 1 T (x) = 1 if 0 x<1. 5. Show that for eah real number x 1 there is one and only one value of T (x) given by the reurrene 3xT (x/2) + x 2 if x 2 T (x) = 1 if 1 x<2. 6. How many solutions are there to the reurrene f(n)t ( n/b )+g(n) k if n =1 if b<2? If b =10/9, what would we have to replae the ondition that k if n =1 by in order to get a unique solution? 7. Give a big-θ bound on the solution to the reurrene 3T ( n/2 )+ n +3 ifn>1 8. Give a big-θ bound on the solution to the reurrene 3T ( n/2 )+ n 3 +3 ifn>1 9. Give a big-θ bound on the solution to the reurrene 3T ( n/2 )+ n 4 +3 ifn>1 10. Give a big-θ bound on the solution to the reurrene 2T ( n/2 )+ n 2 +3 ifn>1