1 Chapter 5 Recursio ad Recurreces 5.1 Growth Rates of Solutios to Recurreces Divide ad Coquer Algorithms Oe of the most basic ad powerful algorithmic techiques is divide ad coquer. Cosider, for example, the biary search algorithm, which we will describe i the cotext of guessig a umber betwee 1 ad 100. Suppose someoe picks a umber betwee 1 ad 100, ad allows you to ask questios of the form Is the umber greater tha k? where k is a iteger you choose. Your goal is to ask as few questios as possible to figure out the umber. Your first questio should be Is the umber greater tha 50? Why is this? Well, after askig if the umber is bigger tha 50, you have leared either that the umber is betwee oe ad 50, or that the umber is betwee 51 ad 100. I either case have reduced your problem to oe i which the rage is oly half as big. Thus you have divided the problem up ito a problem that is oly half as big, ad you ca ow (recursively) coquer this remaiig problem. (If you ask ay other questio, oe of the possible rages of values you could ed up with would more tha half the size of the origial problem.) If you cotiue i this fashio, always cuttig the problem size i half, you will be able to get the problem dow to size oe fairly quickly, ad the you will kow what the umber is. Of course it would be easier to cut the problem exactly i half each time if we started with aumber i the rage from oe to 128, but the questio does t soud quite so plausible the. Thus to aalyze the problem we will assume someoe asks you to figure out a umber betwee 0 ad, where is a power of 2. Exercise Let T () be umber of questios i biary search o the rage of umbers betwee 1 ad. Assumig that is a power of 2, get a recurrece of for T (). For Exercise we get: T () = T (/2)+1 if 2 1 if =1 (5.1) That is, the umber of guesses to carry out biary search o items is equal to 1 step (the guess) plus the time to solve biary search o the remaiig /2 items. 159
2 160 CHAPTER 5. RECURSION AND RECURRENCES What we are really iterested i is how much time it takes to use biary search i a computer program that looks for a item i a ordered list. While the umber of questios gives us a feel for the amout of time, processig each questio may take several steps i our computer program. The exact amout of time these steps take might deped o some factors we have little cotrol over, such as where portios of the list are stored. Also, we may have to deal with lists whose legth is ot a power of two. Thus a more realistic descriptio of the time eeded would be T () T ( /2 )+C1 if 2 C 2 if =1, (5.2) where C 1 ad C 2 are costats. It turs out that the solutio to (5.1) ad (5.2) are roughly the same, i a sese that will hopefully become clear later. This is almost always the case; we will come back to this issue. For ow, let us ot worry about floors ad ceiligs ad the distictio betwee thigs that take 1 uit of time ad thigs that take o more tha some costat amout of time. Let s tur to aother example of a divide ad coquer algorithm, mergesort. Ithis algorithm, you wish to sort a list of items. Let us assume that the data is stored i a array A i positios 1 through. Mergesort ca be described as follows: MergeSort(A,low,high) if (low == high) retur else mid = (low + high) /2 MergeSort(A,low,mid) MergeSort(A,mid+1,high) Merge the sorted lists from the previous two steps More details o mergesort ca be foud i almost ay algorithms textbook. Suffice to say that the base case (low = high) takes oe step, while the other case executes 1 step, makes two recursive calls o problems of size /2, ad the executes the Merge istructio, which ca be doe i steps. Thus we obtai the followig recurrece for the ruig time of mergesort: T () = 2T (/2) + if >1 1 if =1 (5.3) Recurreces such as this oe ca be uderstood via the idea of a recursio tree, which we itroduce i the followig. This cocept will allow us to aalyze recurreces that arise i dividead-coquer algorithms, ad those that arise i other recursive situatios, such as the Towers of Haoi, as well. A recursio tree for a recurrece is a visual ad coceptual represetatio of the process of iteratig the recurrece.
3 5.1. GROWTH RATES OF SOLUTIONS TO RECURRENCES 161 Figure 5.1: The iitial stage of drawig a recursio tree /2 Recursio Trees We will itroduce the idea of a recursio tree via several examples. It is helpful to have a algorithmic iterpretatio of a recurrece. For example, (igorig for a momet the base case) we ca iterpret the recurrece T () =2T (/2) + (5.4) as i order to solve a problem of size we must solve 2 problems of size /2 ad do uits of additioal work. Similarly we ca iterpret T () =T (/4) + 2 as i order to solve a problem of size we must solve 1 problems of size /4 ad do 2 uits of additioal work. We ca also iterpret the recurrece T () =3T ( 1) + as i order to solve a problem of size, wemust solve 3 subproblems of size 1 ad do additioal uits of work. I Figure 5.1 we draw the begiig of the recursio diagram for (5.4). For ow, assume is apower of2.arecursio tree diagram has three parts. O the left, we keep track of the problem size, i the middle we draw the tree, ad o right we keep track of the work doe. We draw the diagram i levels, each level of the diagram represetig a level of recursio. Equivaletly, each level of the diagram represets a level of iteratio of the recurrece. So to begi the recursio tree for (5.4), we show i level 0 o the left that we have problem of size. The by drawig aroot vertex with two edges leavig it, we show i the middle that we are splittig it ito 2 problems. We ote o the right that we do uits of work i additio to whatever is doe o the two ew problems we created. So that the diagram cotais the relevat iformatio, we fill i part of level oe. We draw two vertices i the middle represetig the two problems ito which we split our mai problem ad show o the left that each of these problems has size /2. You ca see how the recurrece is reflected i levels 0 ad 1 of the recursio tree. The top vertex of the tree represets T (), a the ext level we have two problems of size /2, givig us the recursive term 2T (/2) of our recurrece. The after we solve these two problems we retur to level 0 of the tree ad do additioal uits of work for the orecursive term of the recurrece.
4 162 CHAPTER 5. RECURSION AND RECURRENCES Figure 5.2: Four levels of a recursio tree. /2 /2 + /2 = /4 /4 + /4 + /4 + /4 = /8 8(/8) = Now we cotiue to draw the tree i the same maer. Fillig i the rest of level oe ad addig a few more a few more levels, we have Figure 5.2. Let us summarize what the diagram tells us so far. At level zero (the top level), uits of work are doe. We see that at each succeedig level, we halve the problem size ad double the umber of subproblems. We also see that at level 1, each of the two subproblems requires /2 uits of additioal work, ad so a total of uits of additioal work are doe. Similarly level 2 has 4 subproblems of size /4 ad so 4(/4) uits of additioal work are doe. To see how iteratio of the recurrece is reflected i the diagram, we iterate the recurrece oce, gettig T () = 2T (/2) + T () = 2(2T (/4) + /2) + T () = 4T (/4) + + =4T (/4) + 2 If we examie levels 0, 1, ad 2 of the diagram, we see that at level 2 we have four vertices represetig four problems each of size /4. This gives us the recursive term that we got after iteratig the recurrece. However after we solve these problems we retur to level 1 where we twice do /2 additioal uits of work ad to level 0 where we do aother additioal uits of work. I this way each time we add a level to the tree we are showig the result of oe more iteratio of the recurrece. We ow have eough iformatio to be able to describe the recursio tree diagram i geeral. To do this, we eed to determie, for each level, three thigs umber of subproblems, size of each subproblem, total work doe. We also eed to figure out how may levels there are i the recursio tree.
5 5.1. GROWTH RATES OF SOLUTIONS TO RECURRENCES 163 We see that for this problem, at level i, wehave 2 i subproblems of size /2 i. Further, sice a problem of size 2 i requires 2 i uits of additioal work, there are (2 i )[/(2 i )] = uits of work doe per level. To figure out how may levels there are i the tree, we just otice that at each level the problem size is cut i half, ad the tree stops whe the problem size is 1. Therefore there are log 2 +1levels of the tree, sice we start with the top level ad cut the problem size i half log 2 times. 1 We ca thus visualize the whole tree i Figure 5.3. Figure 5.3: A fiished recursio tree diagram. log +1 levels /2 /4 /2 + /2 = /4 + /4 + /4 + /4 = /8 8(/8) = 1 (1) = The bottom level is differet from the other levels. I the other levels, the work is described by the recursive part of the recurrece, which i this case is T () =2T (/2) +. Atthe bottom level, the work comes from the base case. Thus we must compute the umber of problems of size 1 (assumig that oe is the base case), ad the multiply this value by T (1). For this particular recurrece, ad for may others we will see, it turs out that if you compute the amout of work o the bottom level as if you were computig the amout of additioal work required after you split a problem of size oe ito 2 problems (which, of course, makes o sese) it will be the same value as if you compute it via the base case. Had we chose to say that T (1) some costat other tha 1, this would ot have bee the case. We emphasize that the correct value always comes from the base case: it is just a coicidece that it sometimes also comes from the recursive part of the recurrece. The bottom level of the tree represets the fial stage of iteratig the recurrece; at this level we have problems each requirig work T (1) = 1, ad after we solve them we have to do all the additioal work from all the earlier levels. Thus iteratio of the recurrece tells us that the solutio to the recurrece is the sum of all the work doe at all the levels of the recursio tree. This is exactly how we use the recursio tree to write dow a solutio to a recurrece. The importat thig is that we ow kow exactly how may levels there are, ad how much work is doe at each level. Oce we kow this, we ca sum the total amout of work doe over all the levels, givig us the solutio to our recurrece. I this case, there are log 2 +1 levels, ad at each level the amout of work we do is uits. Thus we coclude that the total amout 1 To simplify otatio, for the remaider of the book, if we omit the base of a logarithm, it should be assumed to be base 2.
6 164 CHAPTER 5. RECURSION AND RECURRENCES Figure 5.4: A recursio tree diagram for Recurrece 5.5. log + 1 levels /2 /4 /2 /4 /8 /8 1 1 of work doe to solve the problem described by recurrece (5.4) is (log 2 + 1). The total work doe throughout the tree is the solutio to our recurrece, because the tree simply models the process of iteratig the recurrece. Thus the solutio to recurrece (5.3) is T () =(log + 1). More geerally, we ca cosider a recurrece that it idetical to (5.3), except that T (1) = a, for some costat a. Ithis case, T () =a + log, because a uits of work are doe at level 1 ad additioal uits of work are doe at each of the remaiig log levels. It is still true that, T () =Θ( log ), because the differet base case did ot chage the solutio to the recurrece by more tha a costat factor. Sice oe uit of time will vary from computer to computer, ad sice some kids of work might take loger tha other kids, it is the big-θ behavior of T () that is really relevat. Thus although recursio trees ca give us the exact solutios (such as T () =a+ log above) to recurreces, we will ofte just aalyze a recursio tree to determie the big-θ or eve, i complicated cases, just the big-o behavior of the actual solutio to the recurrece. I Problem?? we explore whether the value of T (1) actually iflueces the big-θ behavior of the solutio to a recurrece. Let s look at oe more recurrece. T () = T (/2) + if >1 1 if =1 (5.5) Agai, assume is a power of two. We ca iterpret this as follows: to solve a problem of size, wemust solve oe problem of size /2 ad do uits of additioal work. We draw the tree for this problem i Figure 5.4. We see i this figure that the problem sizes are the same as i the previous tree. The rest, however, is differet. The umber of subproblems does ot double, rather it remais at oe o each level. Cosequetly the amout of work halves at each level. Note that there are still log +1levels, as the umber of levels is determied by how the problem size is chagig, ot by how may subproblems there are. So o level i, wehave 1problem of size /2 i, for total work of /2 i uits.
7 5.1. GROWTH RATES OF SOLUTIONS TO RECURRENCES 165 We ow wish to compute how much work is doe i solvig a problem that gives this recurrece. Note that the additioal work doe is differet o each level, so we have that the total amout of work is ( + /2+/ = ( 1 log2 4 2) ) + +, which is times a geometric series. By Theorem 4.4, the value of a geometric series i which the largest term is oe is Θ(1). This implies that the work doe is described by T () =Θ(). We emphasize that there is exactly oe solutio to recurrece (5.5); it is the oe we get by usig the recurrece to compute T (2) from T (1), the to compute T (4) from T (2), ad so o. What we have doe here is show that T () =Θ(). We have ot actually foud a solutio. I fact, for the kids of recurreces we have bee examiig, oce we kow T (1) we ca compute T () for ay relevat by repeatedly usig the recurrece, so there is o questio that solutios do exist. What is ofte importat to us i applicatios is ot the exact form of the solutio, but a big-o upper boud, or, better, a Big-Θ boud o the solutio. Exercise Fid a big-θ boud for the solutio to the recurrece 3T (/3) + if 3 T () = 1 if <3 usig a recursio tree. Assume that is a power of 3. Exercise Solve the recurrece T () = 4T (/2) + if 2 1 if =1 usig a recursio tree. Assume that is a power of 2. Covert your solutio to a big-θ statemet about the behavior of the solutio. Exercise Ca you give a geeral big-θ boud for solutios to recurreces of the form T () =at (/2) + whe is a power of 2? You may have differet aswers for differet values of a. The recurrece i Exercise is similar to the mergesort recurrece. Oe differece is that at each step we divide ito 3 problems of size /3. Thus we get the picture i Figure 5.5. Aother differece is that the umber of levels, istead of beig log 2 +1isow log 3 +1,so the total work is still Θ( log ) uits. Note that log b = Θ(log 2 ) for ay b>1. Now let s look at the recursio tree for Exercise Now, we have 4 childre of size /2, ad we get the Figure 5.6 Let s look carefully at this tree. Just as i the mergesort tree there are log 2 +1levels. However, i this tree, each ode has 4 childre. Thus level 0 has 1 ode, level 1 has 4 odes, level 2 has 16 odes, ad i geeral level i has 4 i odes. O level i each ode correspods to a problem of size /2 i ad hece requires /2 i uits of additioal work. Thus the total work o level i is 4 i (/2 i )=2 i uits. Summig over the levels, we get log 2 i=0 log 2 2 i = i=0 2 i.
8 166 CHAPTER 5. RECURSION AND RECURRENCES Figure 5.5: The recursio tree diagram for the recurrece i Exercise /3 /3 + /3 + /3 = log + 1 levels /9 9(/9) = 1 (1) = There are may ways to simplify that expressio, for example from our formula for the sum of a geometric series we get. log 2 T () = i=0 2 i = 1 2(log 2 ) = = 2 2 = Θ( 2 ). Figure 5.6: The Recursio tree for Exercise /2 /2 + /2 + /2 + /2 = 2 log + 1 levels /4 16(/4) = 4 1 ^2(1) = ^2
9 5.1. GROWTH RATES OF SOLUTIONS TO RECURRENCES 167 More simply, by Theorem 4.4 we have that T () =Θ2 log =Θ( 2 ). Three Differet Behaviors Now let s compare the trees for the recurreces T () =2T (/2) +, T () =T (/2) + ad T () =4T (/2) +. Note that all three trees have depth 1 + log 2,asthis is determied by the size of the subproblems relative to the paret problem, ad i each case, the size of each subproblem is 1/2 the size of of the paret problem. To see the differeces, i the first case, o every level, there is the same amout of work. I the secod case, the amout of work decreases, with the most work beig at level 0. I fact, it decreases geometrically, so by Theorem 4.4 the total work doe is bouded above ad below by a costat times the work doe at the root ode. I the third case, the umber of odes per level is growig at a faster rate tha the problem size is decreasig, ad the level with the largest amout of work is the bottom oe. Agai we have a geometric series, ad so by Theorem 4.4 the total work is bouded above ad below by a costat times the amout of work doe at the last level. If you uderstad these three cases ad the differeces amog them, you ow uderstad the great majority of the recursio trees that arise i algorithms. So to aswer Exercise 5.1-4, which asks for a geeral Big-Θ boud for the solutios to recurreces of the form T () =at (/2) +, weca coclude the followig : 1. if a<2 the T () =Θ(). 2. if a =2the T () =Θ( log ) 3. if a>2 the T () =Θ( log 2 a ) Cases 1 ad 2 follow immediately from our observatios above. We ca verify case 3 as follows. At each level i we have a i odes, each correspodig to a problem of size /2 i.thusat level i the total amout of work is a i (/2 i )=(a/2) i uits. Summig over the log 2 levels, we get (a/2) i. log 2 i=0 The sum is a geometric series, so the sum will be big-θ of the largest term (see Theorem 4.4). Sice a>2, the largest term i this case is clearly the last oe, amely (a/2) log 2, ad applyig rules of expoets ad logarithms, we get that times the largest term is ( ) a log2 = alog log 2 = 2log2 a log2 2 log 2 = log2 a = log 2 a Thus the total work doe is Θ( log 2 a ). Importat Cocepts, Formulas, ad Theorems 1. Divide ad Coquer Algorithm. A divide ad coquer algorithm is oe that solves a problem by dividig it ito problems that are smaller but otherwise of the same type as the origial oe, recursively solves these problems, ad the assembles the solutio of these so-called subproblems ito a solutio of the origial oe. Not all problems ca be solved by such a strategy, but a great may problems of iterest i computer sciece ca.
10 168 CHAPTER 5. RECURSION AND RECURRENCES 2. Mergesort. I mergesort we sort a list of items that have some uderlyig order by dividig the list i half, sortig the first half (by recursively usig mergesort), sortig the secod half (by recursively usig mergesort), ad the mergig the two sorted list. For a list of legth oe mergesort returs the same list. 3. Recursio Tree. A recursio tree diagram for a recurrece of the form T () =at (/b)+g() has three parts. O the left, we keep track of the problem size, i the middle we draw the tree, ad o right we keep track of the work doe. We draw the diagram i levels, each level of the diagram represetig a level of recursio. The tree has a vertex represetig the iitial problem ad oe represetig each subproblem we have to solve. Each o-leaf vertex has a childre. The vertices are divided ito levels correspodig to (sub-)problems of the same size; to the left of a level of vertices we write the size of the problems the vertices correspod to; to the right of the vertices o a give level we write the total amout of work doe at that level by a algorithm whose work is described by the recurrece, ot icludig the work doe by ay recursive calls. 4. The Base Level of a Recursio Tree. The amout of work doe o the lowest level i a recursio tree is the umber of odes times the value give by the iitial coditio; it is ot determied by attemptig to make a computatio of additioal work doe at the lowest level. 5. Bases for Logarithms. We use log as a alterate otatio for log 2.Afudametal fact about logarithms is that log b = Θ(log 2 ) for ay real umber b>1. 6. Three behaviors of solutios. The solutio to a recurrece of the form T () =at (/2) + behaves i oe of the followig ways: (a) if a<2 the T () =Θ(). (b) if a =2the T () =Θ( log ) (c) if a>2 the T () =Θ( log 2 a ) Problems 1. Draw recursio trees ad fid big-θ bouds o the solutios to the followig recurreces. For all of these, assume that T (1) = 1 ad is a power of the appropriate iteger. (a) T () =8T (/2) + (b) T () =8T (/2) + 3 (c) T () =3T (/2) + (d) T () =T (/4) + 1 (e) T () =3T (/3) Draw recursio trees ad fid fid exact solutios to the followig recurreces. For all of these, assume that T (1) = 1 ad is a power of the appropriate iteger. (a) T () =8T (/2) + (b) T () =8T (/2) + 3 (c) T () =3T (/2) +
11 5.1. GROWTH RATES OF SOLUTIONS TO RECURRENCES 169 (d) T () =T (/4) + 1 (e) T () =3T (/3) Fid the exact solutio to recurrece Show that log b = Θ(log 2 ). 5. Recursio trees will still work, eve if the problems do ot break up geometrically, or eve if the work per level is ot c uits. Draw recursio trees ad ad fid the best big-o bouds you ca for solutios to the followig recurreces. For all of these, assume that T (1) =1. (a) T () =T ( 1) + (b) T () =2T ( 1) + (c) T () =T ( )+1(You may assume has the form =2 2i.) (d) T () =2T (/2) + log (You may assume is a power of 2.) 6. I each case i the previous problem, is the big-o boud you foud a big-θ boud? 7. If S() =as( 1) + g() ad g() <c with 0 c<a,how fast does S() grow (i big-θ terms)? S() =a i S( 1) + i 1 j=0 aj g( j) =a S(0) + 1 j=0 aj g( j) < a S(0) + 1 j=0 aj c j = a S(0) + c 1 ( a ) j j=0 c = a S(0)+Θ( ( ) a c )=Θ(a ) 8. If S() =as( 1) + g() ad g() >c with 0 <a c, how fast does S() grow i big-θ terms? 9. give a recurrece of the form T () =at (/b)+g() with T (1) = c>0 ad g() > 0 for all ad a recurrece of the form S() =as(/b) +g() with S(1) = 0 (ad the same g()), is there ay differece i the big-θ behavior of the solutios to the two recurreces? What does this say about the ifluece of the iitial coditio o the big-θ behavior of such recurreces?