6.042/18.062J Mathematics for Computer Science. Recurrences

Transcription

1 6.04/8.06J Mathematics for Computer Sciece Srii Devadas ad Eric Lehma March 7, 00 Lecture Notes Recurreces Recursio breakig a object dow ito smaller objects of the same type is a major theme i mathematics ad computer sciece. For example, i a iductio proof we establish the truth of a statemet P () from the truth of the statemet P ( ). I programmig, a recursive algorithm solves a problem by applyig itself to smaller istaces of the problem. Back o the mathematical side, a recurrece equatio describes the value of a fuctio i terms of its value for smaller argumets. These various icaratios of recursio work together icely. For example oe might prove a recursive algorithm correct usig iductio or aalyze its ruig time usig a recurrece equatio. I this lecture, we ll lear how to solve a family of recurrece equatios, called liear recurreces, that frequetly arise i computer sciece ad other disciplies. The Towers of Haoi I the Towers of Haoi problem, there are three posts ad seve disks of differet sizes. Each disk has a hole through the ceter so that it fits o a post. At the start, all seve disks are o post # as show below. The disks are arraged by size so that the smallest is o top ad the largest is o the bottom. The goal is to ed up with all seve disks i the same order, but o a differet post. This is ot trivial because of two restrictios. First, the oly permitted actio is removig the top disk from a post ad droppig it oto aother post. Secod, a larger disk ca ever lie above a smaller disk o ay post. (These rules imply, for example, that it is o fair to pick up the whole stack of disks at oce ad the to drop them all o aother post!) Post # Post # Post # It is ot immediately clear that a solutio to this problem exists; maybe the rules are so striget that the disks caot all be moved to aother post!

2 Recurreces Oe approach to this problem is to cosider a simpler variat with oly three disks. We ca quickly exhaust the possibilities of this simpler puzzle ad fid a 7 move solutio such as the oe show below. (The disks o each post are idicated by the umbers immediately to the right. Larger umbers correspod to larger disks.) This problem was iveted i 88 by the Frech mathematicia Edouard Lucas. I his origial accout, there were 64 disks of solid gold. At the begiig of time, all 64 were placed o a sigle post, ad moks were assiged the task of movig them to aother post accordig to the rules described above. Accordig to leged, whe the moks complete their task, the Tower will crumble ad the world will ed! The questios we must aswer are, Give sufficiet time, ca the moks succeed? ad if so, How log util the world eds? ad, most importatly, Will this happe before the 6.04 fial?. Fidig a Recurrece The Towers of Haoi problem ca be solved recursively as follows. Let T be the miimum umber of steps eeded to move a disk tower from oe post to aother. For example, a bit of experimetatio shows that T = ad T =. For disks, the solutio give above proves that T 7. We ca geeralize the approach used for disks to the followig recursive algorithm for disks. Step. Apply this strategy recursively to move the top disks from the first post to the third post. This ca be doe i T steps...

3 Recurreces Step. Move the largest disk from the first post to the to the secod post. This requires just step... Step. Recursively move the disks o the third post over to the secod post. Agai, T steps are required.... This algorithm shows that T, the umber of steps required to move disks to a differet post, is at most T +. We ca use this fact to compute upper bouds o the umber of steps required for various umbers of disks: T T + = 7 T 4 T + The algorithm described above aswers our first questio: give sufficiet time, the moks will fiish their task ad ed the world. (Which is a shame. After all that effort they d probably wat to smack a few high fives ad go out for burgers ad ice cream, but ope world s over.). A Lower Boud for Towers of Haoi We ca ot yet compute the exact umber of steps that the moks eed to move the 64 disks; we ca oly show a upper boud. Perhaps havig podered the problem sice the begiig of time the moks have devised a better algorithm. I fact, there is o better algorithm, ad here is why. At some step, the moks must move the th disk from the first post to a differet post. For this to happe, the smaller disks must all be stacked out of the way o the oly remaiig post. Arragig the smaller disks this way requires at least T moves. After the largest disk is moved, at least aother T moves are required to pile the smaller disks o top.

4 4 Recurreces This argumet shows that the umber of steps required is at least T +. Sice we gave a algorithm usig exactly that umber of steps, we ow have a recurrece for T, the umber of moves required to complete the Tower of Haoi problem with disks: T = T = T + (for ) We ca use this recurrece to coclude that T =, T = 7, T 4 =,..... Guess ad Verify Computig T 64 from the recurrece would require a lot of work. It would be ice to have a closed form expressio for T, so that we could quickly compute the umber of steps required to solve the Towers of Haoi problem for ay give umber of disks. (For example, we might wat to kow how much sooer the world would ed if the moks melted dow oe disk to purchase burgers ad ice cream before the ed of the world.) There are several differet methods for solvig recurreces. The simplest method is to guess the solutio ad the to verify that the guess is correct, usually with a iductio proof. This method is called guess ad verify or substitutio. As a basis for a good guess, let s tabulate T for small values of : T Based o this table, a atural guess is that T =. Wheever you guess a solutio to a recurrece, you should always verify it with a proof by iductio or by some other techique; after all, your guess might be wrog. (But why bother to verify i this case? After all, if we re wrog, its ot the ed of the... o, let s check.) Claim. If: T = T = T + (for ) the: T =

5 Recurreces Proof. The proof is by iductio o. Let P () be the propositio that T =. Base case: P () is true because T = =. Iductive step: Now we assume T = to prove that T + = +, where. T + = T + = ( ) + = + The first equality is the recurrece relatio, ad the secod equatio follows by the assumptio P (). The last step is simplificatio. Our guess is ow verified. This shows, for example, that the 7 disk puzzle will require 7 = 7 moves to complete. We ca also ow resolve our remaiig questios about the 64 disk puzzle. Sice T 64 = 64, the moks must complete more tha 8 billio billio steps before the world eds. Better study for the fial. Graduate Studet Job Prospects I a ew academic field (say, computer sciece), there are oly so may faculty positios available i all the uiversities of the world. Iitially, there were ot eough qualified cadidates, ad may positios were ufilled. But, over time, ew graduates are fillig the positios, makig job prospects for later computer sciece studets ever more bleak. Worse, the icreasig umber of professors are able to trai a icreasig umber of graduate studets, causig positios to fill ever more rapidly. Evetually, the uiversities will be saturated; ew computer sciece graduates will have o chace at a academic career. Our problem is to determie whe the uiversities will stop hirig ew computer sciece faculty ad, i particular, to aswer the questio, Are the 6.04 TAs doomed? Here are the details of the problem. There are a total of N faculty positios available worldwide. This umber ever chages due to budgetary costraits. Cogress has passed a law forbiddig forced retiremet i uiversities, ad o oe will retire volutarily. (This is true ad a problem!) I our aalysis, therefore, oce a faculty positio is filled, it ever opes up agai. Each year, every professor trais exactly studet who will go o to become a professor the followig year. The oly exceptio is that first year professors do ot trai studets; they are too busy publishig, teachig, gettig grats, ad servig o committees. I year 0, there are o computer sciece professors i the world. I year, the first professor is hired.

6 6 Recurreces. Fidig a Recurrece Ideally, we could fid a formula for the umber of professors i the world i a give year. The we could determie the year i which all N faculty positios are filled. Let f() be the umber of professors durig year. To develop some ituitio about the problem, we ca compute values of this fuctio for small by had. f(0) = 0 f() = f() = f() = f(4) = f() = f(6) = 8 No CS professors; the dark ages. ew professor; too busy to trai a studet. old professor; ow traiig a studet. ew prof, old prof; ew prof too busy, old prof traiig. ew prof, old profs; ew prof too busy, both old profs traiig ew profs, old profs ew profs, old profs I geeral, the umber of professors i year is equal to the umber of professors last year plus the umber of ew hires. The umber of professors last year is f( ). The umber of ew hires is equal to the umber of professors two years ago, f( ), sice each of these professors traied a studet last year. These observatios give the followig recurrece equatio for the umber of professors: f(0) = 0 f() = f() = f( ) + f( ) ( ) This is the familiar Fiboacci recurrece. Lookig back, the values of f() that we computed by had are ideed the first few Fiboacci umbers. Fiboacci umbers arise i all sorts of applicatios. Fiboacci himself itroduced the umbers i 0 to study rabbit reproductio. Fiboacci umbers also appear, oddly eough, i the spiral patters o the faces of suflowers. Ad the iput umbers that make Euclid s GCD algorithm require the greatest umber of steps are cosecutive Fiboacci umbers. So how big is f() ayway? Of course, we could compute as may Fiboacci umbers as we like usig the recurrece, but it would be much icer to fid a closed form.. Solvig the Recurrece Solvig the Fiboacci recurrece is easy because the recurrece is liear. (Well, easy i the sese that you ca lear the techique i oe lecture; discoverig it actually took six

7 Recurreces 7 ceturies.) A liear recurrece has the form: f() = a f( ) + a f( ) a d f( d) d = b i f( i) i= where a, a,... a d are costats. The order of the recurrece is d. For example, the Fiboacci recurrece is order ad has coefficiets a = a =. (Later o, we ll slightly expad the defiitio of a liear recurrece.) For ow, let s try to solve just the Fiboacci recurrece; we ll see how to solve geeral liear recurreces later i the lecture. Our rule of thumb is that guess ad verify is the first method to apply to a ufamiliar recurrece. It turs out that for a liear recurrece, a expoetial solutio is a good guess. However, sice we kow othig beyod this, our iitial guess ad erify attempt will really oly be a dry ru ; that is, we will ot make a exact guess ad will ot verify it with a complete proof. Rather, the goal of this first attempt is oly to clarify the form of the solutio. Guess. f() = cx Here c ad x are parameters itroduced to improve our odds of havig a correct guess; i the verificatio step, we ca pick values that make the proof work. To further improve our odds, let s eglect the boudary coditios, f(0) = 0 ad f() =. Verificatio. Pluggig our guess ito the recurrece f() = f( ) + f( ) gives: cx = cx x = x + x x = 0 ± x = + cx I the first step, we divide both sides of the equatio by cx. The we rearrage terms ad fid x with the quadratic formula. This calculatio suggests that the costat c ca be aythig, but that x must be ( ± )/. Evidetly, there are two solutios to the recurrece: ( ( ) + ) f() = c or f() = c I fact, ay liear combiatio of these two solutios is also a solutio. The followig theorem states that this is true i geeral for liear recurreces. Theorem. If f() ad g() are solutios to a liear recurrece (without boudary coditios), the cf () + dg() is also a solutio.

8 8 Recurreces Proof. Sice f() ad g() are both solutios, the: d f() = a i f( i) i= d g() = a i g( i) i= Multiplyig the first equatio by c, the secod by d, ad summig gives: ( ) ( ) d d cf () + dg() = c a i f( i) + d a i g( i) i= i= d ( ) = a i cf ( i) + dg( i) i= Thus, cf () + dg() is a solutio as well. This same pheomeo that a liear combiatio of solutios is aother solutio also arises i differetial equatios ad, cosequetly, may physical systems. I the preset case, the theorem implies that ( ( ) + ) f() = c + c is a solutio to the Fiboacci recurrece without boudary coditios for all costats c ad c. All that remais is to choose these costats to obtai a solutio cosistet with the boudary coditios, f(0) = 0 ad f() =. From the first coditio, we kow: ( ( ) 0 + ) 0 f(0) = c + c = c + c = 0 From the secod boudary coditio, we have: ( ( ) + ) f() = c + c = We ow have two liear equatios i two ukows. The system of equatios is ot degeerate, so there is a uique solutio: c = / ad c = /. We re doe! We have a complete solutio to the Fiboacci recurrece with boudary coditios: ( ) ( ) + f() =

9 Recurreces 9 This looks completely wrog! All Fiboacci umbers are itegers, but this expressio is full of square roots of five! Amazigly, however, the square roots always cacel out. This expressio really does give the Fiboacci umbers if we plug i = 0,,,.... It is easy to see why o oe stumbled across this solutio for six ceturies!. Job Prospects Let s retur to the origial questio: how log util all N faculty positios are take? To aswer this questio, we must fid the smallest such that f() N; that is, we must determie the year i which there are as may potetial professors as uiversity positios. Graduates after year will have to fid other employmet, e.g. shufflig golde disks i a obscure moastic commuity for the ext 0 9 years. Because f() has such a complicated form, it is hard to compute the right value of exactly. However, we ca fid a excellet approximate aswer. Note that i the closed form for Fiboacci umbers, the secod term rapidly goes to zero: ( ) ( ) + f() = ( ) + = + o() This is because ( )/ = <, ad a big power of a umber less tha is tiy. From this approximatio for f(), we ca estimate the year i which all faculty positios will be filled. That happes whe: ( ) + f() N Thus, all jobs are filled i about years where: log( N + o()) = log( + ) = Θ(logN) This makes sese, sice the umber of professors is icreasig expoetially. For example, N = 0, 000 jobs would all be take i about = 0.8 years. Your TAs do t have a momet to lose! The solutio to the Fiboacci recurrece has a iterestig corollary. The umber: ( + ) φ =

10 0 Recurreces is ofte called the Golde Ratio. We ca write the domiat term i the closed form for the th Fiboacci umber i terms of the φ: f() = φ + o() We ve just show that this expressio ivolvig irratioal umbers is actually very close to a iteger for all large amely, the th Fiboacci umber. This is just oe of may curious properties of the Golde Ratio. Geeral Liear Recurreces The method we used to solve the Fiboacci recurrece ca actually be used to solve ay liear recurrece. Recall that a recurrece is liear if it has the form: f() = a f( ) + a f( ) a d f( d) Substitutig the guess f() = x, as with the Fiboacci recurrece, gives: x = a x + a x a d x d d x = a x d + a x d a d x + a d Dividig the first equatio by x d gives the secod. This secod equatio is called the characteristic equatio of the recurrece. The characteristic equatio ca be read off very quickly sice the coefficiets of the equatio are the same as the coefficiets of the recurrece. The solutios to a liear recurrece are defied by the roots of the characteristic equatio. Neglectig boudary coditios for the momet: If r is a orepeated root of the characteristic equatio, the r is a solutio to the recurrece. If r is a repeated root with multiplicity k, the are all solutios to the recurrece. k r, r, r,..., r Futhermore, Theorem implies that every liear combiatio of these solutios is also a solutio. For example, suppose that the characteristic equatio of a recurrece has roots r, r, ad r twice. These four roots imply four distict solutios: f() = r f() = r f() = r f() = r

11 Recurreces Thus, every liear combiatio is also a solutio. f() = a r + b r + c r + d r All that remais is to fid a solutio cosistet with the boudary coditios by choosig the costats appropriately. Each boudary coditio implies a liear equatio ivolvig these costats. So we ca determie the costats by solvig a system of liear equatios. For example, suppoose our boudary coditios were f(0) = 0, f() =, f() = 4 ad f() = 9. The we would obtai four equatios i four ukows: j 0 f(0) = 0 a r + b r + c r + d 0r = 0 f() = a r + b r + c r + d r = f() = 4 a r + b r + c r + d r = 4 f() = 9 a r + b r + c r + d r = 9 All the asty r i thigs are actually just costats. Solvig this system gives values for a, b, c, ad d that defie a solutio to the recurrece cosistet with the boudary coditios A Example Suppose that there is a type of plat that lives forever, but oly reproduces durig its first year of life. How fast will the plat populatio grow? Notice that this is just the reverse of the graduate studet job problem where faculty reproduce i every year except the first. Let f() be the umber of plats i year. As boudary coditios, defie f(0) = 0 ad f() =. Now the plat populatio i year is equal to the populatio from the year before plus the umber of ew plats. The populatio from the year before is f( ). Ad the umber of ew plats this year is equal to the umber of ew plats last year, which is f( ) f( ). Puttig these observatios together, we ca form a recurrece equatio for the plat populatio: f() = f( ) + (f( ) f( )) = f( ) f( ) The characteristic equatio is x x + = 0, which has the sigle root x = with multiplicity. Therefore, the solutio to the recurrece has the form: f() = c () + c () = c + c The boudary coditios imply two liear equatios i two ukows: f(0) = 0 c + c (0) = 0 f() = c + c () =

12 Recurreces The solutio to the liear system is c = 0, c =. Therefore, the solutio to the recurrece is: f() = c + c = 0 + () = The aswer turs out to be very simple! I year, there are exactly plats. Of course, we probably could have solved this problem more easily with guess ad verify. But, as the Fiboacci recurrece demostrated, guessig is ot always so easy. 4 Ihomogeeous Recurreces We ca ow solve all recurreces of the form: f() = a f( ) + a f( ) a d f( d) Strictly speakig, this is the family of homogeeous liear recurreces. Addig a extra, arbitrary fuctio g() o the right side gives the geeral form of a ihomogeeous liear recurrece: f() = a f( ) + a f( ) a d f( d) + g() For example, addig + to the Fiboacci recurrece gives a ihomogeeous liear recurrece: f() = f( ) + f( ) + Solvig ihomogeous liear recurreces is either very differet or very difficult. We ca divide the whole job ito three steps.. Replace g() by 0 ad solve the resultig homogeeous recurrece as before. (Igore boudary coditios for ow; that is, do ot solve for costats c, c,..., c d yet.) The solutio to the homogeeous recurrece is called the homogeeous solutio.. Now restore g() ad fid a sigle solutio to the recurrece, agai igorig boudary coditios. This is called the particular solutio. There are geeral methods for fidig particular solutios, but we advise you to use guess ad verify. I a momet, we ll explai how to guess wisely.. Add the homogeeous ad particular solutios together to obtai the geeral solutio. Now use the boudary coditios to determie costats by the usual method of geeratig ad solvig a system of liear equatios. If you ve studied differetial equatios, the all this probably souds quite familiar. If you have t, the whe you do get aroud to studyig differetial equatios they should seem quite familiar.

13 Recurreces 4. A Example Let s demostrate the method for solvig a ihomogeous liear recurrece o this example: f() = f() = 4f( ) + Step : Solve the Homogeeous Recurrece The homogeeous recurrece is f() = 4f( ). The characteristic equatio is x 4 = 0. The oly root is x = 4. Therefore, the homogeeous solutio is f() = c4. Step : Fid a Particular Solutio Now we must fid a sigle solutio to the full recurrece f() = 4f( ) +. Let s guess that there is a solutio of the form d, where d is a costat. Substitutig this guess ito the recurrece gives: d = 4d + d = 4d + d = Evidetly, f() = = + is a particular solutio. Step : Add Solutios ad Fid Costats We ow add the homogeeous solutio ad the particular solutio to obtai the geeral solutio: f() = c4 + The boudary coditio gives the value of the costat c: f() = c4 + = c = Therefore, the solutio to the recurrece is f() = 4 +. Piece of cake! Sice we could easily have made a mistake, let s check to make sure that our solutio at least works for =. From the recurrece, f() = 4f() + =. From our closed form, f() = 4 = 40 7 =. It looks right!

14 4 Recurreces 4. How to Guess a Particular Solutio The hardest part of solvig ihomogeeous recurreces is fidig a particular solutio. This ivolves guessig, ad you might guess wrog. However, some rules of thumb make this job fairly easy most of the time. Geerally, look for a particular solutio with the same form as the ihomogeeous term g(). If g() is a costat, the guess a particular solutio f() = c. If this does t work, try f() = b + c, the f() = a + b + c, etc. More geerally, if g() is a polyomial, try a polyomial of the same degree, the a polyomial of degree oe higher, the two higher, etc. For example, if g() = 6+, the try f() = b + c ad the f() = a + b + c. If g() is a expoetial, such as, the first guess that f() = c. Failig that, try f() = b + c ad the a + b + c, etc.

15 Recurreces Short Guide to Solvig Liear Recurreces A liear recurrece is a equatio f() = a f( ) + a f( ) a d f( d) +g() }{{}}{{} homogeeous part ihomogeeous part together with boudary coditios such as f(0) = b 0, f() = b, etc.. Fid the roots of the characteristic equatio: x = a x + a x a k. Write dow the homogeeous solutio. Each root geerates oe term ad the homoge r r, eous solutio is the sum of these terms. A orepeated root r geerates the term c where c r is a costat to be determied later. A root r with multiplicity k geerates the terms: c r r, c r r, c r r,..., rk c k r where c r,..., c r k are costats to be determied later.. Fid a particular solutio. This is a solutio to the full recurrece that eed ot be cosistet with the boudary coditios. Use guess ad verify. If g() is a polyomial, try a polyomial of the same degree, the a polyomial of degree oe higher, the two higher, etc. For example, if g() =, the try f() = b+c ad the f() = a +b+c. If g() is a expoetial, such as, the first guess that f() = c. Failig that, try f() = b + c ad the a + b + c, etc. 4. Form the geeral solutio, which is the sum of the homogeeous solutio ad the particular solutio. Here is a typical geeral solutio: f() = c + d( ) + + }{{}}{{} homogeeous solutio particular solutio. Substitute the boudary coditios ito the geeral solutio. Each boudary coditio gives a liear equatio i the ukow costats. For example, substitutig f() = ito the geeral solutio above gives: = c + d ( ) + + = c d Determie the values of these costats by solvig the resultig system of liear equatios.