We have this messy summatio to solve though First observe that the value remais costat throughout the sum, ad so we ca pull it out frot Also ote that we ca write 3 i / i ad (3/) i T () = log 3 (log ) 1 ( ) i 3 Note that this is a geometric series We may apply the formula for the geometric series, which gave i a earlier lecture For x 1: m x i = xm1 1 x 1 I this case x =3/ad m = log 1 We get T () = log 3 (3/)log 1 Applyig our favorite log idetity oce more to the expressio i the umerator (with a =3/ad b =)weget (3/) log = log (3/) = (log 3 log ) = (log 3 1) = log 3 If we plug this back i, we have log 3 T () = log 3 1 = log 3 log 3 1/ = log 3 ( log 3 ) = log 3 ( log 3 ) = 3 log 3 So the fial result (at last!) is T () = 3 log 3 3 079 Θ() It is iterestig to ote the uusual expoet of log 3 079 We have see that two ested loops typically leads to Θ( 2 ) time, ad three ested loops typically leads to Θ( 3 ) time, so it seems remarkable that we could geerate a strage expoet like 079 as part of a ruig time However, as we shall see, this is ofte the case i divide-ad-coquer recurreces Lecture 8: More o Recurreces (Thursday, Feb 19, 1998) Read: Chapt o recurreces, skip Sectio Recap: Last time we discussed recurreces, that is, fuctios that are defied recursively We discussed their importace i aalyzig divide-ad-coquer algorithms We also discussed two methods for solvig recurreces, amely guess-ad-verify (by iductio), ad iteratio These are both very powerful methods, but they are quite mechaical, ad it is difficult to get a quick ad ituitive sese of what is goig o i the recurrece Today we will discuss two more techiques for solvig recurreces The first provides a way of visualizig recurreces ad the secod, called the Master Theorem, is a method of solvig may recurreces that arise i divide-ad-coquer applicatios 27
Visualizig Recurreces Usig the Recursio Tree: Iteratio is a very powerful techique for solvig recurreces But, it is easy to get lost i all the symbolic maipulatios ad lose sight of what is goig o Here is a ice way to visualize what is goig o i iteratio We ca describe ay recurrece i terms of a tree, where each expasio of the recurrece takes us oe level deeper i the tree Recall that the recurrece for MergeSort (which we simplified by assumig that is a power of 2, ad hece could drop the floors ad ceiligs) { 1 if =1, T () = 2 otherwise Suppose that we draw the recursio tree for MergeSort, but each time we merge two lists, we label that ode of the tree with the time it takes to perform the associated (orecursive) merge Recall that to merge two lists of size m/2 to a list of size m takes Θ(m) time, which we will just write as m Below is a illustratio of the resultig recursio tree T() /2 T(/) / / 1 1 1 / /2 / 1 = 2( /2) = ( /) = (/) = lg 1 levels (lg 1) Figure 5: Usig the recursio tree to visualize a recurrece Observe that the total work at the topmost level of the recursio is Θ() (or just for short) At the secod level we have two merges, each takig /2 time, for a total of 2(/2) = At the third level we have merges, each takig / time, for a total of (/) = This cotiues util the bottommost level of the tree Sice the tree exactly lg 1levels (0, 1, 2,,lg ), ad each level cotributes a total of time, the total ruig time is (lg 1)=lg This is exactly what we got by the iteratio method This ca be used for a umber of simple recurreces For example, let s try it o the followig recurrece The tree is illustrated below { 1 if =1, T () = 3 2 otherwise Agai, we label each ode with the amout of work at that level I this case the work for T (m) is m 2 For the top level (or 0th level) the work is 2 At level 1 we have three odes whose work is (/2) 2 each, for a total of 3(/2) 2 This ca be writte as 2 (3/) At the level 2 the work is 9(/) 2, which ca be writte as 2 (9/16) I geeral it is easy to extrapolate to see that at the level i, wehave3 i odes, each ivolvig (/2 i ) 2 work, for a total of 3 i (/2 i ) 2 = 2 (3/) i This leads to the followig summatio Note that we have ot determied where the tree bottoms out, so we have left off the upper boud o the sum? ( ) i 3 T () = 2 28
T() 2 (/2) 2 (/2) 2 (/2) 2 3(/2) 2= 2(3/) T(/) (/) 2 (/) 2 (/) 2 (/) 2 (/) 2 (/) 2 9(/) 2= 2 (9/16) 2(3/) i Figure 6: Aother recursio tree example = 2 If all we wated was a asymptotic expressio, the are essetially doe at this poit Why? The summatio is a geometric series, ad the base (3/) is less tha 1 This meas that this series coverges to some ozero costat (eve if we ra the sum out to ) Thus the ruig time is Θ( 2 ) To get a more exact result, observe that the recursio bottoms out whe we get dow to sigle items, ad sice the sizes of the iputs are cut by half at each level, it is ot hard to see that the fial level is level lg (It is easy to be off by ±1 here, but this sort of small error will ot affect the asymptotic result I this case we happe to be right) So, we ca plug i lg for the? i the above summatio lg ( ) i 3 T () = 2 If we wated to get a more exact aswer, we could plug the summatio ito the formula for the geometric series ad simplify This would lead to a expressio like T () = 2(3/)lg 1 1 This will take some work to simplify, but at this poit it is all just tedious algebra to get the formula ito simple form (This sort of algebraic is typical of algorithm aalysis, so be sure that you follow each step) T () = 2(3/)lg 1 1 = 2 ((3/) lg 1 1) = 2 (1 (3/) lg 1 ) = 2 (1 (3/)(3/) lg ) = 2 (1 (3/) lg(3/) ) = 2 (1 (3/) lg 3 lg ) = 2 (1 (3/) lg 3 2 ) = 2 (1 (3/)( lg 3 / 2 )) = 2 3 lg 3 Note that lg 3 158, so the whole expressio is Θ( 2 ) I coclusio, the techique of drawig the recursio tree is a somewhat more visual way of aalyzig summatios, but it is really equivalet to the method of iteratio (Simplified) Master Theorem: If you aalyze may divide-ad-coquer algorithms, you will see that the same geeral type of recurrece keeps poppig up I geeral you are breakig a problem ito a subproblems, where each subproblem is roughly a factor of 1/b of the origial problem size, ad 29
the time it takes to do the splittig ad combiig o a iput of size is Θ( k ) For example, i MergeSort, a =2,b=2, ad k =1 Rather tha doig every such recurrece from scratch, ca we just come up with a geeral solutio? The aswer is that you ca if all you eed is a asymptotic expressio This result is called the Master Theorem, because it ca be used to master so may differet types of recurrece Our text gives a fairly complicated versio of this theorem We will give a simpler versio, which is geeral eough for most typical situatios I cases where this does t apply, try the oe from the book If the oe from the book does t apply, the you will probably eed iteratio, or some other techique Theorem: (Simplified Master Theorem) Let a 1, b>1be costats ad let T () be the recurrece T () =at (/b) k, defied for 0 (As usual let us assume that is a power of b The basis case, T (1) ca be ay costat value) The Case 1: if a>b k the T () Θ( log b a ) Case 2: if a = b k the T () Θ( k log ) Case 3: if a<b k the T () Θ( k ) Usig this versio of the Master Theorem we ca see that i the MergeSort recurrece a =2,b=2, ad k =1 Thus, a = b k (2=2 1 ) ad so Case 2 applies From this we have T () Θ( log ) I the recurrece above, T () =3 2,wehavea=3,b=2ad k =2Wehavea<b k (3<2 2 ) i this case, ad so Case 3 applies From this we have T () Θ( 2 ) Fially, cosider the recurrece T () =T(/3), i which we have a =,b=3ad k =1I this case we have a>b k (>3 1 ), ad so Case 1 applies From this we have T () Θ( log 3 ) Θ( 126 ) This may seem to be a rather strage ruig time (a o-iteger expoet), but this ot ucommo for may divide-ad-coquer solutios There are may recurreces that caot be put ito this form For example, if the splittig ad combiig steps ivolve sortig, we might have see a recurrece of the form { 1 if =1, T () = 2 log otherwise This solves to T () =Θ(log 2 ), but the Master Theorem (either this form or the oe i CLR) will tell you this However, iteratio works just fie here Recursio Trees Revisited: The recursio trees offer some ituitio about why it is that there are three cases i the Master Theorem Geerally speakig the questio is where is most of the work doe: at the top of the tree (the root level), at the bottom of the tree (the leaf level), or spread equally throughout the etire tree For example, i the MergeSort recurrece (which correspods to Case 2 i the Master Theorem) every level of the recursio tree provides the same total work, amely For this reaso the total work is equal to this value times the height of the tree, amely Θ(log ), for a total of Θ( log ) Next cosider the earlier recurrece T () =3 2 (which correspods to Case 3 i the Master Theorem) I this istace most of the work was cocetrated at the root of the tree Each level of the tree provided a smaller fractio of work By the ature of the geometric series, it did ot matter how may levels the tree had at all Eve with a ifiite umber of levels, the geometric series that result will coverge to a costat value This is a importat observatio to make A commo way to desig the most efficiet divide-ad-coquer algorithms is to try to arrage the recursio so that most of the work is doe at the root, ad at each successive level of the tree the work at this level reduces (by some costat factor) As log as this is the case, Case 3 will apply 30
Fially, i the recurrece T () =T(/3) (which correspods to Case 1), most of the work is doe at the leaf level of the recursio tree This ca be see if you perform iteratio o this recurrece, the resultig summatio is log 3 ( ) i 3 (You might try this to see if you get the same result) Sice /3 > 1, as we go deeper ito the levels of the tree, that is deeper ito the summatio, the terms are growig successively larger The largest cotributio will be from the leaf level Lecture 9: Medias ad Selectio (Tuesday, Feb 2, 1998) Read: Todays material is covered i Sectios 102 ad 103 You are ot resposible for the radomized aalysis of Sectio 102 Our presetatio of the partitioig algorithm ad aalysis are somewhat differet from the oes i the book Selectio: I the last couple of lectures we have discussed recurreces ad the divide-ad-coquer method of solvig problems Today we will give a rather surprisig (ad very tricky) algorithm which shows the power of these techiques The problem that we will cosider is very easy to state, but surprisigly difficult to solve optimally Suppose that you are give a set of umbers Defie the rak of a elemet to be oe plus the umber of elemets that are smaller tha this elemet Sice duplicate elemets make our life more complex (by creatig multiple elemets of the same rak), we will make the simplifyig assumptio that all the elemets are distict for ow It will be easy to get aroud this assumptio later Thus, the rak of a elemet is its fial positio if the set is sorted The miimum is of rak 1 ad the maximum is of rak Of particular iterest i statistics is the media Ifis odd the the media is defied to be the elemet of rak ( 1)/2 Whe is eve there are two atural choices, amely the elemets of raks /2 ad (/2) 1 I statistics it is commo to retur the average of these two elemets We will defie the media to be either of these elemets Medias are useful as measures of the cetral tedecy of a set, especially whe the distributio of values is highly skewed For example, the media icome i a commuity is likely to be more meaigful measure of the cetral tedecy tha the average is, sice if Bill Gates lives i your commuity the his gigatic icome may sigificatly bias the average, whereas it caot have a sigificat ifluece o the media They are also useful, sice i divide-ad-coquer applicatios, it is ofte desirable to partitio a set about its media value, ito two sets of roughly equal size Today we will focus o the followig geeralizatio, called the selectio problem Selectio: Give a set A of distict umbers ad a iteger k, 1 k, output the elemet of A of rak k The selectio problem ca easily be solved i Θ( log ) time, simply by sortig the umbers of A, ad the returig A[k] The questio is whether it is possible to do better I particular, is it possible to solve this problem i Θ() time? We will see that the aswer is yes, ad the solutio is far from obvious The Sieve Techique: The reaso for itroducig this algorithm is that it illustrates a very importat special case of divide-ad-coquer, which I call the sieve techique We thik of divide-ad-coquer as breakig the problem ito a small umber of smaller subproblems, which are the solved recursively The sieve techique is a special case, where the umber of subproblems is just 1 31