Introduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Probabilistic Analysis, randomized quicksort Date: 9/4/14

Transcription

1 Itroductio to Algorithms / Algorithms I Lecturer: Michael Diitz Topic: Probabilistic Aalysis, radomized quicksort Date: 9/4/ Itroductio As we discussed the first lecture, oe issue with tryig to do a average-case aalysis of a algorithm is figurig out what the average case actually is. Differet applicatios might result i vastly differet empirical iput distributios, ad whe we desig basic, low-level algorithms ad data structures we wat our guaratees to hold i every possible applicatio. O the other had, as we ll see today, sometimes it turs out that addig radomess ca help, eve i the worst-case! This is the paradigm of radomized algorithms: give a (worst-case) iput, we use radomess i our algorithm ad provide guaratees either i expectatio or with highprobability. Note how differet this is from tryig to do a average-case aalysis: istead of usig radomess to model the iput, we still allow arbitrary iputs but use radomess iterally i our algorithm. The first approach, of modelig the iput usig a distributio, is commoly see i machie learig, computatioal biology, ad other applicatios. There is much iterestig work there, but if we model the iput as a distributio we are fudametally tied to that distributio, ad lose geerality. Addig radomess ca be couterituitive to some people, but let s us desig algorithms that do ot lose geerality but istead have guaratees that hold (for ay iput) i expectatio or with high probability. As a side ote, there is a class beig taught right ow ad (I thik) durig most fall semesters by Vova Braverma o radomized algorithms. I highly recommed this class. This lecture (ad possibly others) is my attempt to add some radomized algorithms to this class, sice i the past years radomized algorithms have moved from a specialized topic to the maistream. 3. Basics of Quicksort So today we re goig to discuss a particularly ice use of radomizatio: radomized quicksort. First, recall the Quicksort algorithm. For ow, let s assume that we are give a array of legth i which all elemets are distict it s easy to see that allowig equal elemets oly helps us. Give a array of legth, quicksort does the followig: 1. If is 0 or 1, retur.. Pick some elemet p as the pivot. 3. Compare each elemet of the array to p, makig a array cosistig of L (the elemets less tha p), followed by p, ad the followed by G (the elemets greater tha p). I other words, p is ow at its fial locatio ad all elemets are o the correct side of p. 4. Recursively sort L ad G. 1

2 Note that this algorithm is ot totally well-specified, sice we have ot defied how to pick the pivot p. The traditioal thig to do is come up with some arbitrary but determiistic choice like, for example, always choosig the first (ot ecessarily the smallest) elemet as the pivot. Next lecture we will see ext lecture a better way of determiistically choosig a pivot, but the origial (ad probably most commo) versio of quicksort just picks the first elemet. What is the ruig time of this basic versio of quicksort? Everyoe should kow that the worstcase ruig time is Θ( ), but let s prove it agai. For a upper boud, ote that by defiitio if we pick a pivot p i step the after step 3 p is i the right place, ad thus step ca be executed at most times. Each time we execute step we also have to execute step 3, but this takes at most O() time. Thus the total ruig time is O( ). For a lower boud, suppose that the array is already sorted. The whe we choose p to be the first elemet, all other elemets are compared to it ad put i G. The pivot the stays where it is, ad we recurse o all remaiig elemets. So the ruig time i this case is T () = T ( 1) + c, which we ow kow is Ω( ). Thus the ruig time of basic quicksort is Θ( ). Radomized quicksort is the simple variatio that picks a pivot p uiformly at radom from the elemets. We will show that i expectatio the ruig time of radomized quicksort is O( log ). Moreover, sice the radomess is from the algorithm itself rather tha over the distributio of the iputs, this boud holds for every possible iput. So o ay fixed iput the ruig time is a radom variable, but we will show that this radom variable always has expectatio at most O( log ) (ad clearly has maximum at most O( )). This is better tha a average-case boud because we do t have to assume aythig about the iput it could be that i some applicatio the arrays ted to be already sorted or i some other order which makes determiistic quicksort slow, but sice our boud is worst-case (although i expectatio over the radomess used by the algorithm) it still holds. Before we prove this, though, we will eed to cover some basic probability. 3.3 Probability Basics I m ot goig to get too formal, but I highly, highly recommed learig basic probability theory either takig a class or teachig yourself. It s good for you, ad also super useful. What follows is semi-formal, ad oly what we really eed for this kid of basic aalysis. You should look at CLRS Chapter 5 for some more details. The example we will use is rollig a pair of dice ad lookig at the sum. The the sample space is the set of possible outcomes, i.e. the 36 possible pairs of rolls. A evet is a subset of the sample space. So you could talk about the evet that the first die is 3 or the evet that the dice add up to 7 or 11, etc. A radom variable is a fuctio from the sample space to umbers (real or itegers). For example we could have a radom variable X 1 equal to the first die, ad a radom variable X equal to the roll of the secod die. The we could also defie a radom variable X = X 1 + X which is the sum of the dice.

3 Radom variables are icredibly importat, ad are i some sese the right way to thik about much of probability theory. For us, ote that the ruig time of radomized quicksort is a radom variable: the sample space is the order of the pivots we pick, ad the ruig time is some fuctio of that (with values betwee 1 ad Θ( )). Oe of the most importat properties of a radom variable is its expectatio this is what we will aalyze for the ruig time of radomized quicksort, for example. Defiitio Let X be a radom variable over sample space Ω. The the expected value of x is E[X] = e Ω Pr[e]X(e). (3.3.1) So the expectatio of a radom variable is its average value across the sample space, but weighted accordig to the probability of each poit i the sample space. It s commo to rearrage this by groupig together terms o which the radom variable has the same value, givig us E[X] = y y Pr[X = y] (3.3.) The coditioal expectatio of X is exactly what it souds like: the expected value of X coditioed o evet A. E[X A] = 1 Pr[e]X(e) Pr[A] Oe amazig feature of expectatios, which we will use over ad over agai, is liearity of expectatios: for ay two radom variables X ad Y, E[X + Y ] = E[X] + E[Y ]. The amazig thig about this is that it is true for all radom variables o matter how weirdly they are correlated. So we ca may times break up very complicated radom variables ito sums of simpler radom variables, ad the it suffices to just determie the expectatio of each of the simpler radom variables. Cosider our dice example. What is the expected sum of the dice? Call this radom variable X. We could aalyze it usig equatio (3.3.1), but that would require a 36-term sum. Or we could aalyze it usig equatio (3.3.), but that would require figurig out the probability that the sum equaled, the probability i equaled 3, etc. Neither of these are impossible (or eve particularly difficult), but they require some combiatorics. O the other had, we ca defie two radom other radom variables: let X 1 be the outcome of the first dies, ad let X be the outcome of the secod. The X = X 1 + X, so by liearity of expectatios E[X] = E[X 1 + X ] = E[X 1 ] + E[X ]. But ow by defiitio we have that ad thus E[X] = 7. E[X 1 ] = E[X ] = e A 6 y=1 1 6 y = 1 6 = 3.5, Theorem 3.3. (Liearity of Expectatios) For ay two radom variables X ad Y, ad ay costats α ad β, E[αX + βy ] = αe[x] + βe[y ]. 3

4 Proof: We will simply use equatio (3.3.1): as claimed. E[αX + βy ] = Pr[x](αX(e) + βy (e)) = α e Ω e Ω = αe[x] + βe[y ], 3.4 Radomized Quicksort Pr[e]X(e) + β e Ω Pr[e]Y (e) There are a few ways to aalyze Radomized Quicksort I thik that this aalysis is really slick, ad demostrated the power of liearity of expectatios. Recall that we re assumig all elemets are distict, ad ote that the ruig time is essetially the total umber of comparisos. Theorem The expected umber of comparisos i Radomized Quicksort is at most O( log ). Proof: Let X be the total umber of comparisos made (so X is a radom variable). Let e i be the ith smallest value i the give array, for i {1,..., }. We defie aother radom variable X ij for each i, j {1,..., } with i < j. We set X ij = 1 if the algorithm at some poit compares e i ad e j, ad otherwise we set X ij = 0. It is easy to see that Radomized Quicksort will ever compare the same two elemets more tha oce, sice if we compare e i ad e j the we must have picked either e i or e j as the pivot, so after that iteratio whichever oe is the pivot will ot be compared to ay other elemets. This meas that X = 1 j=i+1 X ij, so by liearity of expectatios we kow that E[X] = 1 j=i+1 E[X ij ]. (3.4.3) So ow we just eed to uderstad E[X ij ]. Let s start by cosiderig two extremes. First, suppose that j = i + 1. The X ij = 1, sice we will ievitably compare e i ad e i+1 (they will stay i the same subarray throughout the recursio util evetually oe of them is chose as the pivot, whe they will be compared). O the other had, suppose that i = 1 ad j =. The whe we choose the first pivot, if we choose either e 1 or e the e 1 L ad e G, so the recursive calls separate them ad we will ever compare them. So the probability that we compare e 1 ad e is /, ad thus E[X 1 ] = /. Now cosider geeral i ad j. Let e p be the pivot that we pick. If p = i or p = j the we will compare e i ad e j. If i < p < j the e i L while e j G, so we will ot compare them. The more subtle cases are if p < i or p > j. Suppose p < i. The e i ad e j are both i G, so are i the same subarray i the recursive call. So they may be compared, but they may ot. Similarly, is p > j the e i ad d j are both i L, so may be compared later o but may ot. But let s look more closely. What if we coditio o i p j? The the probability that X ij = 1 is exactly j i+1. This is idepedet of, ad sice if we pick p < i or p > j we simply retry (albeit 4

5 with a smaller value of, this meas that evetually i p j ad so X ij = 1 with probability j i+1. Thus E[X ij] = j i+1. Puttig this together with equatio (3.4.3), we get that E[X] = 1 j=i+1 1 j i + 1 = ( ) H = O( log ), i + 1 where H = is the th harmoic umber, which is O(log ) (you ca prove this by cosiderig the itegral of 1/x). A possibly easier way to see the boud o E[X ij ] is through a dart game 1. Whe we pick a pivot p, we are essetially throwig a dart uiformly at radom at the (sorted) array. If p < i the sice we recurse o G ad e i ad e j are both i G, we simply throw aother dart but ow uiformly distributed over G. Similarly, if p > j we throw aother dart but uiformly distributed over L. We keep doig this util evetually a dart lads i {i, i + 1,..., j}, whe we stop. If this fial dart hits i or j the X ij = 1, otherwise X ij = Aother aalysis There s aother aalysis of radomized quicksort that is much less slick, but is a little more direct. It goes via a recurrece relatio. Let T () deote the expected umber of comparisos o a iput of size. Whe we pick a pivot e p, we do 1 comparisos ad the call quicksort recursively o pieces of size p 1 ad p. Sice each p is equally likely, by liearity of expectatios we get that T () = ( 1) + 1 (T (i 1) + T ( i)) (3.4.4) Clearly T (0) = 0 (sice there are o comparisos made for a array of size 1), so we ca rewrite equatio (3.4.4) as T () = ( 1) + 1 T (i) To solve this recurrece, we will use the guess ad check method. Let s guess that T () c l for some costat c. This is clearly true i the base case of T (1). Now to verify it iductively, we do the followig: T () = ( 1) + 1 T (i) 1 ( 1) + ci l i ( 1) + c 1 (x l x)dx 1 I first saw this aalysis from Avrim Blum i the udergraduate algorithms class at CMU. 5

6 c l, = ( 1) + c ( 1 = ( 1) + c ) x l x 1 4 x x=1 ( 1 l c l c + c ) as log as c. 6