6.5 Probability Calculations in Hashing

Transcription

1 6.5. PROBABILITY CALCULATIONS IN HASHING Probability Calculatios i Hashig We ca use our kowledge of probability ad expected values to aalyze a umber of iterestig aspects of hashig icludig:. expected umber of items per locatio, 2. expected time for a search, 3. expected umber of collisios, 4. expected umber of empty locatios, 5. expected time util all locatios have at least oe item, 6. expected maximum umber of items per locatio. Expected Number of Items per Locatio Exercise 6.5- We are goig to compute the expected umber of items that hash to ay particular locatio i a hash table. Our model of hashig items ito a table of size k allows us to thik of the process as idepedet trials, each with k possible outcomes (the k locatios i the table). O each trial we hash aother key ito the table. If we hash items ito a table with k locatios, what is the probability that ay oe item hashes ito locatio? Let X i be the radom variable that couts the umber of items that hash to locatio i trial i (so that X i is either 0 or ). What is the expected value of X i? Let X be the radom variable X + X X. What is the expected value of X? What is the expected umber of items that hash to locatio? Was the fact that we were talkig about locatio special i ay way? That is, does the same expected value apply to every locatio? Exercise Agai we are hashig items ito k locatios. Our model of hashig is that of Exercise What is the probability that a locatio is empty? What is the expected umber of empty locatios? Suppose we ow hash items ito the same umber of locatios. What limit does the expected fractio of empty places approach as gets large? I Exercise 6.5-, the probability that ay oe item hashes ito locatio is /k, because all k locatios are equally likely. The expected value of X i is the /k. The expected value of X is the /k, the sum of terms each equal to /k. Of course the same expected value applies to ay locatio. Thus we have proved the followig theorem. Theorem 6.3 I hashig items ito a hash table of size k, the expected umber of items that hash to ay oe locatio is /k.

2 246 CHAPTER 6. PROBABILITY Expected Number of Empty Locatios I Exercise the probability that positio i will be empty after we hash item ito the table will be k. (Why?) I fact, we ca thik of our process as a idepedet trials process with two outcomes: the key hashes to slot i or it does t. From this poit of view, it is clear that the probability of othig hashig to slot i i trials is ( k ).Now cosider the origial sample space agai ad let X i be ifslot i is empty for a give sequece of hashes or 0 if it is ot. The the umber of empty slots for a give sequece of hashes is X + X X k evaluated at that sequece. Therefore, the expected umber of empty slots is, by Theorem 6.9, k( k ). Thus we have proved aother ice theorem about hashig. Theorem 6.4 I hashig items ito a hash table with k locatios, the expected umber of empty locatios is k( k ). Proof: Give above. If we have the same umber of slots as places, the expected umber of empty slots is ( ), so the expected fractio of empty slots is ( ). What does this fractio approach as grows? Youmay recall that lim ( + ) is e, the base for the atural logarithm. I the problems at the ed of the sectio, we show you how to derive from this that lim ( ) is e.thusfor a reasoably large hash table, if we hash i as may items as we have slots, we expect a fractio /e of those slots to remai empty. I other words, we expect /e empty slots. O the other had, we expect items per locatio, which suggests that we should expect each slot to have a item ad therefore expect to have o empty locatios. Is somethig wrog? No, but we simply have to accept that our expectatios about expectatio just do t always hold true. What wet wrog i that apparet cotradictio is that our defiitio of expected value does t imply that if we have a expectatio of oe key per locatio the every locatio must have a key, but oly that empty locatios have to be balaced out by locatios with more tha oe key. Whe we wat to make a statemet about expected values, we must use either our defiitios or theorems to back it up. This is aother example of why we have to back up ituitio about probability with careful aalysis. Expected Number of Collisios We say that we have a collisio whe we hash a item to a locatio that already cotais a item. How ca we compute the expected umber of collisios? The umber of collisios will be the umber of keys hashed mius the umber of occupied locatios because each occupied locatio will cotai oe key that will ot have collided i the process of beig hashed. Thus, by Theorems 6.9 ad 6.0, E(collisios) E(occupied locatios) k + E(empty locatios) (6.26) where the last equality follows because the expected umber of occupied locatios is k mius the expected umber of uoccupied locatios. This gives us yet aother theorem. Theorem 6.5 I hashig items ito a hash table with k locatios, the expected umber of collisios is k + k( k ).

3 6.5. PROBABILITY CALCULATIONS IN HASHING 247 Proof: We have already show i Theorem 2 that the expected umber of empty locatios is k( k ). Substitutig this ito Equatio 6.26 gives our formula. Exercise I real applicatios, it is ofte the case that the hash table size is ot fixed i advace, sice you do t kow, i advace, how may items you will isert. The most commo heuristic for dealig with this is to start k, the hash table size, at some reasoably small value, ad the whe, the umber of items gets to be greater tha 2k, you double the hash table size. I this exercise, we propose a differet idea. Suppose you waited util every sigle slot i the hash table had at least oe item i it, ad the you icreased the table size. What is the expected umber of items that will be i the table whe you icrease the size? I other words, how may items do you expect to isert ito a hash table i order to esure that every slot has at least oe item? (Hit: Let X i be the umber of items added betwee the time that there are i occupied slots for the first time ad the first time that there are i occupied slots.) For Exercise 6.5-3, the key is to let X i be the umber of items added betwee the time that there are i full slots for the first time ad i full slots for the first time. Let s thik about this radom variable. E(X ), sice after oe isertio there is oe full slot. I fact X itself is equal to. To compute the expected value of X 2,weote that X 2 ca take o ay value greater tha. I fact if we thik about it, what we have here (util we actually hash a item to a ew slot) is a idepedet trials process with two outcomes, with success meaig our item hashes to a uused slot. X 2 couts the umber of trials util the first success. The probability of success is (k )/k. I askig for the expected value of X 2,weare askig for expected umber of steps util the first success. Thus we ca apply Lemma 6.2 to get that it is k/(k ). Cotiuig, X 3 similarly couts the umber of steps i a idepedet trials process (with two outcomes) that stops at the first success ad has probability of success (k 2)/k. Thusthe expected umber of steps util the first success is k/(k 2). I geeral, we have that X i couts the umber of trials util success i a idepedet trials process with probability of success (k i +)/k ad thus the expected umber of steps util the first success is k/(k i + ), which is the expected value of X i. The total time util all slots are full is just X X + + X k. Takig expectatios ad usig Lemma 6.2 we get E(X) k E(X j ) j k k k j + j k k k j + j k k j + k k j + k k i, i

4 248 CHAPTER 6. PROBABILITY where the last lie follows just by switchig the variable of the summatio, that is, lettig k j +i ad summig over i. 6 Now the quatity k i i is kow as a harmoic umber, ad is sometimes deoted by H k.itiswell kow (ad you ca see why i the problems at the ed of the sectio) that k i i Θ(log k), ad more precisely 4 +lk H k +lk, (6.27) ad i fact, 2 +lk H k +lk, (6.28) whe k is large eough. As gets large, H l approaches a limit called Euler s costat; Euler s costat is about.58. Equatio 6.27 gives us that E(X) O(k log k). Theorem 6.6 The expected umber of items eeded to fill all slots of a hash table of size k is betwee k l k + 2k ad k l k + k. Proof: Give above.. So i order to fill every slot i a hash table of size k, weeed to hash roughly k l k items. This problem is sometimes called the coupo collectors problem. Expected maximum umber of elemets i a slot of a hash table I a hash table, the time to fid a item is related to the umber of items i the slot where you are lookig. Thus a iterestig quatity is the expected maximum legth of the list of items i a slot i a hash table. This quatity is more complicated tha may of the others we have bee computig, ad hece we will oly try to upper boud it, rather tha compute it exactly. I doig so, we will itroduce a few upper bouds ad techiques that appear frequetly ad are useful i may areas of mathematics ad computer sciece. We will be able to prove that if we hash items ito a hash table of size, the expected legth of the logest list is O(log / log log ). Oe ca also prove, although we wo t do it here, that with high probability, there will be some list with Ω(log / log log ) items i it, so our boud is, up to costat factors, the best possible. Before we start, we give some useful upper bouds. The first allows us to boud terms that look like ( + x )x, for ay positive x, bye. Lemma 6.7 For all x>0, ( + x )x e. Proof: lim x ( + x )x e, ad + ( x )x has positive first derivative. Secod, we will use a approximatio called Stirlig s formula, ( ) x x 2πx( x! + Θ(/)), e 6 ote that k j +rus from k to asj rus from to k, soweare describig exactly the same sum.

5 6.5. PROBABILITY CALCULATIONS IN HASHING 249 which tells us, roughly, that (x/e) x is a good approximatio for x!. Moreover the costat i the Θ(/) term is extremely small, so for our purposes we will just say that ( ) x x 2πx. x! e (We use this equality oly i our proof of Lemma 6.8. You will see i that Lemma that we make the statemet that 2π>. I fact, 2π>2, ad this is more tha eough to make up for ay lack of accuracy i our approximatio.) Usig Stirlig s formula, we ca get a boud o ( t), Lemma 6.8 For >t>0, ( ) t t t ( t) t. Proof: ( ) t! t!( t)! (6.29) (/e) 2π (t/e) t 2πt(( t)/e) t 2π( t) (6.30) t t ( t) t 2π t( t) (6.3) Now if <t<, we have t( t), sothat t( t).further 2π>. We ca use these to facts to upper boud the quatity marked 6.3 by t t ( t) t Whe t ort, the iequality i the statemet of the lemma is /( ) which is true sice <. We are ow ready to attack the problem at had, the expected value of the maximum list size. Let s start with a related quatity that we already kow how to compute. Let H it be the evet that t keys hash to slot i. P (H it )isjust the probability of t successes i a idepedet trials process with success probability /, so P (H it ) ( t ) ( ) t ( ) t. (6.32) Now we relate this kow quatity to the probability of the evet M t that the maximum list size is t. Lemma 6.9 Let M t be the evet that t is the maximum list size i hashig items ito a hash table of size. LetH t be the evet that t keys hash to positio. The P (M t ) P (H t )

6 250 CHAPTER 6. PROBABILITY Proof: We begi by lettig M it be the evet that the maximum list size is t ad this list appears i slot i. Observe that that sice M it is a subset of H it, We kow that, by defiitio, ad so P (M it ) P (H it ). (6.33) M t M t M t, P (M t )P (M t M t ). Therefore, sice the sum of the probabilities of the idividual evets must be at least as large as the probability of the uio, P (M t ) P (M t )+P (M 2t )+ + P (M t ). (6.34) (Recall that we itroduced the Priciple of Iclusio ad Exclusio because the right had side overestimated the probability of the uio. Note that the iequality i Equatio 6.34 holds for ay uio, ot just this oe: it is sometimes called Boole s iequality.) I this case, for ay i ad j, P (M it )P(M jt ), sice there is o reaso for slot i to be more likely tha slot j to be the maximum. We ca therefore write that P (M t )P (M t ) P (H t ). Now we ca use Equatio 6.32 for P (H t ) ad the apply Lemma 6.8 to get that P (H t ) ( t ) ( ) t ( ) t t t ( t) t ( ) t ( ) t. We cotiue, usig algebra, the fact that ( ) t ad Lemma 6.7 to get t t ( t) t t t t t ( t) t ( ) t t t t ( + t ) t t t t ( ( + t ) t ) t t t t t et t t. We have show the followig:

7 6.5. PROBABILITY CALCULATIONS IN HASHING 25 Lemma 6.20 P (M t ), the probability that the maximum list legth is t, isatmost e t /t t. Proof: Our sequece of equatios ad iequalities above showed that P (H t ) et t. Multiplyig by ad applyig Lemma 6.9 gives our t result. Now that we have a boud o P (M t )weca compute a boud o the expected legth of the logest list, amely P (M t )t. t0 However, if we thik carefully about the boud i Lemma 6.20, we see that we have a problem. For example whe t,the lemma tells us that P (M ) e. This is vacuous, as we kow that ay probability is at most, We could make a stroger statemet that P (M t ) max{e t /t t, }, but eve this would t be sufficiet, sice it would tell us thigs like P (M )+P (M 2 ) 2, which is also vacuous. All is ot lost however. Our lemma causes this problem oly whe t is small. We will split the sum defiig the expected value ito two parts ad boud the expectatio for each part separately. The ituitio is that whe we restrict t to be small, the P (M t )t is small because t is small (ad over all t, P (M t ) ). Whe t gets larger, Lemma 6.20 tells us that P (M t )isvery small ad so the sum does t get big i that case either. We will choose a way to split the sum so that this secod part of the sum is bouded by a costat. I particular we split the sum up by P (M t )t t0 5 log / log log t0 P (M t )t + t 5 log / log log P (M t )t (6.35) For the sum over the smaller values of t, wejust observe that i each term t 5 log / log log so that 5 log / log log t0 P (M t )t 5 log / log log t0 P (M t )5 log / log log (6.36) 5 log / log log 5log / log log P (M t ) (6.37) t0 5 log / log log (6.38) (Note that we are ot usig Lemma 6.20 here; oly the fact that the probabilities of disjoit evets caot add to more tha.) For the rightmost sum i Equatio 6.35, we wat to first compute a upper boud o P (M t ) for t (5log / log log ). Usig Lemma 6.20, ad doig a bit of calculatio we get that i this case P (M t ) / 2. Sice the boud o P (M t ) from Lemma 6.20 decreases as t grows, ad t, weca boud the right sum by t5 log / log log P (M t )t t5 log / log log 2 t5 log / log log Combiig Equatios 6.38 ad 6.39 with 6.35 we get the desired result.. (6.39) Theorem 6.2 If we hash items ito a hash table of size, the expected maximum list legth is O(log / log log ).

8 252 CHAPTER 6. PROBABILITY The choice to break the sum ito two pieces here ad especially the breakpoit we chose may have seemed like magic. What is so special about log / log log? Cosider the boud o P (M t ). If you asked what is the value of t for which the boud equals a certai value, say / 2, you get the equatio e t /t t 2.Ifwetry to solve the equatio e t /t t 2 for t, wequickly see that we get a form that we do ot kow how to solve. (Try typig this ito Mathematica or Maple, to see that it ca t solve this equatio either.) The equatio we eed to solve is somewhat similar to the simpler equatio t t. While this equatio does ot have a closed form solutio, oe ca show that the t that satisfies this equatio is roughly c log / log log, for some costat c. This is why some multiple of log / log log made sese to try as the the magic value. For values much less tha log / log log the boud provided o P (M t )isfairly large. Oce we get past log / log log, however, the boud o P (M t ) starts to get sigificatly smaller. The factor of 5 was chose by experimetatio to make the secod sum come out to be less tha. We could have chose ay umber betwee 4 ad 5 to get the same result; or we could have chose 4 ad the secod sum would have grow o faster tha the first. Importat Cocepts, Formulas, ad Theorems. Expected Number of Keys per Slot i Hash Table. I hashig items ito a hash table of size k, the expected umber of items that hash to ay oe locatio is /k. 2. Expected Number of Empty Slots i Hash Table. I hashig items ito a hash table with k locatios, the expected umber of empty locatios is k( k ). 3. Collisio i Hashig. We say that we have a collisio whe we hash a item to a locatio that already cotais a item. 4. The Expected Number of Collisios i Hashig. I hashig items ito a hash table with k locatios, the expected umber of collisios is k + k( k ). 5. Harmoic Number. The quatity k i i is kow as a harmoic umber, ad is sometimes deoted by H k.itisafact that that k i i Θ(log k), ad more precisely 2 +lk H k +lk. 6. Euler s Costat. As gets large, H l approaches a limit called Euler s costat; Euler s costat is about Expected Number of Hashes util all Slots of a Hash Table Are Occupied. The expected umber of items eeded to fill all slots of a hash table of size k is betwee k l k + 2 k ad k l k + k. 8. Expected Maximum Number of Keys per Slot. If we hash items ito a hash table of size, the expected maximum list legth is O(log / log log ).

9 6.5. PROBABILITY CALCULATIONS IN HASHING 253 Problems. A cady machie i a school has d differet kids of cady. Assume (for simplicity) that all these kids of cady are equally popular ad there is a large supply of each. Suppose that c childre come to the machie ad each purchases oe package of cady. Oe of the kids of cady is a Sackers bar. What is the probability that ay give child purchases a Sackers bar? Let Y i be the umber of Sackers bars that child i purchases, so that Y i is either 0 or. What is the expected value of Y i? Let Y be the radom variable Y + Y Y c. What is the expected value of Y? What is the expected umber of Sackers bars that is purchased? Does the same result apply to ay of the varieties of cady? 2. Agai as i the previous exercise, we have c childre choosig from amog ample supplies of d differet kids of cady, oe package for each child, ad all choices equally likely. What is the probability that a give variety of cady is chose by o child? What is the expected umber of kids of cady chose by o child? Suppose ow that c d. What happes to the expected umber of kids of cady chose by o child? 3. How may childre do we expect to have to observe buyig cady util someoe has bought a Sackers bar? 4. How may childre to we expect to have to observe buyig cady util each type of cady has bee selected at least oce? 5. If we have 20 kids of cady, how may childre have to buy cady i order for the probability to be at least oe half that (at least) two childre buy the same kid of cady? 6. What is the expected umber of duplicatios amog all the cady the childre have selected? 7. Compute the values o the left-had ad right-had side of the iequality i Lemma 6.8 for 2,t 0,, 2 ad for 3,t 0,, 2, Whe we hash items ito k locatios, what is the probability that all items hash to differet locatios? What is the probability that the ith item is the first collisio? What is the expected umber of items we must hash util the first collisio? Use a computer program or spreadsheet to compute the expected umber of items hashed ito a hash table util the first collisio with k 20ad with k We have see a umber of occasios whe our ituitio about expected values or probability i geeral fails us. Whe we wrote dow Equatio 6.26 we said that the expected umber of occupied locatios is k mius the expected umber of uoccupied locatios. While this seems obvious, there is a short proof. Give the proof. 0. Write a computer program that prits out a table of values of the expected umber of collisios with keys hashed ito a table with k locatios for iterestig values of ad k. Does this value vary much as ad k chage?. Suppose you hash items ito a hash table of size k. Itisatural to ask about the time it takes to fid a item i the hash table. We ca divide this ito two cases, oe whe the item is ot i the hash table (a usuccessful search), ad oe whe the item is i the hash table (a successful search). Cosider first the usuccessful search. Assume the keys hashig

10 254 CHAPTER 6. PROBABILITY to the same locatio are stored i a list with the most recet arrival at the begiig of the list. Use our expected list legth to boud the expected time for a usuccessful search. Next cosider the successful search. Recall that whe we isert items ito a hash table, we typically isert them at the begiig of a list, ad so the time for a successful search for item i should deped o how may etries were iserted after item i. Carefully compute the expected ruig time for a successful search. Assume that the item you are searchig for is radomly chose from amog the items already i the table. (Hit: The usuccessful search should take roughly twice as log as the successful oe. Be sure to explai why this is the case.) 2. Suppose I hash log items ito buckets. What is the expected maximum umber of items i a bucket? 3. The fact that lim ( + ) e (where varies over itegers) is a cosequece of the fact that lim h 0 ( + h) h e (where h varies over real umbers). Thus if h varies over egative real umbers, but approaches 0, the limit still exists ad equals e. What does this tell you about lim ( + )? Usig this ad rewritig ( ) as ( + ) show that 4. What is the expected umber of empty slots whe we hash 2k items ito a hash table with k slots? What is the expected fractio of empty slots close to whe k is reasoably large? 5. Usig whatever methods you like (had calculatios or computer), give upper ad/or lower bouds o the value of the x satisfyig x x. 6. Professor Max Weiberger decides that the method proposed for computig the maximum list size is much too complicated. He proposes the followig solutio. Let X i be the size of list i. The what we wat to compute is E(max i (X i )). Well E(max i What is the flaw i his solutio? (X i )) max(e(x i )) max(). i 7. Prove as tight upper ad lower bouds as you ca o k i i.for this purpose it is useful to remember the defiitio of the atural logarithm as a itegral ivolvig /x ad to draw rectagles ad other geometric figures above ad below the curve. 8. Notice that l! i l i. Sketch a careful graph of y lx, ad by drawig i geometric figures above ad below the graph, show that l i 2 l l xdx l i. i i Based o your drawig, which iequality do you thik is tighter? Use itegratio by parts to evaluate the itegral. What bouds o! ca you get from these iequalities? Which oe do you thik is tighter? How does it compare to Stirlig s approximatio? What big Oh boud ca you get o!? i