How fast can we sort? Sorting. Decision-tree model. Decision-tree for insertion sort Sort a 1, a 2, a 3. CS Spring 2009

Transcription

1 CS 4 -- Spring 2009 Sorting Crol Wenk Slides courtesy of Chrles Leiserson with smll chnges by Crol Wenk CS 4 Anlysis of Algorithms 1 How fst cn we sort? All the sorting lgorithms we hve seen so fr re comprison sorts: only use comprisons to determine the reltive order of elements. E.g., insertion sort, merge sort, quicksort, hepsort. The best worst-cse running time tht we ve seen for comprison sorting is O(n log n). Is O(n log n) the best we cn do? Decision trees cn help us nswer this question. CS 4 Anlysis of Algorithms 2 Decision-tree model A decision tree models the execution of ny comprison sorting lgorithm: One tree per input size n. The tree contins ll possible comprisons (= if-brnches) tht could be executed for ny input of size n. The tree contins ll comprisons long ll possible instruction trces (= control flows) for ll inputs of size n. For one input, only one pth to lef is executed. Running time = length of the pth tken. Worst-cse running time = height of tree. CS 4 Anlysis of Algorithms Decision-tree for insertion sort Sort 1, 2, 1 2 insert 2 insert < 1 : 2 2 : 1 : 2 1 insert 1 2 < : : Ech internl node is lbeled i : j for i, j {1, 2,, n}. The left subtree shows subsequent comprisons if i < j. The right subtree shows subsequent comprisons if i j. CS 4 Anlysis of Algorithms 4

2 Decision-tree for insertion sort Sort 1, 2, = <9,4,6> 1 2 insert 2 insert < 1 : 2 2 : 1 : 2 1 insert 1 2 < : : Ech internl node is lbeled i : j for i, j {1, 2,, n}. The left subtree shows subsequent comprisons if i < j. The right subtree shows subsequent comprisons if i j. CS 4 Anlysis of Algorithms 5 Decision-tree for insertion sort Sort 1, 2, = <9,4,6> 1 2 insert 2 insert < 1 : : 1 : 2 1 insert 1 2 < : : Ech internl node is lbeled i : j for i, j {1, 2,, n}. The left subtree shows subsequent comprisons if i < j. The right subtree shows subsequent comprisons if i j. CS 4 Anlysis of Algorithms 6 Decision-tree for insertion sort Sort 1, 2, = <9,4,6> 1 2 insert 2 insert < 1 : 2 2 : 1 : 2 1 insert < : : Ech internl node is lbeled i : j for i, j {1, 2,, n}. The left subtree shows subsequent comprisons if i < j. The right subtree shows subsequent comprisons if i j. CS 4 Anlysis of Algorithms 7 Decision-tree for insertion sort Sort 1, 2, = <9,4,6> 1 2 insert 2 insert < 1 : 2 2 : 1 : 2 1 insert 1 2 < : : 4 < Ech internl node is lbeled i : j for i, j {1, 2,, n}. The left subtree shows subsequent comprisons if i < j. The right subtree shows subsequent comprisons if i j. CS 4 Anlysis of Algorithms 8

3 Decision-tree for insertion sort Sort 1, 2, = <9,4,6> 1 2 insert 2 insert < 1 : 2 2 : 1 : 2 1 insert 1 2 < : : <6 9 Ech internl node is lbeled i : j for i, j {1, 2,, n}. The left subtree shows subsequent comprisons if i < j. The right subtree shows subsequent comprisons if i j. CS 4 Anlysis of Algorithms 9 Decision-tree for insertion sort Sort 1, 2, = <9,4,6> 1 2 insert 2 insert < 1 : 2 2 : 1 : 2 1 insert 1 2 < : : <6 9 Ech lef contins permuttion π(1), π(2),, π(n) to indicte tht the ordering π(1) π(2)... π(n) hs been estblished. CS 4 Anlysis of Algorithms 10 Decision-tree model A decision tree models the execution of ny comprison sorting lgorithm: One tree per input size n. The tree contins ll possible comprisons (= if-brnches) tht could be executed for ny input of size n. The tree contins ll comprisons long ll possible instruction trces (= control flows) for ll inputs of size n. For one input, only one pth to lef is executed. Running time = length of the pth tken. Worst-cse running time = height of tree. Lower bound for comprison sorting Theorem. Any decision tree tht cn sort n elements must hve height Ω(n log n). Proof. The tree must contin n! leves, since there re n! possible permuttions. A height-h binry tree hs 2 h leves. Thus, n! 2 h. h log(n!) (log is mono. incresing) log ((n/2) n/2 ) = n/2 log n/2 h Ω(n log n). CS 4 Anlysis of Algorithms 11 CS 4 Anlysis of Algorithms 12

4 Lower bound for comprison sorting Corollry. Hepsort nd merge sort re symptoticlly optiml comprison sorting lgorithms. Sorting in liner time Counting sort: No comprisons between elements. Input: A[1.. n], where A[ j] {1, 2,, k}. Output: B[1.. n], sorted. Auxiliry storge: C[1.. k]. CS 4 Anlysis of Algorithms 1 CS 4 Anlysis of Algorithms 14 Counting sort Counting-sort exmple 1. for i 1 to k do C[i] 0 2. for j 1 to n do C[A[ j]] C[A[ j]] + 1. for i 2 to k do C[i] C[i] + C[i 1] C[i] = {key = i} C[i] = {key i} 5 C: CS 4 Anlysis of Algorithms 15 CS 4 Anlysis of Algorithms 16

5 Loop 1 Loop 2 5 C: C: for i 1 to k do C[i] 0 2. for j 1 to n do C[A[ j]] C[A[ j]] + 1 C[i] = {key = i} CS 4 Anlysis of Algorithms 17 CS 4 Anlysis of Algorithms 18 Loop 2 Loop 2 5 C: C: for j 1 to n do C[A[ j]] C[A[ j]] + 1 C[i] = {key = i} 2. for j 1 to n do C[A[ j]] C[A[ j]] + 1 C[i] = {key = i} CS 4 Anlysis of Algorithms 19 CS 4 Anlysis of Algorithms 20

6 Loop 2 Loop 2 5 C: C: for j 1 to n do C[A[ j]] C[A[ j]] + 1 C[i] = {key = i} 2. for j 1 to n do C[A[ j]] C[A[ j]] + 1 C[i] = {key = i} CS 4 Anlysis of Algorithms 21 CS 4 Anlysis of Algorithms 22 Loop Loop 5 C: C: C': C': for i 2 to k do C[i] C[i] + C[i 1] C[i] = {key i}. for i 2 to k do C[i] C[i] + C[i 1] C[i] = {key i} CS 4 Anlysis of Algorithms 2 CS 4 Anlysis of Algorithms 24

7 Loop 5 C: C: C': C': for i 2 to k do C[i] C[i] + C[i 1] C[i] = {key i} CS 4 Anlysis of Algorithms 25 CS 4 Anlysis of Algorithms 26 5 C: C: C': C': CS 4 Anlysis of Algorithms 27 CS 4 Anlysis of Algorithms 28

8 5 C: C: C': C': CS 4 Anlysis of Algorithms 29 CS 4 Anlysis of Algorithms 0 5 C: C: C': C': CS 4 Anlysis of Algorithms 1 CS 4 Anlysis of Algorithms 2

9 5 C: C: C': C': CS 4 Anlysis of Algorithms CS 4 Anlysis of Algorithms C: C': Anlysis Θ(k) Θ(n) Θ(k) Θ(n) Θ(n + k) 1. for i 1 to k do C[i] 0 2. for j 1 to n do C[A[ j]] C[A[ j]] + 1. for i 2 to k do C[i] C[i] + C[i 1] CS 4 Anlysis of Algorithms 5 CS 4 Anlysis of Algorithms 6

10 Running time If k = O(n), then counting sort tkes Θ(n) time. But, sorting tkes Ω(n log n) time! Where s the fllcy? Answer: Comprison sorting tkes Ω(n log n) time. Counting sort is not comprison sort. In fct, not single comprison between elements occurs! Stble sorting Counting sort is stble sort: it preserves the input order mong equl elements Exercise: Wht other sorts hve this property? CS 4 Anlysis of Algorithms 7 CS 4 Anlysis of Algorithms 8 Rdix sort Opertion of rdix sort Origin: Hermn Hollerith s crd-sorting mchine for the 1890 U.S. Census. (See Appendix.) Digit-by-digit sort. Hollerith s originl (bd) ide: sort on most-significnt digit first. Good ide: Sort on lest-significnt digit first with n uxiliry stble sorting lgorithm (like counting sort) CS 4 Anlysis of Algorithms 9 CS 4 Anlysis of Algorithms 40

11 Correctness of rdix sort Correctness of rdix sort Induction on digit position Assume tht the numbers re sorted by their low-order t 1digits. Sort on digit t Induction on digit position Assume tht the numbers re sorted by their low-order t 1digits. Sort on digit t Two numbers tht differ in digit t re correctly sorted CS 4 Anlysis of Algorithms 41 CS 4 Anlysis of Algorithms 42 Correctness of rdix sort Anlysis of rdix sort Induction on digit position Assume tht the numbers re sorted by their low-order t 1digits. Sort on digit t Two numbers tht differ in digit t re correctly sorted. Two numbers equl in digit t re put in the sme order s the input correct order CS 4 Anlysis of Algorithms 4 Sort n computer words of b bits ech. View ech word s hving b/r bse-2 r digits. Exmple: 2-bit word (b=2) r = 1: 2 bse-2 digits 2 b/r =2psses of counting 1 2 sort on bse-2 digits r = 8: 2/8 bse-28 digits (2 8 ) (2 8 ) 2 (2 8 ) 1 (2 8 ) 0 b/r =4psses of counting sort on bse-2 8 digits r = 16: 2/16 bse-216 digits (2 16 ) 1 (2 16 ) 0 b/r =2psses of counting sort on bse-2 16 digits CS 4 Anlysis of Algorithms 44

12 Anlysis of rdix sort Sort n computer words of b bits ech. View ech word s hving b/r bse-2 r digits. Assume counting sort is the uxiliry stble sort. Mke b/r psses of counting sort on bse-2 r digits How mny psses should we mke? CS 4 Anlysis of Algorithms 45 Anlysis (continued) Recll: Counting sort tkes Θ(n + k) time to sort n numbers in the rnge from 0 to k 1. If ech b-bit word is broken into r-bit pieces, ech pss of counting sort tkes Θ(n + 2 r ) time. Since there re b/r psses, we hve T ( n, b) = Θ b ( n + 2r ) r. Choose r to minimize T(n, b): Incresing r mens fewer psses, but s r >> log n, the time grows exponentilly. CS 4 Anlysis of Algorithms 46 Choosing r T ( n, b) = Θ b ( n + 2r ) r Minimize T(n, b) by differentiting nd setting to 0. Or, just observe tht we don t wnt 2 r > n, nd there s no hrm symptoticlly in choosing r s lrge s possible subject to this constrint. Choosing r = log n implies T(n, b) = Θ(bn/log n). CS 4 Anlysis of Algorithms 47 Rdix Sort with optimized r Assume counting sort is the uxiliry stble sort. Sort n computer words of b bits ech. The runtime of rdix sort is: T(n, b) = Θ(bn/log n). Exmple: For numbers in the rnge from 0 to n d 1, we hve b = d log n rdix sort runs in Θ(dn) time. Notice tht counting sort runs in O(n+k) time, where ll numbers re in the rnge 1 through k. CS 4 Anlysis of Algorithms 48

13 Conclusions In prctice, rdix sort is fst for lrge inputs, s well s simple to code nd mintin. Exmple (2-bit numbers): At most psses when sorting 2000 numbers. Merge sort nd quicksort do t lest log 2000 = 11 psses. Downside: Unlike quicksort, rdix sort displys little loclity of reference, nd thus well-tuned quicksort fres better on modern processors, which feture steep memory hierrchies. CS 4 Anlysis of Algorithms 49 Appendix: Punched-crd technology Hermn Hollerith ( ) Punched crds Hollerith s tbulting system Opertion of the sorter Origin of rdix sort Modern IBM crd Return to lst slide viewed. CS 4 Anlysis of Algorithms 50 Hermn Hollerith ( ) The 1880 U.S. Census took lmost 10 yers to process. While lecturer t MIT, Hollerith prototyped punched-crd technology. His mchines, including crd sorter, llowed the 1890 census totl to be reported in 6 weeks. He founded the Tbulting Mchine Compny in 1911, which merged with other compnies in 1924 to form Interntionl Business Mchines. CS 4 Anlysis of Algorithms 51 Punched crds Punched crd = dt record. Hole = vlue. Algorithm = mchine + humn opertor. Replic of punch crd from the 1900 U.S. census. [Howells 2000] CS 4 Anlysis of Algorithms 52

14 Opertion of the sorter Hollerith s tbulting system Pntogrph crd punch Hnd-press reder Dil counters Sorting box Figure from [Howells 2000]. CS 4 Anlysis of Algorithms 5 An opertor inserts crd into the press. Pins on the press rech through the punched holes to mke electricl contct with mercuryfilled cups beneth the crd. Whenever prticulr digit vlue is punched, the lid of the corresponding sorting bin lifts. The opertor deposits the crd Hollerith Tbultor, Pntogrph, Press, nd Sorter into the bin nd closes the lid. When ll crds hve been processed, the front pnel is opened, nd the crds re collected in order, yielding one pss of stble sort. CS 4 Anlysis of Algorithms 54 Origin of rdix sort Hollerith s originl 1889 ptent lludes to mostsignificnt-digit-first rdix sort: The most complicted combintions cn redily be counted with comprtively few counters or relys by first ssorting the crds ccording to the first items entering into the combintions, then ressorting ech group ccording to the second item entering into the combintion, nd so on, nd finlly counting on few counters the lst item of the combintion for ech group of crds. Lest-significnt-digit-first rdix sort seems to be folk invention originted by mchine opertors. Modern IBM crd One chrcter per column. Produced by the WWW Virtul Punch- Crd Server. So, tht s why text windows hve 80 columns! CS 4 Anlysis of Algorithms 55 CS 4 Anlysis of Algorithms 56