SFU CMPT Lecture: Week 3

Transcription

1 SFU CMPT Lecture: Week 3 SFU CMPT Lecture: Week 3 Ján Maňuch jmanuch@sfu.ca Lecture on May 20, 2008, 5.30pm-8.20pm

2 SFU CMPT Lecture: Week 3 Important basic procedures for max-heaps 1. Max-Heapify, runs in time Ç ÐÓ Òµ time, a key procedure that maintains the max-heap property (if we change a value in the root of a heap, this procedure will correct the heap, so that it s again a heap) 2. Build-Max-Heap, runs in time Ç Òµ, produces a max-heap from unsorted data 3. Heap-Sort, runs in time Ç Ò ÐÓ Òµ, sorts array in place (uses above) Also Max-Heap-Insert, Heap-Extract-Max, Heap-Increase-Key (run in time Ç ÐÓ Òµ), Heap-Maximum, (run in time Ç ½µ), used when the heap is used to implement a priority queue

3 SFU CMPT Lecture: Week 3 Max-Heapify Maintains the max-heap property Inputs are an array and an index Assumption: sub-trees rooted in LEFT µ and RIGHT µ are proper max-heaps, but may be smaller than its children

4 SFU CMPT Lecture: Week 3 Task of Max-Heapify is to let float down in the max-heap below it so that heap rooted in becomes proper max-heap

5 SFU CMPT Lecture: Week 3 µ Max-Heapify LEFT µ 1: 2: Ö RIGHT µ 3: if heap-size µ and then 4: largest 5: else 6: largest 7: end if 8: if Ö heap-size µ and Ö largest then 9: Ö largest 10: end if 11: if largest then 12: exchange largest 13: Max-Heapify largestµ 14: end if

6 SFU CMPT Lecture: Week 3 Idea: Lines 1 2 are just for convenience Lines 3 10 find the largest of elements,, and Ö Lines first check if there s anything to be done at all, 2. and if yes, (a) move the misplaced element one level down, (b) and make a recursive call one level deeper on this element We know that after the exchange we have the largest of,, and Ö in position, so among these three, everything is OK. However, further down may still be problems. Also note that we check heap-size µ and Ö heap-size µ, so that we don t go checking outside of the heap the recursion will end if these tests fail

7 SFU CMPT Lecture: Week

8 SFU CMPT Lecture: Week 3 Running time of Max-Heapify on subtree of size Ò rooted at is ½µ for finding largest and possibly swapping plus time to run Max-Heapify on sub-tree rooted in one of the children of Size of s sub-tree in consideration is about Ò ½µ ¾ if complete trees We allow for nearly complete trees, but here the size of any sub-tree is at most ¾Ò This worst-case occurs if last level is exactly half full ½½: ¾ ½½ ¾¾ Ò

9 Ì Òµ Ì ¾Ò µ ½µ SFU CMPT Lecture: Week 3 This gives Case 2 of Master Theorem gives Ì Òµ ÐÓ Òµ Alternative proof: running time of Max-Heapify on node of height (counted from bottom!) Ç µ is Ç ÐÓ Òµ and by Assignment Problem 2.5.

10 SFU CMPT Lecture: Week 3 Building a heap Easy using Max-Heapify Suppose we have given an unordered array Ò with Ò length µ ½ One can show that the elements ½ Ò ¾ ¾ Ò Ò ¾ are the leaves of a heap.. Thus, it s OK to initially consider them as ½-element heaps, and run Max-Heapify on top of them, once for each non-leaf element (½-element heaps are always proper heaps!). Build-Max-Heap µ heap-size µ length µ 1: 2: for length µ ¾ downto ½ do 3: Max-Heapify µ 4: end for

11 SFU CMPT Lecture: Week Input array A Binary tree representing A Result

12 SFU CMPT Lecture: Week 3 Build-Max-Heap Correctness Loop invariant: At the start of each iteration, each node ½ ¾ Ò is a root of a proper max-heap Initialization: Initially, Ò ¾. Nodes Ò ¾ ½ Ò ¾ ¾ Ò are leaves. Leaves are always roots of trivial max-heaps. Maintenance: Observe, children nodes of are numbered higher than. By invariant, both are roots of max-heaps. Thus, we can call Max-Heapify µ. Now is a root of max-heap. Max-Heapify does not destroy the max-heap property of ½ ¾ Ò, thus they re still roots of max-heaps. Decrementing reestablishes invariant for next iteration. Termination: Now ¼, thus ½ ¾ Ò are roots of max-heaps (in particular, ½ is).

13 SFU CMPT Lecture: Week 3 Build-Max-Heap Running time Simple bound: calls to Max-Heapify cost Ç ÐÓ Òµ, there are Ç Òµ of them, thus Ç Ò ÐÓ Òµ. Correct, but not tight (asymptotically). Truth is Òµ. First observation: time for Max-Heapify depends on height of node (clearly). Second observation: Ò-element heap has height ÐÓ Ò (Problem 2.5). Third observation: it has at most Ò ¾ ½ nodes of any height (Problem 3.1).

14 SFU CMPT Lecture: Week 3 Assignment Problem 3.1. (deadline: May 27, 5:30pm) Prove by mathematical induction on that there are at most Ò ¾ ½ nodes of height in any Ò-element heap. Hint: For the induction step, your induction hypothesis is For any Ò, Ò-element heap has at most Ò ¾ nodes of height ½. and you want to prove that For any Ò, Ò-element heap has at most Ò ¾ ½ nodes of height. Hence, if you are proving the claim for a heap À with Ò elements, then you can apply the induction hypothesis on any heap (for instance on a heap which contains only a part of À).

15 Ì Òµ Ò ½ Ñ Ð Ç Ò Ç µ ¾ ½ Ü ¾ Ü Üµ ¾ ½ ¾ ¾ SFU CMPT Lecture: Week 3 We know: Time required by Max-Heapify on node of height is Ç µ. Thus, running time of Build-Max-Heap is upper-bounded by Ò ÐÓ Ò ÐÓ ¼ ¼ By formula and substituting Ü ½ ¾ we obtain ¼ and thus ½ ¼ ½ ¾ ½ ½ ¾µ ¾ ¾ ¾ Ò ÐÓ ½ Ç Ò Òµ Ì Ç Òµ Ò Ç ¼ ¼ Hence, Build-Max-Heap runs in linear time.

16 SFU CMPT Lecture: Week 3 Heapsort Now it s very easy to write down the algorithm for Heapsort. Idea as follows: 1. Given unsorted array, build heap on, using Build-Max-Heap 2. Extract largest element (is in ½ ), and move it to the end of array (swap ½ and Ò ) 3. Decrease the size of the heap by 1 4. The root might not satisfy the heap property (that s where the element formerly in Ò now is), hence, using Max-Heapify, correct the heap 5. Extract the 2nd-largest element (again, in ½ ), and so on...

17 SFU CMPT Lecture: Week 3 Heapsort µ Build-Max-Heap µ 1: 2: for length µ downto ¾ do 3: exchange ½ 4: heap-size µ heap-size µ ½ 5: Max-Heapify ½µ 6: end for Running time: Ç Ò ÐÓ Òµ (Build-Max takes Ç Òµ, and then Ç Òµ rounds with Ç ÐÓ Òµ each). Correctness Loop invariant: At the start of each iteration of the for loop of lines 2 5, the subarray ½ is a max-heap containing the smallest elements of ½ Ò, and the subarray ½ Ò contains the Ò largest elements in sorted order.

18 SFU CMPT Lecture: Week 3 Initialization: After running Build-Max-Heap, the elements ½ Ò form a max-heap. Since, innitially Ò, the first subarray is ½ Ò (a max-heap), while the second is empty, and so satisfies all conditions trivially. Maintenance: Since ½ contains the maximal element of ½, but is smaller than any of the elements in ½ Ò, after swapping ½ and : Ò is sorted (since ½ Ò was sorted) elements in ½ ½ are smaller than elements in Ò ½ ½ is a max-heap except for the root Lines 4-5 correct the max-heap property in the root, hence, in the end of the ½ ½ iteration is a proper max-heap. Termination: in the end, we have that the loop invariant is satisfied for ½: elements ¾ Ò are sorted and greater than ½... done

19 SFU CMPT Lecture: Week 3 (Simple) Queues dequeue (remove) What is a queue? a b c enqueue (insert) We insert new elements at the tail, and remove elements from the head. The standard queue is a First-In-First-Out (FIFO) buffer. Many applications, for instance CD burning: reader process reads files from a hard disk and inserts data into a queue writer process reads from the queue and writes onto CD

20 SFU CMPT Lecture: Week 3 Priority queues They are different: there s no real FIFO rule anymore. A priority queue maintains set Ë of elements, each with a key (priority). Two kinds: max-priority queues and min-priority queues, usually implemented by max-heaps and min-heaps. Max-priority queue Operations: Insert Ë Üµ inserts element Ü into set Ë Maximum ˵ returns element of Ë with largest key Extract-Max ˵ removes and returns element of Ë with largest key Increase-Key Ë Ü µ increases Ü s key to new value, assuming is at least as large as Ü s old key Min-priority queue is used via operations: Insert, Minimum, Extract-Min, and Decrease-Key.

21 SFU CMPT Lecture: Week 3 Example for Max-priority queue: job scheduling on a shared computer What are they good for? jobs with priorities, can be stored in a max-priority queue each time a new job is to be scheduled, it s got to be one of highest priority (Extract-Max operation) new jobs can be inserted using Insert operation in order to avoid starvation, priorities can be increased (Increase-Key operation) Example for Min-priority queue: event-driven simulator items in the queue are events to be simulated, and the key is the time of the event when it should occur (happen) events are simulated in time order, hence we always extract the event with the smallest time from the queue and simulate it new events are procuded, so they have to be inserted in the queue

22 SFU CMPT Lecture: Week 3 Heaps are very convenient here: using max-heaps, we know that the largest element is in ½ : we Ç ½µ have access to largest element removing/inserting elements and increasing keys means that we (basically) can call Max-Heapify or a similar procedure (fixing the heap from bottom up) at the right place (relatively efficient operation Òµ) Ç ÐÓ Min-priority queues analogous. Implementation We re considering max-priority queues, implemented using max-heaps Let s start with a simple operation. Heap-Maximum µ 1: ½ return implements Maximum operation in Ç ½µ time.

23 SFU CMPT Lecture: Week 3 Extract max Now, how to we remove the largest element from the queue/heap so that we will still have a proper max-heap? Heap-Extract-Max µ 1: heap-size µ ½ if then 2: error heap underflow 3: end if 4: max ½ 5: ½ heap-size µ 6: heap-size µ heap-size µ ½ Max-Heapify ½µ 7: 8: return max implements Extract-Max Running time is Ç ÐÓ Òµ

24 SFU CMPT Lecture: Week 3 Increase key Suppose that the element which key is to be increased is identified by index We first update the key Clearly, this can destroy the max-heap property, thus we need to find a new place for this element The idea is to move the updated element as far up toward the root as necessary

25 SFU CMPT Lecture: Week 3 keyµ Heap-Increase-Key 1: key if then 2: error new key is smaller than current key 3: end if 4: key 5: while ½ and Parent µ do 6: exchange Parent µ Parent µ 7: 8: end while Running time is Ç ÐÓ Òµ (height of tree)

26 SFU CMPT Lecture: Week 3 Example: ½ µ Heap-Increase-Key The node with index : update the key from ½ to

27 SFU CMPT Lecture: Week 3 Assignment Problem 3.2. (deadline: May 27, 5:30pm) Argue the correctness of Heap-Increase-Key using the following loop invariant At the start of each iteration of the while loop of lines 5 8, the array ½ heap-size µ satisfies the max-heap property with possible one exception: may be larger than Parent µ.

28 SFU CMPT Lecture: Week 3 Insert Inserting is now easy: 1. add a new element to the end of heap 2. set its key to desired value keyµ Heap-Insert heap-size µ heap-size µ ½ 1: 2: heap-size µ ½ 3: Heap-Increase-Key heap-size µ keyµ Running time is Ç ÐÓ Òµ Conclusion: all considered operations can be implemented to run in time Ç ÐÓ Òµ, Maximum µ even in Ç ½µ Note: there are other, more efficient implementations around!

29 SFU CMPT Lecture: Week 3 Exercise 3.1. Consider an implementation of priority queue using a sorted array instead of a max-heap. What are the running time of the four operations needed for priority queue?

30 SFU CMPT Lecture: Week 3 A lower bound for comparison-based sorting We ve seen a few sorting algorithms: Selection-Sort: in place, upper bound Ç Ò ¾ µ Merge-Sort: upper bound Ç Ò ÐÓ Òµ Heap-Sort: in place, upper bound Ç Ò ÐÓ Òµ Can we get any better algorithm? Or is Ò ÐÓ Òµ an inherent barrier? a comparison sort information about the elements is collected only via comparing the elements We prove that any comparison sort must make Å Ò ÐÓ Òµ comparisons in the worst case to sort Ò elements.

31 SFU CMPT Lecture: Week 3 input sequence ½ ¾ Ò for and we can perform any of the tests,,, to determine their relative order we are not interested in actual values of the elements for simplicity let s us assume that for all, and hence we can assume that all comparisons have the form Decision tree a full binary tree (every node has either zero or two children) represents comparisons between elements performed by particular algorithm run on all possible inputs only comparisons are relevant, everything else is ignored

32 SFU CMPT Lecture: Week 3 internal nodes are labelled for ½ Ò, meaning elements and are compared edges are labelled or, depending on the outcome of the comparisons ¾µ Òµ ½µ i.e., at this point representing the output of the algorithm, leaves are labelled with a permutation ½µ ¾µ Òµ a branch from the root to a leaf describes a sequence of comparisons (nodes and edges) resulting in a permutation (a leaf) a reachable node a node into which we can get on an input; especially we are interested in reachable leaves

33 SFU CMPT Lecture: Week 3 Example: SELECTION-SORT Ò ½ ½ Ò 1: for to do 2: /* 3 10: find min in */ 3: smallest element ½ Ò 4: smallest position 5: for to do 6: if smallest element then 7: smallest element 8: smallest position 9: end if 10: end for 11: swap and smallest position 12: end for

34 ½ ½µ, ¾ ¾µ,..., Ò Òµ SFU CMPT Lecture: Week 3 Note: distinguish between the input sequence ½ ¾ Ò and the array ½ ¾ Ò used to store the sequence. in the beginning we have ½, ¾ ¾,..., Ò Ò ½ but in the end we want

35 SFU CMPT Lecture: Week 3 Decision tree for Selection-Sort on -element inputs < 1:2 > < 1:3 2:3 > < > 2:3 1:2 1:3 1:2 < > < > < > < > 1,2,3 1,3,2 3,1,2 3,1,2 2,1,3 2,3,1 3,2,1 3,2,1 SELECTION-SORT Ò ½ ½ Ò 1: for to do 2: /* 3 10: find min in */ 3: smallest element ½ Ò 4: smallest position 5: for to do 6: if smallest element then 7: smallest element 8: smallest position 9: end if 10: end for 11: swap and smallest position 12: end for

36 SFU CMPT Lecture: Week 3 More efficient decision tree < 1:2 > < 1:3 2:3 > < > < 2:3 > 3,1,2 1:3 3,2,1 < > 1,2,3 1,3,2 2,1,3 2,3,1

37 SFU CMPT Lecture: Week 3 Assignment Problem 3.3. (deadline: May 27, 5:30pm) Draw an efficient decision tree for inputs with elements. Hint: The height of the tree should be.

38 SFU CMPT Lecture: Week 3 Any correct sorting algorithm must be able to produce each permutation of its input Thus, a necessary condition is that each of the Ò permutations must appear in a reachable leaf of the decision tree Lower bound for worst case Length of longest path from root of decision tree to any leaf represents worst case number of comparisons Lower bound on heights of all decision trees where each permutation appears as leaf is thus lower bound on running time of any comparison based sort algorithm

39 Binary tree of height has at most ¾ leaves, thus Ò ¾ SFU CMPT Lecture: Week 3 Theorem. Any comparison sort algorithm requires Å Ò ÐÓ Òµ comparisons in the worst case. Proof. Sufficient to determine the height of a decision tree in which each permutation appears as a leaf Consider decision tree of height with leaves corresponding to a comparison sort on Ò elements Each of the Ò permutations of the input appears as a leaf, therefore Ò Take logs: ÐÓ Ò µ Å Ò ÐÓ Òµ Exercise 3.2. Show that there is no comparison sort whose running time is linear for at least half of the Ò inputs of length Ò. What about a fraction of ½ Ò of the inputs of length Ò? What about a fraction ½ ¾ Ò?

40 SFU CMPT Lecture: Week 3 Quicksort Quicksort is based on Divide&Conquer paradigm its worst-case running time is Ò ¾ µ so why should we consider it? efficient in average: the expected running time is Ò ÐÓ Òµ with a small constant (hidden in Theta-notation)! in place in practice, we don t care that it performs very bad for some few inputs, if it performs very good on the most of them µ the best practical choice for sorting

41 SFU CMPT Lecture: Week 3 Input: array ½ Ò how to sort a subarray Ô Ö : Divide: Compute (in some way) some index Õ, and partition Ô Ö into two (possibly empty) subarrays Ô Õ ½ and Õ ½ Ö s.t. each element of Ô Õ ½ is less than or equal to Õ, and each element of Õ ½ Ö is greater than Õ element Õ is called a pivot Conquer: Sort the two subarrays Ô Õ ½ and Õ ½ Ö recursively Combine: Subarrays are sorted in-place, nothing is needed here; the entire Ô Ö is sorted the performance of Quicksort depends extremely on how the pivots are selected!

42 SFU CMPT Lecture: Week 3 Ô Öµ Quicksort 1: Ô Ö if then 2: Õ Partition Ô Öµ 3: Quicksort Ô Õ ½µ 4: Quicksort Õ ½ Öµ 5: end if To sort the entire array, call Quicksort ½ length µµ Note: this is a very naive implementation because it recurses down to the very end. Efficient implementations do special things at the bottom of recursion (calls optimal procedures to sort sub-arrays of lengths up to, say, )

43 SFU CMPT Lecture: Week 3 So, a more practical solution would look something like this: Ô Öµ Quicksort 1: Ö Ô ¾ if then 2: Sort2 Ôµ 3: else if Ö Ô then 4: Sort3 Ôµ 5: else if Ö Ô then 6: Sort4 Ôµ 7: else if Ö Ô then 8: Sort5 Ôµ 9: else if Ö Ô then 10: Sort6 Ôµ 11: else if Ö Ô then 12: Sort7 Ôµ 13: else if Ô Ö then 14: Õ Partition Ô Öµ 15: Quicksort Ô Õ ½µ 16: Quicksort Õ ½ Öµ 17: end if

44 SFU CMPT Lecture: Week 3 Partition The most important part of Quicksort. Ô Öµ Partition Ü Ö 1: /* choose a Ü pivot */ 2: Ô ½ 3: for Ô to Ö ½ do 4: if Ü then 5: ½ 6: exchange 7: end if 8: end for 9: exchange ½ Ö 10: return ½

45 SFU CMPT Lecture: Week 3 Ô Öµ Partition Ü Ö 1: /* choose a Ü pivot */ 2: Ô ½ 3: for Ô to Ö ½ do 4: if Ü then 5: ½ 6: exchange 7: end if 8: end for 9: exchange ½ Ö 10: return ½ the loop invariant: at the beginning of each iteration of the loop each element Ô of is smaller than or equal to Ü pivot 2. each element ½ ½ of is greater Ü than 3. Ö ½ elements are so-far arbitrary 4. element Ö Ü

46 SFU CMPT Lecture: Week 3 i j p r i j p r i j p r i j p r i j p r i j p r i j p r i p r

47 SFU CMPT Lecture: Week 3 Correctness: enough to prove first 2 conditions (condition 4 satisfied by line 1) Initialization: before 1st iteration, Ô ½ and Ô; first two conditions are trivial Maintenance: 2 cases: Ü only is increased in the loop; enough to check condition 2 for element ½ Ü is incremented, then and are swapped, then is incremented; enough to check condition 1 for (ok, because of swapping) and condition 2 ½ for (ok, by the loop invariant) Termination: Ö, hence there are no elements in part 3, the others are divided as required finally, to get correct order of these 3 parts, we swap the first element of the second part Ö with running time Òµ, where Ò Ö Ô ½

48 SFU CMPT Lecture: Week 3 This particular implementation of Partition is not good because it always uses the rightmost element as pivot. It s easy to come up with examples that force this implementation to be very bad (meaning overall running time Å Ò ¾ µ). What we want is an even partition of Ô Ö into two sub-arrays of (about) the same size. That s the second very important feature of efficient implementations of Quicksort. There are several not-too-bad ways: look at, say, fixed array elements and pick the median pick a randomly chosen element look at, say randomly chosen elements and pick the median many more In practice, the random variant usually works best (unless you happen to know something about your inputs)