Design and Analysis of Algorithms 演算法設計與分析 Lecture 8 November 6, 206 洪國寶
Outline Review Amortized analysis Advanced data structures Binary heaps Binomial heaps Fibonacci heaps Data structures for disjoint sets 2
Huffman code (Implementation) Time complexity and data structure: Let S be the set of n weights (nodes). Constructing a Huffamn code based on greedy strategy can be described as following Repeat until S = Find two min nodes x and y from S and remove them from S Construct a new node z with weight w(z)=w(x)+w(y) and insert z into S 3
Huffman code (Implementation) Why data structures are important? An algorithm for constructing Huffman code (Tree): Repeat until S = Find two min nodes x and y from S and remove them from S Construct a new node z with weight w(z)=w(x)+w(y) and insert z into S The time complexity of the algorithm depends on how S is implemented. Data structure for S each iteration total linked list O(n) O() O(n 2 ) Sorted array O() O(n) O(n 2 )? O(log n) O(log n) O(n log n) 4
Priority Queues A priority queue is an abstract data type which is like a regular queue or stack data structure, but where additionally each element has a "priority" associated with it. In a priority queue, an element with high priority is served before an element with low priority. Priority queues are often implemented with heaps. Applications. Dijkstra's shortest path algorithm. Prim's MST algorithm. Event-driven simulation. Huffman encoding. (Lecture 6) Heapsort.... 5
Mergeable Heaps Support the following operations. MAKE-HEAP() creates and returns a new heap containing no elements. INSERT(H, x) inserts node x, whose key field has already been filled in, into heap H. MINIMUM(H) returns a pointer to the node in heap H whose key is minimum. EXTRACT-MIN(H) deletes the node from heap H whose key is minimum, returning a pointer to the node. UNION(Hl, H2) creates and returns a new heap that contains all the nodes of heaps H and H2. Heaps H and H2 are "destroyed" by this operation. DECREASE-KEY(H, x, k) assigns to node x within heap H the new key value k, which is assumed to be no greater than its current key value. DELETE(H, x) deletes node x from heap H. 6
Priority Queues Operation Linked List Mergeable Heaps Binary Binomial Fibonacci * make-heap insert log N log N minimum N log N extract-min N log N log N log N union N log N decrease-key log N log N delete N log N log N log N Dijkstra/Prim make-heap V insert V extract-min E decrease-key O( V 2 ) O( E log V ) O( E + V log V ) 7
Binary Heap: Definition Binary heap. Almost complete binary tree. filled on all levels, except last, where filled from left to right Min-heap ordered. every child greater than (or equal to) parent 06 4 45 78 8 47 53 83 9 8 77 84 99 64 8
Properties. Binary Heap: Properties Min element is in root. Heap with N elements has height = log 2 N. 4 06 45 N = 4 Height = 3 78 8 47 53 83 9 8 77 84 99 64 9
Binary Heaps: Array Implementation Implementing binary heaps. Use an array: no need for explicit parent or child pointers. Parent(i) = i/2 Left(i) = 2i Right(i) = 2i + 4 06 45 2 3 78 8 47 53 4 5 6 7 83 9 8 77 84 99 64 8 9 0 2 3 4 0
Binary heap operations Make-heap make-heap( ) allocate an array H on size N heap-size[h] 0 Minimum minimum(h) return H[]
Inserting into a binary heap There are two constraints that must be met an almost complete tree the heap-order property We increase the size of the heap to make a hole in the next spot It the item can be legally inserted in the hole, then do so, otherwise move the parent value down and try again 2
Binary Heap: Insertion example Insert element x (key = 42) into heap. Insert into next available slot. Bubble up until it's heap ordered. 06 4 45 78 8 47 53 83 9 8 77 84 99 64 42 next free slot 3
Binary Heap: Insertion example Insert element x (key = 42) into heap. Insert into next available slot. Bubble up until it's heap ordered. 06 swap with parent 4 45 78 8 47 53 83 9 8 77 84 99 64 42 4
Binary Heap: Insertion example Insert element x (key = 42) into heap. Insert into next available slot. Bubble up until it's heap ordered. 06 swap with parent 4 45 78 8 47 42 83 9 8 77 84 99 64 42 53 5
Binary Heap: Insertion example Insert element x (key = 42) into heap. Insert into next available slot. Bubble up until it's heap ordered. 06 stop: heap ordered 4 42 78 8 47 45 83 9 8 77 84 99 64 53 6
Binary Heap: Insertion algorithm insert(h,k) heap-size[h] heap-size[h]+ i heap-size[h] while i > and H[parent(i)] > k do H[i] H[parent(i)] i parent(i) H[i] k Time complexity = O(tree height) = O(log n) 7
Binary Heap: Decrease Key Decrease key of element x to k. Bubble up until it's heap ordered. 06 4 42 78 8 47 45 83 9 8 77 84 99 64 53 8
Binary Heap: Decrease Key decrease-key(h,i,k) while i > and H[parent(i)] > k do H[i] H[parent(i)] i parent(i) H[i] k Time complexity = O(tree height) = O(log n) 9
Binary Heap: Extract-Min Delete minimum element from heap. Exchange root with rightmost leaf. Bubble root down until it's heap ordered. power struggle principle: better subordinate is promoted 06 4 42 78 8 47 45 83 9 8 77 84 99 64 53 20
Binary Heap: Extract-Min Delete minimum element from heap. Exchange root with rightmost leaf. Bubble root down until it's heap ordered. power struggle principle: better subordinate is promoted 53 4 42 78 8 47 45 83 9 8 77 84 99 64 06 2
Binary Heap: Extract-Min Delete minimum element from heap. Exchange root with rightmost leaf. Bubble root down until it's heap ordered. power struggle principle: better subordinate is promoted 53 exchange with left child 4 42 78 8 47 45 83 9 8 77 84 99 64 22
Binary Heap: Extract-Min Delete minimum element from heap. Exchange root with rightmost leaf. Bubble root down until it's heap ordered. power struggle principle: better subordinate is promoted 4 exchange with right child 53 42 78 8 47 45 83 9 8 77 84 99 64 23
Binary Heap: Extract-Min Delete minimum element from heap. Exchange root with rightmost leaf. Bubble root down until it's heap ordered. power struggle principle: better subordinate is promoted Time complexity = O(tree height) = O(log n) 4 stop: heap ordered 8 42 78 53 47 45 83 9 8 77 84 99 64 24
Binary Heap: Extract-Min extract-min(h) if heap-size[h] < then error heap underflow min H[] H[] H[heap-size(H)] heap-size[h] heap-size[h] - heapify(h, ) return min Bubble root down until it's heap ordered. 25
Binary Heap: Heapify When heapify(h,i) is called, it is assumed that the binary trees rooted at left(i) and right(i) are heaps, but that A[i] may be larger than its children, thus violating the heap property. The function of heapify is to let the value at A[i] float down in the heap so that the subtree rooted at index i becomes a heap. The running time is O(h) if the height of node i is h. 26
Binary Heap: Heapify heapify(h,i) l left(i) r right(i) if l heap-size[h] and H[l] < H[i] then smallest l else smallest i if r heap-size[h] and H[r] < H[smallest] then smallest r if smallest i then H[i] H[smallest] heapify(h,smallest) 27
Binary Heap: Delete delete(h,i) H[i] H[heap-size[H]] heap-size[h] heap-size[h] heapify(h,i) 28
Union. Binary Heap: Union Combine two binary heaps H and H 2 into a single heap. No easy solution. (N) operations apparently required Can support fast union with fancier heaps. H H 2 4 78 8 53 62 4 9 8 77 84 99 64 29
Binary Heap: Union Union(H,H2) H make-heap() heap-size[h] heap-size[h] + heapsize[h2] for i to heap-size[h] do H[i] H[i] for i to heap-size[h2] do H[heap-size[H]+i] H2[i] for i heap-size[h]/2 down to do heapify(h,i) 30
Binary Heap: Union H 4 4 78 8 78 8 4 9 8 77 4 9 8 77 H 2 53 62 84 99 64 53 62 84 99 64 3
Binary Heap: Union Proof of correctness by loop invariant At the start of each iteration, each node i+, i+2,,n is the root of a heap Time complexity Simple upper bound: each call to HEAPIFY: O(lg n) O(n) such calls running time at most O(n lg n) A tighter bound is O(n). 32
Binary Heap: Union Time complexity A tighter bound is O(n). (Use blackboard) 33
Binary Heap: Heapsort To sort N items: Insert N items into a binary max-heap. Perform N extract-max operations. O(N log N) sort. No extra storage (in-place sort). Which one is better? Merge sort or heap sort? 34
Build-Max-Heap Operation We start by recognizing that if each subtree of a node is a heap, then when we connect the two subtrees, we can obtain a new heap by finding where the root value should go All leaf nodes are, by definition, heaps 35
36
37
Example: The action of max-heapify(a,2) 38
39
Example: The operation of build-max-heap 40
Running Time of BUILD-MAX-HEAP Simple upper bound: each call to MAX-HEAPIFY: O(lg n) O(n) such calls running time at most O(n lg n) A tighter bound is O(n). 4
42
43
Running Time of HEAPSORT Upper bound: Build max-heap: O(n) each call to MAX-HEAPIFY: O(log n) O(n) such calls running time O(n log n) 44
Some sorting algorithms Insertion sort Merge sort Heapsort Quicksort complexity O(n 2 ) O(n log n) O(n log n) O(n log n) In-place sort Data structure Remark yes no yes Linked list On-line sorting Linked list Array 45
Sorting algorithms The ideal sorting algorithm would have the following properties: Stable: Equal keys aren't reordered. Operates in place, requiring O() extra space. Worst-case O(n lg(n)) key comparisons. Worst-case O(n) swaps. Adaptive: Speeds up to O(n) when data is nearly sorted or when there are few unique keys. There is no algorithm that has all of these properties, and so the choice of the optimal sorting algorithm depends on the application. 46
Animated Sorting Algorithms http://www.sorting-algorithms.com/ 47
Priority Queues Operation make-heap insert minimum extract-min union decrease-key delete Linked List N N N Mergeable Heaps Binary Binomial Fibonacci * log N log N N log N log N log N log N log N log N log N log N log N log N Coming up next 48
Binomial Trees Binomial tree. Recursive definition: B 0 B k B k- B k- B 0 B B 2 B 3 B 4 49
Binomial Trees Useful properties of order k binomial tree B k. Number of nodes = 2 k. Height = k. Degree of root = k. Deleting root yields binomial trees B k-,, B 0. B k+ Proof. By induction on k. B k B B 2 B 0 B 0 B B 2 B 3 B 4 50
Binomial Trees A property useful for naming the data structure. k B k has i nodes at depth i. depth 0 depth 4 2 6 depth 2 depth 3 depth 4 B 4 5
Binomial Heap Binomial heap. Vuillemin, 978. Sequence of binomial trees that satisfy binomial heap property. each tree is min-heap ordered 0 or binomial tree of order k 6 3 8 8 29 0 44 37 30 23 22 48 3 7 45 32 24 50 55 B 4 B B 0 52
53
Binomial Heap: Properties Properties of N-node binomial heap. Min key contained in root of B 0, B,..., B k. (root list) Contains binomial tree B i iff b i = where b n b 2 b b 0 is binary representation of N. At most log 2 N + binomial trees. Height log 2 N. 6 3 8 8 29 0 44 37 30 45 32 23 22 24 48 3 7 50 N = 9 # trees = 3 height = 4 binary = 00 55 B 4 B B 0 54
Binomial Heap Operation Create a binomial heap make-binomial-heap( ) H allocate-node( ) head[h] nil return H Running time = O() 55
Binomial Heap Operation 2 Minimum Running time = O(log n) 56
Binomial Heap Operation 3: Union Create heap H that is union of heaps H' and H''. "Mergeable heaps." Easy if H' and H'' are each order k binomial trees. connect roots of H' and H'' choose smaller key to be root of H (Binomial-link) 6 8 29 0 44 30 23 22 48 3 7 45 32 24 50 55 H' H'' 57
Binomial Heap: Union 58
Binomial Heap: Union 6 3 8 8 29 0 44 37 30 23 22 48 3 7 5 7 2 45 32 24 50 + 55 28 4 33 25 0 0 9 + 7 = 26 + 0 0 0 0 59
Binomial Heap: Union 6 3 8 8 29 0 44 37 30 23 22 48 3 7 5 7 2 45 32 24 50 + 55 28 4 33 25 60
2 Binomial Heap: Union 8 6 3 8 8 29 0 44 37 30 23 22 48 3 7 5 7 2 45 32 24 50 + 55 28 4 33 25 6
3 2 7 37 8 25 6 3 8 8 29 0 44 37 30 23 22 48 3 7 5 7 2 45 32 24 50 + 55 28 4 33 25 2 8 62
3 3 2 5 7 37 7 37 8 28 33 25 25 4 6 3 8 8 29 0 44 37 30 23 22 48 3 7 5 7 2 45 32 24 50 + 55 28 4 33 25 2 8 63
3 3 2 5 7 37 7 37 8 28 33 25 25 4 6 3 8 8 29 0 44 37 30 23 22 48 3 7 5 7 2 45 32 24 50 + 55 28 4 33 25 3 2 5 7 37 8 28 33 25 4 64
3 3 2 5 7 37 7 37 8 28 33 25 25 4 6 3 8 8 29 0 44 37 30 23 22 48 3 7 5 7 2 45 32 24 50 + 55 28 4 33 25 6 3 2 8 29 0 44 5 7 37 8 30 23 22 48 3 7 28 33 25 45 32 24 50 4 65 55
66
67
Binomial Heap Operation 3: Union Create heap H that is union of heaps H' and H''. Analogous to binary addition. Running time = O(log N) Proportional to number of trees in root lists 2( log 2 N + ). 0 0 9 + 7 = 26 + 0 0 0 0 68
Binomial Heap Operation 3: Union Running time = O(log N) Binomial-heap-merge: merge two sorted linked lists Proportional to number of trees in root lists 2( log 2 N + ). while loop (line 9- line 2) Each iteration executes one binomial-link O() Proportional to number of trees in root lists 2( log 2 N + ). 0 0 9 + 7 = 26 + 0 0 0 0 69
Binomial Heap Operation 3: Extract-min Delete node with minimum key in binomial heap H. Find root x with min key in root list of H, and delete H' broken binomial trees H Union(H', H) Running time. O(log N) 6 8 8 29 0 44 37 30 23 22 48 3 7 H 45 32 24 50 55 H' 70
7
72
Binomial Heap: Extract-min Running time. O(log N). Find the root with minimum key from root list: O(# of trees) = O(log n) 2. Make-binomial-heap: O() 3. Reverse the order of the linked list of x s children: O(degree(x)) = O(log n) 4. Binomial-heap-union: O(log n) 6 8 8 29 0 44 37 30 23 22 48 3 7 H 45 32 24 50 55 H' 73
Binomial Heap Operation 5: Insert Insert a new node x into binomial heap H. H' MakeHeap(x) H Union(H', H) Running time. O(log N) 3 6 8 x 8 29 0 44 37 30 23 22 48 3 7 H H' 45 32 24 50 55 74
Binomial Heap Operation 5: Insert 75
Binomial Heap: Sequence of Inserts Insert a new node x into binomial heap H. If N =...0, then only steps. If N =...0, then only 2 steps. If N =...0, then only 3 steps. If N =...0, then only 4 steps. 29 0 3 44 6 x 37 Inserting item can take (log N) time. If N =..., then log 2 N steps. 48 3 7 50 But, inserting sequence of N items takes O(N) time! (N/2)() + (N/4)(2) + (N/8)(3) +... 2N Amortized analysis. Basis for getting most operations down to constant time. N n n n 2 2 2 N 2 N 2 N 76
Binomial Heap Operation 6: Decrease Key Decrease key of node x in binomial heap H. Suppose x is in binomial tree B k. Bubble node x up the tree if x is too small. Running time. O(log N) Proportional to depth of node x log 2 N. 3 6 8 depth = 3 8 29 0 44 37 30 23 22 48 3 7 H x 32 24 50 55 77
78
79
Binomial Heap Operation 7: Delete Delete node x in binomial heap H. Decrease key of x to -. Delete min. Running time. O(log N) 80
Binomial Heap Operation 7: Delete 8
Questions? 82