# CMSC 451 Design and Analysis of Computer Algorithms 1

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 CMSC 4 Design and Analysis of Computer Algorithms David M. Mount Department of Computer Science University of Maryland Fall 003 Copyright, David M. Mount, 004, Dept. of Computer Science, University of Maryland, College Park, MD, 074. These lecture notes were prepared by David Mount for the course CMSC 4, Design and Analysis of Computer Algorithms, at the University of Maryland. Permission to use, copy, modify, and distribute these notes for educational purposes and without fee is hereby granted, provided that this copyright notice appear in all copies. Lecture Notes CMSC 4

2 Lecture : Course Introduction Read: (All readings are from Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, nd Edition). Review Chapts. in CLRS. What is an algorithm? Our text defines an algorithm to be any well-defined computational procedure that takes some values as input and produces some values as output. Like a cooking recipe, an algorithm provides a step-by-step method for solving a computational problem. Unlike programs, algorithms are not dependent on a particular programming language, machine, system, or compiler. They are mathematical entities, which can be thought of as running on some sort of idealized computer with an infinite random access memory and an unlimited word size. Algorithm design is all about the mathematical theory behind the design of good programs. Why study algorithm design? Programming is a very complex task, and there are a number of aspects of programming that make it so complex. The first is that most programming projects are very large, requiring the coordinated efforts of many people. (This is the topic a course like software engineering.) The next is that many programming projects involve storing and accessing large quantities of data efficiently. (This is the topic of courses on data structures and databases.) The last is that many programming projects involve solving complex computational problems, for which simplistic or naive solutions may not be efficient enough. The complex problems may involve numerical data (the subject of courses on numerical analysis), but often they involve discrete data. This is where the topic of algorithm design and analysis is important. Although the algorithms discussed in this course will often represent only a tiny fraction of the code that is generated in a large software system, this small fraction may be very important for the success of the overall project. An unfortunately common approach to this problem is to first design an inefficient algorithm and data structure to solve the problem, and then take this poor design and attempt to fine-tune its performance. The problem is that if the underlying design is bad, then often no amount of fine-tuning is going to make a substantial difference. The focus of this course is on how to design good algorithms, and how to analyze their efficiency. This is among the most basic aspects of good programming. Course Overview: This course will consist of a number of major sections. The first will be a short review of some preliminary material, including asymptotics, summations, and recurrences and sorting. These have been covered in earlier courses, and so we will breeze through them pretty quickly. We will then discuss approaches to designing optimization algorithms, including dynamic programming and greedy algorithms. The next major focus will be on graph algorithms. This will include a review of breadth-first and depth-first search and their application in various problems related to connectivity in graphs. Next we will discuss minimum spanning trees, shortest paths, and network flows. We will briefly discuss algorithmic problems arising from geometric settings, that is, computational geometry. Most of the emphasis of the first portion of the course will be on problems that can be solved efficiently, in the latter portion we will discuss intractability and NP-hard problems. These are problems for which no efficient solution is known. Finally, we will discuss methods to approximate NP-hard problems, and how to prove how close these approximations are to the optimal solutions. Issues in Algorithm Design: Algorithms are mathematical objects (in contrast to the must more concrete notion of a computer program implemented in some programming language and executing on some machine). As such, we can reason about the properties of algorithms mathematically. When designing an algorithm there are two fundamental issues to be considered: correctness and efficiency. It is important to justify an algorithm s correctness mathematically. For very complex algorithms, this typically requires a careful mathematical proof, which may require the proof of many lemmas and properties of the solution, upon which the algorithm relies. For simple algorithms (BubbleSort, for example) a short intuitive explanation of the algorithm s basic invariants is sufficient. (For example, in BubbleSort, the principal invariant is that on completion of the ith iteration, the last i elements are in their proper sorted positions.) Lecture Notes CMSC 4

3 Establishing efficiency is a much more complex endeavor. Intuitively, an algorithm s efficiency is a function of the amount of computational resources it requires, measured typically as execution time and the amount of space, or memory, that the algorithm uses. The amount of computational resources can be a complex function of the size and structure of the input set. In order to reduce matters to their simplest form, it is common to consider efficiency as a function of input size. Among all inputs of the same size, we consider the maximum possible running time. This is called worst-case analysis. It is also possible, and often more meaningful, to measure average-case analysis. Average-case analyses tend to be more complex, and may require that some probability distribution be defined on the set of inputs. To keep matters simple, we will usually focus on worst-case analysis in this course. Throughout out this course, when you are asked to present an algorithm, this means that you need to do three things: Present a clear, simple and unambiguous description of the algorithm (in pseudo-code, for example). They key here is keep it simple. Uninteresting details should be kept to a minimum, so that the key computational issues stand out. (For example, it is not necessary to declare variables whose purpose is obvious, and it is often simpler and clearer to simply say, Add X to the end of list L than to present code to do this or use some arcane syntax, such as L.insertAtEnd(X). ) Present a justification or proof of the algorithm s correctness. Your justification should assume that the reader is someone of similar background as yourself, say another student in this class, and should be convincing enough make a skeptic believe that your algorithm does indeed solve the problem correctly. Avoid rambling about obvious or trivial elements. A good proof provides an overview of what the algorithm does, and then focuses on any tricky elements that may not be obvious. Present a worst-case analysis of the algorithms efficiency, typically it running time (but also its space, if space is an issue). Sometimes this is straightforward, but if not, concentrate on the parts of the analysis that are not obvious. Note that the presentation does not need to be in this order. Often it is good to begin with an explanation of how you derived the algorithm, emphasizing particular elements of the design that establish its correctness and efficiency. Then, once this groundwork has been laid down, present the algorithm itself. If this seems to be a bit abstract now, don t worry. We will see many examples of this process throughout the semester. Lecture : Mathematical Background Read: Review Chapters in CLRS. Algorithm Analysis: Today we will review some of the basic elements of algorithm analysis, which were covered in previous courses. These include asymptotics, summations, and recurrences. Asymptotics: Asymptotics involves O-notation ( big-oh ) and its many relatives, Ω, Θ, o ( little-oh ), ω. Asymptotic notation provides us with a way to simplify the functions that arise in analyzing algorithm running times by ignoring constant factors and concentrating on the trends for large values of n. For example, it allows us to reason that for three algorithms with the respective running times n 3 log n +4n +nlog n Θ(n 3 log n) n +7nlog 3 n Θ(n ) 3n + 4 log n +9n Θ(n ). Thus, the first algorithm is significantly slower for large n, while the other two are comparable, up to a constant factor. Since asymptotics were covered in earlier courses, I will assume that this is familiar to you. Nonetheless, here are a few facts to remember about asymptotic notation: Lecture Notes 3 CMSC 4

4 Ignore constant factors: Multiplicative constant factors are ignored. For example, 347n is Θ(n). Constant factors appearing exponents cannot be ignored. For example, 3n is not O( n ). Focus on large n: Asymptotic analysis means that we consider trends for large values of n. Thus, the fastest growing function of n is the only one that needs to be considered. For example, 3n log n +nlog n + (log n) 7 is Θ(n log n). Polylog, polynomial, and exponential: These are the most common functions that arise in analyzing algorithms: Polylogarithmic: Powers of log n, such as (log n) 7. We will usually write this as log 7 n. Polynomial: Powers of n, such as n 4 and n = n /. Exponential: A constant (not ) raised to the power n, such as 3 n. An important fact is that polylogarithmic functions are strictly asymptotically smaller than polynomial function, which are strictly asymptotically smaller than exponential functions (assuming the base of the exponent is bigger than ). For example, if we let mean asymptotically smaller then log a n n b c n for any a, b, and c, provided that b>0and c>. Logarithm Simplification: It is a good idea to first simplify terms involving logarithms. For example, the following formulas are useful. Here a, b, c are constants: log b n = log a n log a b = Θ(log a n) log a (n c ) = clog a n = Θ(log a n) b log a n = n log a b. Avoid using log n in exponents. The last rule above can be used to achieve this. For example, rather than saying 3 log n, express this as n log 3 n.8. Following the conventional sloppiness, I will often say O(n ), when in fact the stronger statement Θ(n ) holds. (This is just because it is easier to say oh than theta.) Summations: Summations naturally arise in the analysis of iterative algorithms. Also, more complex forms of analysis, such as recurrences, are often solved by reducing them to summations. Solving a summation means reducing it to a closed form formula, that is, one having no summations, recurrences, integrals, or other complex operators. In algorithm design it is often not necessary to solve a summation exactly, since an asymptotic approximation or close upper bound is usually good enough. Here are some common summations and some tips to use in solving summations. Constant Series: For integers a and b, b = max(b a +,0). i=a Notice that when b = a, there are no terms in the summation (since the index is assumed to count upwards only), and the result is 0. Be careful to check that b a before applying this formula blindly. Arithmetic Series: For n 0, n i=0 i =++ +n= n(n+). This is Θ(n ). (The starting bound could have just as easily been set to as 0.) Lecture Notes 4 CMSC 4

5 Geometric Series: Let x be any constant (independent of n), then for n 0, n i=0 x i =+x+x + +x n = xn+ x. If 0 <x<then this is Θ(). Ifx>, then this is Θ(x n ), that is, the entire sum is proportional to the last element of the series. Quadratic Series: For n 0, n i=0 i = + + +n = n3 +3n +n. 6 Linear-geometric Series: This arises in some algorithms based on trees and recursion. Let x be any constant, then for n 0, n ix i = x +x +3x 3 +nx n = (n )x(n+) nx n + x (x ). i=0 As n becomes large, this is asymptotically dominated by the term (n )x (n+) /(x ). The multiplicative term n is very nearly equal to n for large n, and, since x is a constant, we may multiply this times the constant (x ) /x without changing the asymptotics. What remains is Θ(nx n ). Harmonic Series: This arises often in probabilistic analyses of algorithms. It does not have an exact closed form solution, but it can be closely approximated. For n 0, H n = n i= i = n = (ln n)+o(). There are also a few tips to learn about solving summations. Summations with general bounds: When a summation does not start at the or 0, as most of the above formulas assume, you can just split it up into the difference of two summations. For example, for a b b f(i) = i=a b i=0 a f(i) f(i). i=0 Linearity of Summation: Constant factors and added terms can be split out to make summations simpler. (4 + 3i(i )) = 4+3i 6i= 4+3 i 6 i. Now the formulas can be to each summation individually. Approximate using integrals: Integration and summation are closely related. (Integration is in some sense a continuous form of summation.) Here is a handy formula. Let f(x) be any monotonically increasing function (the function increases as x increases). n 0 f(x)dx n f(i) i= n+ f(x)dx. Example: Right Dominant Elements As an example of the use of summations in algorithm analysis, consider the following simple problem. We are given a list L of numeric values. We say that an element of L is right dominant if it is strictly larger than all the elements that follow it in the list. Note that the last element of the list Lecture Notes CMSC 4

6 is always right dominant, as is the last occurrence of the maximum element of the array. For example, consider the following list. L = 0, 9,, 3,, 7,, 8, 4, 6, 3 The sequence of right dominant elements are 3, 8, 6, 3. In order to make this more concrete, we should think about how L is represented. It will make a difference whether L is represented as an array (allowing for random access), a doubly linked list (allowing for sequential access in both directions), or a singly linked list (allowing for sequential access in only one direction). Among the three possible representations, the array representation seems to yield the simplest and clearest algorithm. However, we will design the algorithm in such a way that it only performs sequential scans, so it could also be implemented using a singly linked or doubly linked list. (This is common in algorithms. Chose your representation to make the algorithm as simple and clear as possible, but give thought to how it may actually be implemented. Remember that algorithms are read by humans, not compilers.) We will assume here that the array L of size n is indexed from to n. Think for a moment how you would solve this problem. Can you see an O(n) time algorithm? (If not, think a little harder.) To illustrate summations, we will first present a naive O(n ) time algorithm, which operates by simply checking for each element of the array whether all the subsequent elements are strictly smaller. (Although this example is pretty stupid, it will also serve to illustrate the sort of style that we will use in presenting algorithms.) Right Dominant Elements (Naive Solution) // Input: List L of numbers given as an array L[..n] // Returns: List D containing the right dominant elements of L RightDominant(L) { D = empty list for (i = to n) isdominant = true for (j = i+ to n) if (A[i] <= A[j]) isdominant = false if (isdominant) append A[i] to D return D If I were programming this, I would rewrite the inner (j) loop as a while loop, since we can terminate the loop as soon as we find that A[i] is not dominant. Again, this sort of optimization is good to keep in mind in programming, but will be omitted since it will not affect the worst-case running time. The time spent in this algorithm is dominated (no pun intended) by the time spent in the inner (j) loop. On the ith iteration of the outer loop, the inner loop is executed from i +to n, for a total of n (i +)+=n i times. (Recall the rule for the constant series above.) Each iteration of the inner loop takes constant time. Thus, up to a constant factor, the running time, as a function of n, is given by the following summation: T (n) = n (n i). To solve this summation, let us expand it, and put it into a form such that the above formulas can be used. i= T (n) = (n ) + (n ) = (n ) + (n ) = n i = i=0 (n )n. Lecture Notes 6 CMSC 4

7 The last step comes from applying the formula for the linear series (using n in place of n in the formula). As mentioned above, there is a simple O(n) time algorithm for this problem. As an exercise, see if you can find it. As an additional challenge, see if you can design your algorithm so it only performs a single left-to-right scan of the list L. (You are allowed to use up to O(n) working storage to do this.) Recurrences: Another useful mathematical tool in algorithm analysis will be recurrences. They arise naturally in the analysis of divide-and-conquer algorithms. Recall that these algorithms have the following general structure. Divide: Divide the problem into two or more subproblems (ideally of roughly equal sizes), Conquer: Solve each subproblem recursively, and Combine: Combine the solutions to the subproblems into a single global solution. How do we analyze recursive procedures like this one? If there is a simple pattern to the sizes of the recursive calls, then the best way is usually by setting up a recurrence, that is, a function which is defined recursively in terms of itself. Here is a typical example. Suppose that we break the problem into two subproblems, each of size roughly n/. (We will assume exactly n/ for simplicity.). The additional overhead of splitting and merging the solutions is O(n). When the subproblems are reduced to size, we can solve them in O() time. We will ignore constant factors, writing O(n) just as n, yielding the following recurrence: T (n) = if n =, T(n) = T(n/) + n if n>. Note that, since we assume that n is an integer, this recurrence is not well defined unless n is a power of (since otherwise n/ will at some point be a fraction). To be formally correct, I should either write n/ or restrict the domain of n, but I will often be sloppy in this way. There are a number of methods for solving the sort of recurrences that show up in divide-and-conquer algorithms. The easiest method is to apply the Master Theorem, given in CLRS. Here is a slightly more restrictive version, but adequate for a lot of instances. See CLRS for the more complete version of the Master Theorem and its proof. Theorem: (Simplified Master Theorem) Let a, b>be constants and let T (n) be the recurrence defined for n 0. Case : a>b k then T (n) is Θ(n log b a ). Case : a = b k then T (n) is Θ(n k log n). Case 3: a<b k then T (n) is Θ(n k ). T (n) =at (n/b)+cn k, Using this version of the Master Theorem we can see that in our recurrence a =,b=, and k =,soa=b k and Case applies. Thus T (n) is Θ(n log n). There many recurrences that cannot be put into this form. For example, the following recurrence is quite common: T (n) =T(n/) + n log n. This solves to T (n) =Θ(nlog n), but the Master Theorem (either this form or the one in CLRS will not tell you this.) For such recurrences, other methods are needed. Lecture 3: Review of Sorting and Selection Read: Review Chapts. 6 9 in CLRS. Lecture Notes 7 CMSC 4

8 Review of Sorting: Sorting is among the most basic problems in algorithm design. We are given a sequence of items, each associated with a given key value. The problem is to permute the items so that they are in increasing (or decreasing) order by key. Sorting is important because it is often the first step in more complex algorithms. Sorting algorithms are usually divided into two classes, internal sorting algorithms, which assume that data is stored in an array in main memory, and external sorting algorithm, which assume that data is stored on disk or some other device that is best accessed sequentially. We will only consider internal sorting. You are probably familiar with one or more of the standard simple Θ(n ) sorting algorithms, such as Insertion- Sort, SelectionSort and BubbleSort. (By the way, these algorithms are quite acceptable for small lists of, say, fewer than 0 elements.) BubbleSort is the easiest one to remember, but it widely considered to be the worst of the three. The three canonical efficient comparison-based sorting algorithms are MergeSort, QuickSort, and HeapSort. All run in Θ(n log n) time. Sorting algorithms often have additional properties that are of interest, depending on the application. Here are two important properties. In-place: The algorithm uses no additional array storage, and hence (other than perhaps the system s recursion stack) it is possible to sort very large lists without the need to allocate additional working storage. Stable: A sorting algorithm is stable if two elements that are equal remain in the same relative position after sorting is completed. This is of interest, since in some sorting applications you sort first on one key and then on another. It is nice to know that two items that are equal on the second key, remain sorted on the first key. Here is a quick summary of the fast sorting algorithms. If you are not familiar with any of these, check out the descriptions in CLRS. They are shown schematically in Fig. QuickSort: It works recursively, by first selecting a random pivot value from the array. Then it partitions the array into elements that are less than and greater than the pivot. Then it recursively sorts each part. QuickSort is widely regarded as the fastest of the fast sorting algorithms (on modern machines). One explanation is that its inner loop compares elements against a single pivot value, which can be stored in a register for fast access. The other algorithms compare two elements in the array. This is considered an in-place sorting algorithm, since it uses no other array storage. (It does implicitly use the system s recursion stack, but this is usually not counted.) It is not stable. There is a stable version of QuickSort, but it is not in-place. This algorithm is Θ(n log n) in the expected case, and Θ(n ) in the worst case. If properly implemented, the probability that the algorithm takes asymptotically longer (assuming that the pivot is chosen randomly) is extremely small for large n. x QuickSort: x partition < x x > x sort sort MergeSort: split merge sort HeapSort: buildheap extractmax Heap Fig. : Common O(n log n) comparison-based sorting algorithms. Lecture Notes 8 CMSC 4

10 is, up to constant factors, roughly (n/e) n. Plugging this in and simplifying yields the Ω(n log n) lower bound. This can also be generalized to show that the average-case time to sort is also Ω(n log n). Linear Time Sorting: The Ω(n log n) lower bound implies that if we hope to sort numbers faster than in O(n log n) time, we cannot do it by making comparisons alone. In some special cases, it is possible to sort without the use of comparisons. This leads to the possibility of sorting in linear (that is, O(n)) time. Here are three such algorithms. Counting Sort: Counting sort assumes that each input is an integer in the range from to k. The algorithm sorts in Θ(n + k) time. Thus, if k is O(n), this implies that the resulting sorting algorithm runs in Θ(n) time. The algorithm requires an additional Θ(n + k) working storage but has the nice feature that it is stable. The algorithm is remarkably simple, but deceptively clever. You are referred to CLRS for the details. Radix Sort: The main shortcoming of CountingSort is that (due to space requirements) it is only practical for a very small ranges of integers. If the integers are in the range from say, to a million, we may not want to allocate an array of a million elements. RadixSort provides a nice way around this by sorting numbers one digit, or one byte, or generally, some groups of bits, at a time. As the number of bits in each group increases, the algorithm is faster, but the space requirements go up. The idea is very simple. Let s think of our list as being composed of n integers, each having d decimal digits (or digits in any base). To sort these integers we simply sort repeatedly, starting at the lowest order digit, and finishing with the highest order digit. Since the sorting algorithm is stable, we know that if the numbers are already sorted with respect to low order digits, and then later we sort with respect to high order digits, numbers having the same high order digit will remain sorted with respect to their low order digit. An example is shown in Figure. Input Output 76 49[4] 9[]4 [] [4] [7]6 [] [4] [7]6 [] = 7[6] = [7]8 = []96 = [6] 4[9]4 [4] [6] [9]4 [] [8] [9]6 [9]4 94 Fig. : Example of RadixSort. The running time is Θ(d(n + k)) where d is the number of digits in each value, n is the length of the list, and k is the number of distinct values each digit may have. The space needed is Θ(n + k). A common application of this algorithm is for sorting integers over some range that is larger than n, but still polynomial in n. For example, suppose that you wanted to sort a list of integers in the range from to n. First, you could subtract so that they are now in the range from 0 to n. Observe that any number in this range can be expressed as -digit number, where each digit is over the range from 0 to n. In particular, given any integer L in this range, we can write L = an + b, where a = L/n and b = L mod n. Now, we can think of L as the -digit number (a, b). So, we can radix sort these numbers in time Θ((n + n)) = Θ(n). In general this works to sort any n numbers over the range from to n d,in Θ(dn) time. BucketSort: CountingSort and RadixSort are only good for sorting small integers, or at least objects (like characters) that can be encoded as small integers. What if you want to sort a set of floating-point numbers? In the worst-case you are pretty much stuck with using one of the comparison-based sorting algorithms, such as QuickSort, MergeSort, or HeapSort. However, in special cases where you have reason to believe that your numbers are roughly uniformly distributed over some range, then it is possible to do better. (Note Lecture Notes 0 CMSC 4

11 that this is a strong assumption. This algorithm should not be applied unless you have good reason to believe that this is the case.) Suppose that the numbers to be sorted range over some interval, say [0, ). (It is possible in O(n) time to find the maximum and minimum values, and scale the numbers to fit into this range.) The idea is the subdivide this interval into n subintervals. For example, if n = 00, the subintervals would be [0, 0.0), [0.0, 0.0), [0.0, 0.03), and so on. We create n different buckets, one for each interval. Then we make a pass through the list to be sorted, and using the floor function, we can map each value to its bucket index. (In this case, the index of element x would be 00x.) We then sort each bucket in ascending order. The number of points per bucket should be fairly small, so even a quadratic time sorting algorithm (e.g. BubbleSort or InsertionSort) should work. Finally, all the sorted buckets are concatenated together. The analysis relies on the fact that, assuming that the numbers are uniformly distributed, the number of elements lying within each bucket on average is a constant. Thus, the expected time needed to sort each bucket is O(). Since there are n buckets, the total sorting time is Θ(n). An example illustrating this idea is given in Fig. 3. B A Fig. 3: BucketSort. Lecture 4: Dynamic Programming: Longest Common Subsequence Read: Introduction to Chapt, and Section.4 in CLRS. Dynamic Programming: We begin discussion of an important algorithm design technique, called dynamic programming (or DP for short). The technique is among the most powerful for designing algorithms for optimization problems. (This is true for two reasons. Dynamic programming solutions are based on a few common elements. Dynamic programming problems are typically optimization problems (find the minimum or maximum cost solution, subject to various constraints). The technique is related to divide-and-conquer, in the sense that it breaks problems down into smaller problems that it solves recursively. However, because of the somewhat different nature of dynamic programming problems, standard divide-and-conquer solutions are not usually efficient. The basic elements that characterize a dynamic programming algorithm are: Substructure: Decompose your problem into smaller (and hopefully simpler) subproblems. Express the solution of the original problem in terms of solutions for smaller problems. Table-structure: Store the answers to the subproblems in a table. This is done because subproblem solutions are reused many times. Bottom-up computation: Combine solutions on smaller subproblems to solve larger subproblems. (Our text also discusses a top-down alternative, called memoization.) Lecture Notes CMSC 4

12 The most important question in designing a DP solution to a problem is how to set up the subproblem structure. This is called the formulation of the problem. Dynamic programming is not applicable to all optimization problems. There are two important elements that a problem must have in order for DP to be applicable. Optimal substructure: (Sometimes called the principle of optimality.) It states that for the global problem to be solved optimally, each subproblem should be solved optimally. (Not all optimization problems satisfy this. Sometimes it is better to lose a little on one subproblem in order to make a big gain on another.) Polynomially many subproblems: An important aspect to the efficiency of DP is that the total number of subproblems to be solved should be at most a polynomial number. Strings: One important area of algorithm design is the study of algorithms for character strings. There are a number of important problems here. Among the most important has to do with efficiently searching for a substring or generally a pattern in large piece of text. (This is what text editors and programs like grep do when you perform a search.) In many instances you do not want to find a piece of text exactly, but rather something that is similar. This arises for example in genetics research and in document retrieval on the web. One common method of measuring the degree of similarity between two strings is to compute their longest common subsequence. Longest Common Subsequence: Let us think of character strings as sequences of characters. Given two sequences X = x,x,...,x m and Z = z,z,...,z k, we say that Z is a subsequence of X if there is a strictly increasing sequence of k indices i,i,...,i k ( i <i <...<i k n) such that Z = X i,x i,...,x ik. For example, let X = ABRACADABRA and let Z = AADAA, then Z is a subsequence of X. Given two strings X and Y, the longest common subsequence of X and Y is a longest sequence Z that is a subsequence of both X and Y. For example, let X = ABRACADABRA and let Y = YABBADABBADOO. Then the longest common subsequence is Z = ABADABA. See Fig. 4 X = A B R A C A D A B R A LCS = A B A D A B A Y = Y A B B A D A B B A D O O Fig. 4: An example of the LCS of two strings X and Y. The Longest Common Subsequence Problem (LCS) is the following. Given two sequences X = x,...,x m and Y = y,...,y n determine a longest common subsequence. Note that it is not always unique. For example the LCS of ABC and BAC is either AC or BC. DP Formulation for LCS: The simple brute-force solution to the problem would be to try all possible subsequences from one string, and search for matches in the other string, but this is hopelessly inefficient, since there are an exponential number of possible subsequences. Instead, we will derive a dynamic programming solution. In typical DP fashion, we need to break the problem into smaller pieces. There are many ways to do this for strings, but it turns out for this problem that considering all pairs of prefixes will suffice for us. A prefix of a sequence is just an initial string of values, X i = x,x,...,x i. X 0 is the empty sequence. The idea will be to compute the longest common subsequence for every possible pair of prefixes. Let c[i, j] denote the length of the longest common subsequence of X i and Y j. For example, in the above case we have X = ABRAC and Y 6 = YABBAD. Their longest common subsequence is ABA. Thus, c[, 6] = 3. Which of the c[i, j] values do we compute? Since we don t know which will lead to the final optimum, we compute all of them. Eventually we are interested in c[m, n] since this will be the LCS of the two entire strings. The idea is to compute c[i, j] assuming that we already know the values of c[i,j ], for i i and j j (but not both equal). Here are the possible cases. Lecture Notes CMSC 4

14 Y: =n Y: =n B D C B B D C B X = BACDB B 0 Y = BDCB X: A 0 X: 3 C B A C D 0 4 D 0 m= B 0 3 LCS = BCB m= B 0 3 start here LCS Length Table with back pointers included Fig. 6: Longest common subsequence example for the sequences X = BACDB and Y = BCDB. The numeric table entries are the values of c[i, j] and the arrow entries are used in the extraction of the sequence. Build LCS Table LCS(x[..m], y[..n]) { // compute LCS table int c[0..m, 0..n] for i = 0 to m // init column 0 c[i,0] = 0; b[i,0] = SKIPX for j = 0 to n // init row 0 c[0,j] = 0; b[0,j] = SKIPY for i = to m // fill rest of table for j = to n if (x[i] == y[j]) // take X[i] (Y[j]) for LCS c[i,j] = c[i-,j-]+; b[i,j] = addxy else if (c[i-,j] >= c[i,j-]) // X[i] not in LCS c[i,j] = c[i-,j]; b[i,j] = skipx else // Y[j] not in LCS c[i,j] = c[i,j-]; b[i,j] = skipy return c[m,n] // return length of LCS getlcs(x[..m], y[..n], b[0..m,0..n]) { LCSstring = empty string i = m; j = n while(i!= 0 && j!= 0) switch b[i,j] // start at lower right // go until upper left case addxy: // add X[i] (=Y[j]) add x[i] (or equivalently y[j]) to front of LCSstring i--; j--; break case skipx: i--; break // skip X[i] case skipy: j--; break // skip Y[j] return LCSstring Extracting the LCS Lecture Notes 4 CMSC 4

15 The running time of the algorithm is clearly O(mn) since there are two nested loops with m and n iterations, respectively. The algorithm also uses O(mn) space. Extracting the Actual Sequence: Extracting the final LCS is done by using the back pointers stored in b[0..m, 0..n]. Intuitively b[i, j] =add XY means that X[i] and Y [j] together form the last character of the LCS. So we take this common character, and continue with entry b[i,j ] to the northwest ( ). If b[i, j] =skip X, then we know that X[i] is not in the LCS, and so we skip it and go to b[i,j]above us ( ). Similarly, if b[i, j] =skip Y, then we know that Y [j] is not in the LCS, and so we skip it and go to b[i, j ] to the left ( ). Following these back pointers, and outputting a character with each diagonal move gives the final subsequence. Lecture : Dynamic Programming: Chain Matrix Multiplication Read: Chapter of CLRS, and Section. in particular. Chain Matrix Multiplication: This problem involves the question of determining the optimal sequence for performing a series of operations. This general class of problem is important in compiler design for code optimization and in databases for query optimization. We will study the problem in a very restricted instance, where the dynamic programming issues are easiest to see. Suppose that we wish to multiply a series of matrices A A...A n Matrix multiplication is an associative but not a commutative operation. This means that we are free to parenthesize the above multiplication however we like, but we are not free to rearrange the order of the matrices. Also recall that when two (nonsquare) matrices are being multiplied, there are restrictions on the dimensions. A p q matrix has p rows and q columns. You can multiply a p q matrix A times a q r matrix B, and the result will be a p r matrix C. (The number of columns of A must equal the number of rows of B.) In particular for i p and j r, q C[i, j] = A[i, k]b[k, j]. k= This corresponds to the (hopefully familiar) rule that the [i, j] entry of C is the dot product of the ith (horizontal) row of A and the jth (vertical) column of B. Observe that there are pr total entries in C and each takes O(q) time to compute, thus the total time to multiply these two matrices is proportional to the product of the dimensions, pqr. A * B = C p q r = p Multiplication time = pqr q r Fig. 7: Matrix Multiplication. Note that although any legal parenthesization will lead to a valid result, not all involve the same number of operations. Consider the case of 3 matrices: A be 4, A be 4 6 and A 3 be 6. multcost[((a A )A 3 )] = ( 4 6) + ( 6 ) = 80, multcost[(a (A A 3 ))] = (4 6 ) + ( 4 )=88. Even for this small example, considerable savings can be achieved by reordering the evaluation sequence. Lecture Notes CMSC 4

16 Chain Matrix Multiplication Problem: Given a sequence of matrices A,A,...,A n and dimensions p 0,p,...,p n where A i is of dimension p i p i, determine the order of multiplication (represented, say, as a binary tree) that minimizes the number of operations. Important Note: This algorithm does not perform the multiplications, it just determines the best order in which to perform the multiplications. Naive Algorithm: We could write a procedure which tries all possible parenthesizations. Unfortunately, the number of ways of parenthesizing an expression is very large. If you have just one or two matrices, then there is only one way to parenthesize. If you have n items, then there are n places where you could break the list with the outermost pair of parentheses, namely just after the st item, just after the nd item, etc., and just after the (n )st item. When we split just after the kth item, we create two sublists to be parenthesized, one with k items, and the other with n k items. Then we could consider all the ways of parenthesizing these. Since these are independent choices, if there are L ways to parenthesize the left sublist and R ways to parenthesize the right sublist, then the total is L R. This suggests the following recurrence for P (n), the number of different ways of parenthesizing n items: { if n =, P (n) = n k= P (k)p (n k) if n. This is related to a famous function in combinatorics called the Catalan numbers (which in turn is related to the number of different binary trees on n nodes). In particular P (n) =C(n ), where C(n) is the nth Catalan number: C(n) = n+ Applying Stirling s formula (which is given in our text), we find that C(n) Ω(4 n /n 3/ ). Since 4 n is exponential and n 3/ is just polynomial, the exponential will dominate, implying that function grows very fast. Thus, this will not be practical except for very small n. In summary, brute force is not an option. Dynamic Programming Approach: This problem, like other dynamic programming problems involves determining a structure (in this case, a parenthesization). We want to break the problem into subproblems, whose solutions can be combined to solve the global problem. As is common to any DP solution, we need to find some way to break the problem into smaller subproblems, and we need to determine a recursive formulation, which represents the optimum solution to each problem in terms of solutions to the subproblems. Let us think of how we can do this. Since matrices cannot be reordered, it makes sense to think about sequences of matrices. Let A i..j denote the result of multiplying matrices i through j. It is easy to see that A i..j is a p i p j matrix. (Think about this for a second to be sure you see why.) Now, in order to determine how to perform this multiplication optimally, we need to make many decisions. What we want to do is to break the problem into problems of a similar structure. In parenthesizing the expression, we can consider the highest level of parenthesization. At this level we are simply multiplying two matrices together. That is, for any k, k n, ( n n ). A..n = A..k A k+..n. Thus the problem of determining the optimal sequence of multiplications is broken up into two questions: how do we decide where to split the chain (what is k?) and how do we parenthesize the subchains A..k and A k+..n? The subchain problems can be solved recursively, by applying the same scheme. So, let us think about the problem of determining the best value of k. At this point, you may be tempted to consider some clever ideas. For example, since we want matrices with small dimensions, pick the value of k that minimizes p k. Although this is not a bad idea, in principle. (After all it might work. It just turns out that it doesn t in this case. This takes a bit of thinking, which you should try.) Instead, as is true in almost all dynamic programming solutions, we will do the dumbest thing of simply considering all possible choices of k, and taking the best of them. Usually trying all possible choices is bad, since it quickly leads to an exponential Lecture Notes 6 CMSC 4

17 number of total possibilities. What saves us here is that there are only O(n ) different sequences of matrices. (There are ( n ) = n(n )/ ways of choosing i and j to form Ai..j to be precise.) Thus, we do not encounter the exponential growth. Notice that our chain matrix multiplication problem satisfies the principle of optimality, because once we decide to break the sequence into the product A..k A k+..n, we should compute each subsequence optimally. That is, for the global problem to be solved optimally, the subproblems must be solved optimally as well. Dynamic Programming Formulation: We will store the solutions to the subproblems in a table, and build the table in a bottom-up manner. For i j n, let m[i, j] denote the minimum number of multiplications needed to compute A i..j. The optimum cost can be described by the following recursive formulation. Basis: Observe that if i = j then the sequence contains only one matrix, and so the cost is 0. (There is nothing to multiply.) Thus, m[i, i] =0. Step: If i<j, then we are asking about the product A i..j. This can be split by considering each k, i k<j, as A i..k times A k+..j. The optimum times to compute A i..k and A k+..j are, by definition, m[i, k] and m[k +,j], respectively. We may assume that these values have been computed previously and are already stored in our array. Since A i..k is a p i p k matrix, and A k+..j is a p k p j matrix, the time to multiply them is p i p k p j. This suggests the following recursive rule for computing m[i, j]. m[i, i] = 0 m[i, j] = min i p k p j ) i k<j for i<j. A i..j A i..k A k+..j A A... A A... A i i+ k k+ j Fig. 8: Dynamic Programming Formulation.? It is not hard to convert this rule into a procedure, which is given below. The only tricky part is arranging the order in which to compute the values. In the process of computing m[i, j] we need to access values m[i, k] and m[k +,j]for k lying between i and j. This suggests that we should organize our computation according to the number of matrices in the subsequence. Let L = j i+ denote the length of the subchain being multiplied. The subchains of length (m[i, i]) are trivial to compute. Then we build up by computing the subchains of lengths, 3,...,n. The final answer is m[,n]. We need to be a little careful in setting up the loops. If a subchain of length L starts at position i, then j = i + L. Since we want j n, this means that i + L n, orin other words, i n L +. So our loop for i runs from to n L +(in order to keep j in bounds). The code is presented below. The array s[i, j] will be explained later. It is used to extract the actual sequence. The running time of the procedure is Θ(n 3 ). We ll leave this as an exercise in solving sums, but the key is that there are three nested loops, and each can iterate at most n times. Extracting the final Sequence: Extracting the actual multiplication sequence is a fairly easy extension. The basic idea is to leave a split marker indicating what the best split is, that is, the value of k that leads to the minimum Lecture Notes 7 CMSC 4

18 Matrix-Chain(array p[..n]) { array s[..n-,..n] for i = to n do m[i,i] = 0; for L = to n do { for i = to n-l+ do { j = i + L - ; m[i,j] = INFINITY; for k = i to j- do { // initialize // L = length of subchain // check all splits q = m[i, k] + m[k+, j] + p[i-]*p[k]*p[j] if (q < m[i, j]) { m[i,j] = q; s[i,j] = k; return m[,n] (final cost) and s (splitting markers); Chain Matrix Multiplication value of m[i, j]. We can maintain a parallel array s[i, j] in which we will store the value of k providing the optimal split. For example, suppose that s[i, j] =k. This tells us that the best way to multiply the subchain A i..j is to first multiply the subchain A i..k and then multiply the subchain A k+..j, and finally multiply these together. Intuitively, s[i, j] tells us what multiplication to perform last. Note that we only need to store s[i, j] when we have at least two matrices, that is, if j>i. The actual multiplication algorithm uses the s[i, j] value to determine how to split the current sequence. Assume that the matrices are stored in an array of matrices A[..n], and that s[i, j] is global to this recursive procedure. The recursive procedure Mult does this computation and below returns a matrix. Mult(i, j) { if (i == j) return A[i]; else { k = s[i,j] X = Mult(i, k) Y = Mult(k+, j) return X*Y; Extracting Optimum Sequence // basis case // X = A[i]...A[k] // Y = A[k+]...A[j] // multiply matrices X and Y In the figure below we show an example. This algorithm is tricky, so it would be a good idea to trace through this example (and the one given in the text). The initial set of dimensions are, 4, 6,, 7 meaning that we are multiplying A ( 4) times A (4 6) times A 3 (6 ) times A 4 ( 7). The optimal sequence is ((A (A A 3 ))A 4 ). Lecture 6: Dynamic Programming: Minimum Weight Triangulation Read: This is not covered in CLRS. Lecture Notes 8 CMSC 4

19 4 m[i,j] 8 3 j i A A A 3 A 4 j 3 4 s[i,j] i 3 3 Final order p0 p p p 3 p4 A A A 3 A 4 Fig. 9: Chain Matrix Multiplication Example. Polygons and Triangulations: Let s consider a geometric problem that outwardly appears to be quite different from chain-matrix multiplication, but actually has remarkable similarities. We begin with a number of definitions. Define a polygon to be a piecewise linear closed curve in the plane. In other words, we form a cycle by joining line segments end to end. The line segments are called the sides of the polygon and the endpoints are called the vertices. A polygon is simple if it does not cross itself, that is, if the sides do not intersect one another except for two consecutive sides sharing a common vertex. A simple polygon subdivides the plane into its interior, its boundary and its exterior. A simple polygon is said to be convex if every interior angle is at most 80 degrees. Vertices with interior angle equal to 80 degrees are normally allowed, but for this problem we will assume that no such vertices exist. Polygon Simple polygon Convex polygon Fig. 0: Polygons. Given a convex polygon, we assume that its vertices are labeled in counterclockwise order P = v,...,v n. We will assume that indexing of vertices is done modulo n,sov 0 =v n. This polygon has n sides, v i v i. Given two nonadjacent sides v i and v j, where i<j, the line segment v i v j is a chord. (If the polygon is simple but not convex, we include the additional requirement that the interior of the segment must lie entirely in the interior of P.) Any chord subdivides the polygon into two polygons: v i,v i+,...,v j, and v j,v j+,...,v i. A triangulation of a convex polygon P is a subdivision of the interior of P into a collection of triangles with disjoint interiors, whose vertices are drawn from the vertices of P. Equivalently, we can define a triangulation as a maximal set T of nonintersecting chords. (In other words, every chord that is not in T intersects the interior of some chord in T.) It is easy to see that such a set of chords subdivides the interior of the polygon into a collection of triangles with pairwise disjoint interiors (and hence the name triangulation). It is not hard to prove (by induction) that every triangulation of an n-sided polygon consists of n 3 chords and n triangles. Triangulations are of interest for a number of reasons. Many geometric algorithm operate by first decomposing a complex polygonal shape into triangles. In general, given a convex polygon, there are many possible triangulations. In fact, the number is exponential in n, the number of sides. Which triangulation is the best? There are many criteria that are used depending on the application. One criterion is to imagine that you must pay for the ink you use in drawing the triangulation, and you want to minimize the amount of ink you use. (This may sound fanciful, but minimizing wire length is an Lecture Notes 9 CMSC 4

20 important condition in chip design. Further, this is one of many properties which we could choose to optimize.) This suggests the following optimization problem: Minimum-weight convex polygon triangulation: Given a convex polygon determine the triangulation that minimizes the sum of the perimeters of its triangles. (See Fig..) A triangulation Lower weight triangulation Fig. : Triangulations of convex polygons, and the minimum weight triangulation. Given three distinct vertices v i, v j, v k, we define the weight of the associated triangle by the weight function w(v i,v j,v k )= v i v j + v j v k + v k v i, where v i v j denotes the length of the line segment v i v j. Dynamic Programming Solution: Let us consider an (n +)-sided polygon P = v 0,v,...,v n. Let us assume that these vertices have been numbered in counterclockwise order. To derive a DP formulation we need to define a set of subproblems from which we can derive the optimum solution. For 0 i<j n, define t[i, j] to be the weight of the minimum weight triangulation for the subpolygon that lies to the right of directed chord v i v j, that is, the polygon with the counterclockwise vertex sequence v i,v i+,...,v j. Observe that if we can compute this quantity for all such i and j, then the weight of the minimum weight triangulation of the entire polygon can be extracted as t[0,n]. (As usual, we only compute the minimum weight. But, it is easy to modify the procedure to extract the actual triangulation.) As a basis case, we define the weight of the trivial -sided polygon to be zero, implying that t[i, i +]=0. In general, to compute t[i, j], consider the subpolygon v i,v i+,...,v j, where j>i+. One of the chords of this polygon is the side v i v j. We may split this subpolygon by introducing a triangle whose base is this chord, and whose third vertex is any vertex v k, where i<k<j. This subdivides the polygon into the subpolygons v i,v i+,...v k and v k,v k+,...v j whose minimum weights are already known to us as t[i, k] and t[k, j]. In addition we should consider the weight of the newly added triangle v i v k v j. Thus, we have the following recursive rule: t[i, j] = { 0 if j = i + min i<k<j (t[i, k]+t[k, j]+w(v i v k v j )) if j>i+. The final output is the overall minimum weight, which is, t[0,n]. This is illustrated in Fig. Note that this has almost exactly the same structure as the recursive definition used in the chain matrix multiplication algorithm (except that some indices are different by.) The same Θ(n 3 ) algorithm can be applied with only minor changes. Relationship to Binary Trees: One explanation behind the similarity of triangulations and the chain matrix multiplication algorithm is to observe that both are fundamentally related to binary trees. In the case of the chain matrix multiplication, the associated binary tree is the evaluation tree for the multiplication, where the leaves of the tree correspond to the matrices, and each node of the tree is associated with a product of a sequence of two or more matrices. To see that there is a similar correspondence here, consider an (n +)-sided convex polygon P = v 0,v,...,v n, and fix one side of the polygon (say v 0 v n ). Now consider a rooted binary tree whose root node is the triangle containing side v 0 v n, whose internal nodes are the nodes of the dual tree, and whose leaves Lecture Notes 0 CMSC 4

### Memoization/Dynamic Programming. The String reconstruction problem. CS125 Lecture 5 Fall 2016

CS125 Lecture 5 Fall 2016 Memoization/Dynamic Programming Today s lecture discusses memoization, which is a method for speeding up algorithms based on recursion, by using additional memory to remember

### Dynamic Programming. Lecture 11. 11.1 Overview. 11.2 Introduction

Lecture 11 Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to solve many different types of problems in time O(n 2 ) or O(n 3 ) for which a naive approach

### Steven Skiena. skiena

Lecture 3: Recurrence Relations (1997) Steven Skiena Department of Computer Science State University of New York Stony Brook, NY 11794 4400 http://www.cs.sunysb.edu/ skiena Argue the solution to T(n) =

### CS473 - Algorithms I

CS473 - Algorithms I Lecture 9 Sorting in Linear Time View in slide-show mode 1 How Fast Can We Sort? The algorithms we have seen so far: Based on comparison of elements We only care about the relative

### CS268: Geometric Algorithms Handout #5 Design and Analysis Original Handout #15 Stanford University Tuesday, 25 February 1992

CS268: Geometric Algorithms Handout #5 Design and Analysis Original Handout #15 Stanford University Tuesday, 25 February 1992 Original Lecture #6: 28 January 1991 Topics: Triangulating Simple Polygons

### The divide and conquer strategy has three basic parts. For a given problem of size n,

1 Divide & Conquer One strategy for designing efficient algorithms is the divide and conquer approach, which is also called, more simply, a recursive approach. The analysis of recursive algorithms often

### Monotone Partitioning. Polygon Partitioning. Monotone polygons. Monotone polygons. Monotone Partitioning. ! Define monotonicity

Monotone Partitioning! Define monotonicity Polygon Partitioning Monotone Partitioning! Triangulate monotone polygons in linear time! Partition a polygon into monotone pieces Monotone polygons! Definition

### Michal Forišek: Early beta verzia skrípt z ADŠ

Časová zložitosť II Michal Forišek: Early beta verzia skrípt z ADŠ In this part of the article we will focus on estimating the time complexity for recursive programs. In essence, this will lead to finding

### Matrix Product. Matrix-Multiply (A, B) Cost of multiplying these matrices is p q r.

Dynamic Programming A powerful paradigm for algorithm design. Often leads to elegant and efficient algorithms when greedy or diide-and-conquer don t work. DP also breaks a problem into subproblems, but

### A. V. Gerbessiotis CS Spring 2014 PS 3 Mar 24, 2014 No points

A. V. Gerbessiotis CS 610-102 Spring 2014 PS 3 Mar 24, 2014 No points Problem 1. Suppose that we insert n keys into a hash table of size m using open addressing and uniform hashing. Let p(n, m) be the

### CS420/520 Algorithm Analysis Spring 2010 Lecture 04

CS420/520 Algorithm Analysis Spring 2010 Lecture 04 I. Chapter 4 Recurrences A. Introduction 1. Three steps applied at each level of a recursion a) Divide into a number of subproblems that are smaller

### Approximation Algorithms

Approximation Algorithms or: How I Learned to Stop Worrying and Deal with NP-Completeness Ong Jit Sheng, Jonathan (A0073924B) March, 2012 Overview Key Results (I) General techniques: Greedy algorithms

### Catalan Numbers. Thomas A. Dowling, Department of Mathematics, Ohio State Uni- versity.

7 Catalan Numbers Thomas A. Dowling, Department of Mathematics, Ohio State Uni- Author: versity. Prerequisites: The prerequisites for this chapter are recursive definitions, basic counting principles,

### Problem Set #1 Solutions

CS161: Design and Analysis of Algorithms Summer 2004 Problem Set #1 Solutions General Notes Regrade Policy: If you believe an error has been made in the grading of your problem set, you may resubmit it

### Why? A central concept in Computer Science. Algorithms are ubiquitous.

Analysis of Algorithms: A Brief Introduction Why? A central concept in Computer Science. Algorithms are ubiquitous. Using the Internet (sending email, transferring files, use of search engines, online

### Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay

Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding

### 2 Exercises and Solutions

2 Exercises and Solutions Most of the exercises below have solutions but you should try first to solve them. Each subsection with solutions is after the corresponding subsection with exercises. 2.1 Sorting

### The Selection Problem

The Selection Problem The selection problem: given an integer k and a list x 1,..., x n of n elements find the k-th smallest element in the list Example: the 3rd smallest element of the following list

### Recurrences (CLRS )

Recurrences (CLRS 4.1-4.2) Last time we discussed divide-and-conquer algorithms Divide and Conquer To Solve P: 1. Divide P into smaller problems P 1,P 2,P 3...P k. 2. Conquer by solving the (smaller) subproblems

### CS 4310 HOMEWORK SET 1

CS 4310 HOMEWORK SET 1 PAUL MILLER Section 2.1 Exercises Exercise 2.1-1. Using Figure 2.2 as a model, illustrate the operation of INSERTION-SORT on the array A = 31, 41, 59, 26, 41, 58. Solution: Assume

### The Union-Find Problem Kruskal s algorithm for finding an MST presented us with a problem in data-structure design. As we looked at each edge,

The Union-Find Problem Kruskal s algorithm for finding an MST presented us with a problem in data-structure design. As we looked at each edge, cheapest first, we had to determine whether its two endpoints

### Discrete Mathematics, Chapter 3: Algorithms

Discrete Mathematics, Chapter 3: Algorithms Richard Mayr University of Edinburgh, UK Richard Mayr (University of Edinburgh, UK) Discrete Mathematics. Chapter 3 1 / 28 Outline 1 Properties of Algorithms

### Binary Search. Search for x in a sorted array A.

Divide and Conquer A general paradigm for algorithm design; inspired by emperors and colonizers. Three-step process: 1. Divide the problem into smaller problems. 2. Conquer by solving these problems. 3.

### Fast Sequential Summation Algorithms Using Augmented Data Structures

Fast Sequential Summation Algorithms Using Augmented Data Structures Vadim Stadnik vadim.stadnik@gmail.com Abstract This paper provides an introduction to the design of augmented data structures that offer

Answers to Homework 1 p. 13 1.2 3 What is the smallest value of n such that an algorithm whose running time is 100n 2 runs faster than an algorithm whose running time is 2 n on the same machine? Find the

### Many algorithms, particularly divide and conquer algorithms, have time complexities which are naturally

Recurrence Relations Many algorithms, particularly divide and conquer algorithms, have time complexities which are naturally modeled by recurrence relations. A recurrence relation is an equation which

### 9.2 Summation Notation

9. Summation Notation 66 9. Summation Notation In the previous section, we introduced sequences and now we shall present notation and theorems concerning the sum of terms of a sequence. We begin with a

### Solutions to Homework 6

Solutions to Homework 6 Debasish Das EECS Department, Northwestern University ddas@northwestern.edu 1 Problem 5.24 We want to find light spanning trees with certain special properties. Given is one example

### Sorting algorithms. Mergesort and Quicksort. Merge Sort. Partitioning Choice 1. Partitioning Choice 3. Partitioning Choice 2 2/5/2015

Sorting algorithms Mergesort and Quicksort Insertion, selection and bubble sort have quadratic worst case performance The faster comparison based algorithm? O(nlogn) Mergesort and Quicksort Merge Sort

### Merge Sort. 2004 Goodrich, Tamassia. Merge Sort 1

Merge Sort 7 2 9 4 2 4 7 9 7 2 2 7 9 4 4 9 7 7 2 2 9 9 4 4 Merge Sort 1 Divide-and-Conquer Divide-and conquer is a general algorithm design paradigm: Divide: divide the input data S in two disjoint subsets

### Algorithms and Data Structures

Algorithm Analysis Page 1 BFH-TI: Softwareschule Schweiz Algorithm Analysis Dr. CAS SD01 Algorithm Analysis Page 2 Outline Course and Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations

### Divide And Conquer Algorithms

CSE341T/CSE549T 09/10/2014 Lecture 5 Divide And Conquer Algorithms Recall in last lecture, we looked at one way of parallelizing matrix multiplication. At the end of the lecture, we saw the reduce SUM

### Writing programs to solve problem consists of a large. how to represent aspects of the problem for solution

Algorithm Efficiency & Sorting Algorithm efficiency Big-O notation Searching algorithms Sorting algorithms 1 Overview 2 If a given ADT (i.e. stack or queue) is attractive as part of a solution How will

### CSE 326, Data Structures. Sample Final Exam. Problem Max Points Score 1 14 (2x7) 2 18 (3x6) 3 4 4 7 5 9 6 16 7 8 8 4 9 8 10 4 Total 92.

Name: Email ID: CSE 326, Data Structures Section: Sample Final Exam Instructions: The exam is closed book, closed notes. Unless otherwise stated, N denotes the number of elements in the data structure

### Lecture 12: Chain Matrix Multiplication

Lecture 12: Chain Matrix Multiplication CLRS Section 15.2 Outline of this Lecture Recalling matrix multiplication. The chain matrix multiplication problem. A dynamic programming algorithm for chain matrix

### Algorithms. Margaret M. Fleck. 18 October 2010

Algorithms Margaret M. Fleck 18 October 2010 These notes cover how to analyze the running time of algorithms (sections 3.1, 3.3, 4.4, and 7.1 of Rosen). 1 Introduction The main reason for studying big-o

### Two General Methods to Reduce Delay and Change of Enumeration Algorithms

ISSN 1346-5597 NII Technical Report Two General Methods to Reduce Delay and Change of Enumeration Algorithms Takeaki Uno NII-2003-004E Apr.2003 Two General Methods to Reduce Delay and Change of Enumeration

### Design and Analysis of Algorithms

Design and Analysis of Algorithms CSE 5311 Lecture 3 Divide-and-Conquer Junzhou Huang, Ph.D. Department of Computer Science and Engineering CSE5311 Design and Analysis of Algorithms 1 Reviewing: Θ-notation

### Introduction to Algorithms March 10, 2004 Massachusetts Institute of Technology Professors Erik Demaine and Shafi Goldwasser Quiz 1.

Introduction to Algorithms March 10, 2004 Massachusetts Institute of Technology 6.046J/18.410J Professors Erik Demaine and Shafi Goldwasser Quiz 1 Quiz 1 Do not open this quiz booklet until you are directed

### 4.4 The recursion-tree method

4.4 The recursion-tree method Let us see how a recursion tree would provide a good guess for the recurrence = 3 4 ) Start by nding an upper bound Floors and ceilings usually do not matter when solving

### Data Structures and Algorithms Week 3

Data Structures and Algorithms Week 3 1. Divide and conquer 2. Merge sort, repeated substitutions 3. Tiling 4. Recurrences Recurrences Running times of algorithms with recursive calls can be described

### CS2223 Algorithms B Term 2013 Exam 2 Solutions

CS2223 Algorithms B Term 2013 Exam 2 Solutions Dec. 3, 2013 By Prof. Carolina Ruiz Dept. of Computer Science WPI PROBLEM 1: Solving Recurrences (15 points) Solve the recurrence T (n) = 2T (n/3) + n using

### The Running Time of Programs

CHAPTER 3 The Running Time of Programs In Chapter 2, we saw two radically different algorithms for sorting: selection sort and merge sort. There are, in fact, scores of algorithms for sorting. This situation

### Lecture 2: Analysis of Algorithms (CS )

Lecture 2: Analysis of Algorithms (CS583-002) Amarda Shehu September 03 & 10, 2014 1 Outline of Today s Class 2 Big-Oh, Big-Omega, Theta Techniques for Finding Asymptotic Relationships Some more Fun with

### Cost Model: Work, Span and Parallelism. 1 The RAM model for sequential computation:

CSE341T 08/31/2015 Lecture 3 Cost Model: Work, Span and Parallelism In this lecture, we will look at how one analyze a parallel program written using Cilk Plus. When we analyze the cost of an algorithm

### Divide-and-Conquer Algorithms Part One

Divide-and-Conquer Algorithms Part One Announcements Problem Set One completely due right now. Solutions distributed at the end of lecture. Programming section today in Gates B08 from from 3:45PM 5:00PM.

### Merge Sort and Recurrences. COSC 581, Algorithms January 14, 2014

Merge Sort and Recurrences COSC 581, Algorithms January 14, 2014 Reading Assignments Today s class: Chapter 2, 4.0, 4.4 Reading assignment for next class: Chapter 4.2, 4.5 3 Common Algorithmic Techniques

### Intro. to the Divide-and-Conquer Strategy via Merge Sort CMPSC 465 CLRS Sections 2.3, Intro. to and various parts of Chapter 4

Intro. to the Divide-and-Conquer Strategy via Merge Sort CMPSC 465 CLRS Sections 2.3, Intro. to and various parts of Chapter 4 I. Algorithm Design and Divide-and-Conquer There are various strategies we

### Chapters 1 and 5.2 of GT

Subject 2 Spring 2014 Growth of Functions and Recurrence Relations Handouts 3 and 4 contain more examples;the Formulae Collection handout might also help. Chapters 1 and 5.2 of GT Disclaimer: These abbreviated

### December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS

December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation

### Offline sorting buffers on Line

Offline sorting buffers on Line Rohit Khandekar 1 and Vinayaka Pandit 2 1 University of Waterloo, ON, Canada. email: rkhandekar@gmail.com 2 IBM India Research Lab, New Delhi. email: pvinayak@in.ibm.com

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### ON THE COMPLEXITY OF THE GAME OF SET. {kamalika,pbg,dratajcz,hoeteck}@cs.berkeley.edu

ON THE COMPLEXITY OF THE GAME OF SET KAMALIKA CHAUDHURI, BRIGHTEN GODFREY, DAVID RATAJCZAK, AND HOETECK WEE {kamalika,pbg,dratajcz,hoeteck}@cs.berkeley.edu ABSTRACT. Set R is a card game played with a

### Dynamic Programming. Applies when the following Principle of Optimality

Dynamic Programming Applies when the following Principle of Optimality holds: In an optimal sequence of decisions or choices, each subsequence must be optimal. Translation: There s a recursive solution.

### 2.3 Scheduling jobs on identical parallel machines

2.3 Scheduling jobs on identical parallel machines There are jobs to be processed, and there are identical machines (running in parallel) to which each job may be assigned Each job = 1,,, must be processed

### Computer Algorithms. Merge Sort. Analysis of Merge Sort. Recurrences. Recurrence Relations. Iteration Method. Lecture 6

Computer Algorithms Lecture 6 Merge Sort Sorting Problem: Sort a sequence of n elements into non-decreasing order. Divide: Divide the n-element sequence to be sorted into two subsequences of n/ elements

### Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

### Efficiency of algorithms. Algorithms. Efficiency of algorithms. Binary search and linear search. Best, worst and average case.

Algorithms Efficiency of algorithms Computational resources: time and space Best, worst and average case performance How to compare algorithms: machine-independent measure of efficiency Growth rate Complexity

### The Greedy Method. Fundamental Techniques. The Greedy Method. Fractional Knapsack Problem

Fundamental Techniques There are some algorithmic tools that are quite specialised. They are good for problems they are intended to solve, but they are not very versatile. There are also more fundamental

### A Note on Maximum Independent Sets in Rectangle Intersection Graphs

A Note on Maximum Independent Sets in Rectangle Intersection Graphs Timothy M. Chan School of Computer Science University of Waterloo Waterloo, Ontario N2L 3G1, Canada tmchan@uwaterloo.ca September 12,

### An Interesting Way to Combine Numbers

An Interesting Way to Combine Numbers Joshua Zucker and Tom Davis November 28, 2007 Abstract This exercise can be used for middle school students and older. The original problem seems almost impossibly

### Quicksort An Example. When lo and hi cross, we are done. The final value of hi is the position in which the partitioning element ends up.

Quicksort An Example We sort the array A = (38 81 22 48 13 69 93 14 45 58 79 72) with quicksort, always choosing the pivot element to be the element in position (left+right)/2. The partitioning during

### CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis. Linda Shapiro Winter 2015

CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis Linda Shapiro Today Registration should be done. Homework 1 due 11:59 pm next Wednesday, January 14 Review math essential

### Lecture 18: Applications of Dynamic Programming Steven Skiena. Department of Computer Science State University of New York Stony Brook, NY 11794 4400

Lecture 18: Applications of Dynamic Programming Steven Skiena Department of Computer Science State University of New York Stony Brook, NY 11794 4400 http://www.cs.sunysb.edu/ skiena Problem of the Day

### CMPS 102 Solutions to Homework 1

CMPS 0 Solutions to Homework Lindsay Brown, lbrown@soe.ucsc.edu September 9, 005 Problem..- p. 3 For inputs of size n insertion sort runs in 8n steps, while merge sort runs in 64n lg n steps. For which

### 1 Limiting distribution for a Markov chain

Copyright c 2009 by Karl Sigman Limiting distribution for a Markov chain In these Lecture Notes, we shall study the limiting behavior of Markov chains as time n In particular, under suitable easy-to-check

### Analysis of Algorithms, I

Analysis of Algorithms, I CSOR W4231.002 Eleni Drinea Computer Science Department Columbia University Thursday, February 26, 2015 Outline 1 Recap 2 Representing graphs 3 Breadth-first search (BFS) 4 Applications

### Big-O analysis. CS 420, Fall 16, McConnell

Big-O analysis CS 420, Fall 16, McConnell 1 A numerical tug of war The worst-case number of elementary steps of an algorithm is a precise function of its parameters. For instance, sorting n numbers with

### O(n) linear time This means that the algorithm requires a number of steps proportional to the size of the task.

Time complexity : Big O notation f(n) = O(g(n)) means There are positive constants c and k such that: 0

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### WHAT ARE MATHEMATICAL PROOFS AND WHY THEY ARE IMPORTANT?

WHAT ARE MATHEMATICAL PROOFS AND WHY THEY ARE IMPORTANT? introduction Many students seem to have trouble with the notion of a mathematical proof. People that come to a course like Math 216, who certainly

### Algebra Unpacked Content For the new Common Core standards that will be effective in all North Carolina schools in the 2012-13 school year.

This document is designed to help North Carolina educators teach the Common Core (Standard Course of Study). NCDPI staff are continually updating and improving these tools to better serve teachers. Algebra

### Relations Graphical View

Relations Slides by Christopher M. Bourke Instructor: Berthe Y. Choueiry Introduction Recall that a relation between elements of two sets is a subset of their Cartesian product (of ordered pairs). A binary

### CHAPTER 3 Numbers and Numeral Systems

CHAPTER 3 Numbers and Numeral Systems Numbers play an important role in almost all areas of mathematics, not least in calculus. Virtually all calculus books contain a thorough description of the natural,

### Can you design an algorithm that searches for the maximum value in the list?

The Science of Computing I Lesson 2: Searching and Sorting Living with Cyber Pillar: Algorithms Searching One of the most common tasks that is undertaken in Computer Science is the task of searching. We

### Algorithms. Theresa Migler-VonDollen CMPS 5P

Algorithms Theresa Migler-VonDollen CMPS 5P 1 / 32 Algorithms Write a Python function that accepts a list of numbers and a number, x. If x is in the list, the function returns the position in the list

### 1 Point Inclusion in a Polygon

Comp 163: Computational Geometry Tufts University, Spring 2005 Professor Diane Souvaine Scribe: Ryan Coleman Point Inclusion and Processing Simple Polygons into Monotone Regions 1 Point Inclusion in a

### How to prove an algorithm is correct?

How to prove an algorithm is correct? To prove the incorrectness of an algorithm, one counter-example is enough. Proving the correctness of an algorithm is similar to proving a mathematical theorem; fundamentally,

### ECEN 5682 Theory and Practice of Error Control Codes

ECEN 5682 Theory and Practice of Error Control Codes Convolutional Codes University of Colorado Spring 2007 Linear (n, k) block codes take k data symbols at a time and encode them into n code symbols.

### Introduction to Algorithms

Introduction to Algorithms (2 nd edition) by Cormen, Leiserson, Rivest & Stein Chapter 4: Recurrences (slides enhanced by N. Adlai A. DePano) Overview Define what a recurrence is Discuss three methods

### CSE 5311 Homework 2 Solution

CSE 5311 Homework 2 Solution Problem 6.2-6 Show that the worst-case running time of MAX-HEAPIFY on a heap of size n is Ω(lg n). (Hint: For a heap with n nodes, give node values that cause MAX- HEAPIFY

### 1 Review of Newton Polynomials

cs: introduction to numerical analysis 0/0/0 Lecture 8: Polynomial Interpolation: Using Newton Polynomials and Error Analysis Instructor: Professor Amos Ron Scribes: Giordano Fusco, Mark Cowlishaw, Nathanael

### Pseudocode Analysis. COMP3600/6466 Algorithms Lecture 5. Recursive Algorithms. Asymptotic Bounds for Recursions

COMP600/6466 Algorithms Lecture 5 Pseudocode Analysis Iterative Recursive S 0 Dr. Hassan Hijazi Prof. Weifa Liang Recursive Algorithms Asymptotic Bounds for Recursions Total running time = Sum of times

### Mathematics for Economics (Part I) Note 5: Convex Sets and Concave Functions

Natalia Lazzati Mathematics for Economics (Part I) Note 5: Convex Sets and Concave Functions Note 5 is based on Madden (1986, Ch. 1, 2, 4 and 7) and Simon and Blume (1994, Ch. 13 and 21). Concave functions

### CSC 143. Two important problems. Searching and Sorting. Review: Linear Search. Example. Review: Binary Search. How Efficient Is Linear Search?

CSC 43 Searching and Sorting [Chapter 9, pp. 402-432] Two important problems Search: finding something in a set of data Sorting: putting a set of data in order Both very common, very useful operations

### 1. What s wrong with the following proofs by induction?

ArsDigita University Month : Discrete Mathematics - Professor Shai Simonson Problem Set 4 Induction and Recurrence Equations Thanks to Jeffrey Radcliffe and Joe Rizzo for many of the solutions. Pasted

### Planar Tree Transformation: Results and Counterexample

Planar Tree Transformation: Results and Counterexample Selim G Akl, Kamrul Islam, and Henk Meijer School of Computing, Queen s University Kingston, Ontario, Canada K7L 3N6 Abstract We consider the problem

### PURSUITS IN MATHEMATICS often produce elementary functions as solutions that need to be

Fast Approximation of the Tangent, Hyperbolic Tangent, Exponential and Logarithmic Functions 2007 Ron Doerfler http://www.myreckonings.com June 27, 2007 Abstract There are some of us who enjoy using our

### Induction Problems. Tom Davis November 7, 2005

Induction Problems Tom Davis tomrdavis@earthlin.net http://www.geometer.org/mathcircles November 7, 2005 All of the following problems should be proved by mathematical induction. The problems are not necessarily

### Symbol Tables. IE 496 Lecture 13

Symbol Tables IE 496 Lecture 13 Reading for This Lecture Horowitz and Sahni, Chapter 2 Symbol Tables and Dictionaries A symbol table is a data structure for storing a list of items, each with a key and

### Recurrence Relations

Recurrence Relations Introduction Determining the running time of a recursive algorithm often requires one to determine the big-o growth of a function T (n) that is defined in terms of a recurrence relation.

### Notes on Factoring. MA 206 Kurt Bryan

The General Approach Notes on Factoring MA 26 Kurt Bryan Suppose I hand you n, a 2 digit integer and tell you that n is composite, with smallest prime factor around 5 digits. Finding a nontrivial factor

### CS60003 Algorithm Design and Analysis, Autumn

CS60003 Algorithm Design and Analysis, Autumn 2010 11 Class test 1 Maximum marks: 40 Time: 07 08 09 10 Duration: 1 + ǫ hour Roll no: Name: [ Write your answers in the question paper itself. Be brief and

### Lecture 2: The Complexity of Some Problems

IAS/PCMI Summer Session 2000 Clay Mathematics Undergraduate Program Basic Course on Computational Complexity Lecture 2: The Complexity of Some Problems David Mix Barrington and Alexis Maciel July 18, 2000

### CS/COE 1501 http://cs.pitt.edu/~bill/1501/

CS/COE 1501 http://cs.pitt.edu/~bill/1501/ Lecture 01 Course Introduction Meta-notes These notes are intended for use by students in CS1501 at the University of Pittsburgh. They are provided free of charge

### COLORED GRAPHS AND THEIR PROPERTIES

COLORED GRAPHS AND THEIR PROPERTIES BEN STEVENS 1. Introduction This paper is concerned with the upper bound on the chromatic number for graphs of maximum vertex degree under three different sets of coloring