Introduction to Computational Linear Algebra

Transcription

1 Introduction to Computational Linear Algebra Dr. Philippe B. Laval The University of Stewart Island September 14, 2001 Abstract This is an introduction to Numerical Analysis and Computational Linear Algebra. It gives a brief explanation about what this branch of mathematics is. It then discusses the issues involved when doing mathematics onacomputer. 1 Introduction Computational Linear Algebra falls under the larger field of Numerical Analysis. Numerical Analysis is the study of the methods used to obtain numerical results (as opposed to analytical results) to various problems and analyze (or predict) the errors produced by those approximations. So, Computational Linear Algebra is Numerical Analysis, applied to matrices. Emphasis will be placed on developing the numerical methods, understanding why they work, as well as on understanding their limitations. Some of the problems we will study this semester include: 1. Direct methods to solve linear systems of equations 2. Iterative methods to solve linear systems of equations 3. Approximating eigenvalues and eigenvectors 4. Applications of the above techniques Numerical methods require such tedious and repetitive arithmetic operations that it makes only sense to use them with a computer. A human would make too many mistakes. It would take a human too long to perform the required computations. However, a computer is essentially dumb and must be given detailed and complete instructions for every single step it is to perform. In other words, a computer program must be written. To better understand what is involved in finding the numerical solution of a problems, it helps to visualize the situation as shown on the flowchart in Figure 1. The various stages are described below. 1

2 Figure 1: Stages involved in solving a problem numerically 1. Input information. This is the given problem. It can be a set of data points we are trying to fit to a function. It could be a function we are trying to approximate. For example, you may need to know an approximate value for ln 5, and you forgot your calculator. It could be an integral you need to evaluate because you need to know the area of a region. This will be given. 2. The algorithm. Analgorithmisastepbystepmethodusedtosolvea problem, usually in a finite number of steps. Thus, an algorithm has to be developed, then it has to be coded so a computer can execute it. 3. Output information. This is what you want. Often, the answer given by the previous stage has to be interpreted for it to make sense to the average user. It appears then that to go from the input information to the output information, one has to find an algorithm which will perform the task. Thus, Numerical Analysis seems to be the science of finding algorithms. We realize very quickly there is much more to it. Very often, several algorithms exist to solve a problem. We must select one. There are various reasons for preferring one algorithm over another. The factors to take into account include: Speed. All other things being equal, the faster method will win. The company using the algorithm may have to pay to use the computer running it. In this case, time is also money. Accuracy. Numerical methods usually involve errors. Errors are present for different reasons, we will outline them later. But we have to deal with them. An algorithm which minimizes error growth will be highly considered. Very often, speed is at the expense of accuracy. In this case, we will select the fastest algorithm which gives us the desired accuracy. Computer resources. Some algorithms require a lot of memory to run, more than the machine being used may have. Thus, this aspect has to be taken into account. There are at least three main differences between an analytical solution and a numerical solution. 2

3 First, it is important to realize that a numerical analysis solution is always numerical. Often, an analytical method gives its answer in terms of a function which can then be used for specific instances. By studying the function, one can then deduce some of the properties the solution may have. Because the numerical solution is a number, we cannot analyze the properties of the answer. However, numerical solutions can be plotted to show the behavior of the solution. Another important distinction is that the result from numerical analysis is an approximation. However, the result can usually be made as accurate as needed. There are limitations to the achievable level of accuracy because of the way computers do arithmetic. In any case, the analysis of errors in a numerical analysis solution is a critically important part of the solution. Finally, an important advantage of numerical analysis is that it will give a solution, even when an analytical solution is not known. We now look more closely at the notion of error and how computers do arithmetic. 2 Computer Arithmetic and Error Analysis 2.1 Where do Errors Come From? In the practice of Numerical Analysis, it is important to be aware that computed solutions are not exact mathematical solutions. Errors can be introduced at the input information stage, or the algorithm stage. 1. Some errors at the input information stage include: (a) Error in the input data. The input information could be data coming from some measuring device. Such devices are rarely completely accurate. (b) Human errors. Computers are often blamed whenever there is an error. However, because humans are involved in the input phase, as well as in the algorithm and oputput phases, gross errors due to humans do occur. This is why every numerical result must be checked for reasonableness. (c) Error in the representation of the data. Even if we know the exact values of the data, the computer may only be able to handle an approximation of these values. For example, 1 is an irrational number; 3 it has an infinite number of digits. A computer can only handle a finite number of digits, therefore 1 might be approximated. In addition, due to hardware limitations, the computer can only represent 3 a small subset of the real numbers. Many numbers are therefore approximated, this introduces an error. 3

4 2. Errors introduced at the algorithm stage include: (a) Truncation errors. For example, when we approximate functions with Taylor series. We know that e x x n =. Since the computer can n! n=0 only add a finite number of terms, we can only approximate e x.we must truncate the series, thus the name truncation error. (b) Round-off errors. When most numbers are stored in a computer, they are approximated. See the example of 1 3 above. (c) Propagated errors. This refers to an error in the succeeding steps of a process due to an earlier error. 2.2 Representation of Numbers in a Computer Different computers use slightly different techniques, but the general procedure is similar. In computers, floating-point numbers have three parts: the sign the fraction part, often called the mantissa. the exponent part, often called the characteristic. The three parts together have a fixed total length (32 or 64 bits). Of the three parts, the fraction part uses the most bits (24 or 52). That number determines the precision. The exponent uses 7 or 11 bits. That number determines the range of values. The sign uses one bit. Computers represent their floating point numbers in the general form where B is the base used. Usually 2, 10 or 16. ±.d 1 d 2 d 3...d p B e (1) p is the number of significant bits (digits), that is the precision. e is an integer exponent d i are digits ranging from 0 to B 1. The exponent is adjusted so that d 1 is never 0. Because working with binary or hexadecimal numbers is awkward, and because the purpose of these notes is to give the student an idea of how numbers are represented, rather than great details, we will only look at examples using base 10. Example 1 Suppose that B =10and p =4then 4

5 27.39 would be represented as would be represented as would be represented as Because the number of bits used to represent a number is fixed, there is a finite number of distinct values a computer can represent. Since there are infinitely many numbers, not all numbers can be represented. Many of them will be approximated. This is a source of error. 2.3 Absolute Versus Relative Error, Significant Digits Since numerical methods only give approximate answers, knowing how far a numerical answer is from the exact answer, that is knowing the error, is very important. There are two common ways used to express the size of the error. They are the absolute error and the relative error. Definition 2 Suppose that ˆp is an approximation for p. 1. The absolute error is E p = p ˆp 2. The relative error is R p = E p p Example 3 Find the absolute and relative errors if x = and ˆx =3.14 and E x = x ˆx = = R x = = Example 4 Find the absolute and relative errors if x = and ˆx = E x =4 and 4 R x = =

6 Example 5 Find the absolute and relative errors if x = and ˆx = and E x = = R x = =.25 The relative error is usually more useful than the absolute error, it gives the size of the error relative to the quantity being studied. Another term commonly used to express accuracy is significant digits. Definition 6 (significant digits) We say that a quantity ˆp approximates p to d significant digits if d is the largest positive integer such that R p < 5 10 d Example 7 If x = and ˆx = 3.14, then we find above that R x = Tofind how many significant digits we have, we solve < 5 10 d < 10 d ( 5 log ) < d < d d< The largest integer satisfying the inequality is 3, thus, our approximation is to 3 significant digits. 2.4 Uncertainty in Data This type of error occurs in the input stage. Data from real world applications contains errors in most cases. These errors are due to the way the data is collected. It may come from a measuring device, and no mechanical device is perfect. It is important to know the number of significant digits in the input data. Conclusions should never be reported with more significant digits than the input data. For example, suppose that p 1 = = and p 2 = = Both have four significant digits of accuracy. If we perform addition, the calculator would give p 1 + p 2 = = But if the computer uses 4 significant digits, the proper answer would be =

7 2.5 Chopping off and Rounding off This type of error occurs both at the input stage and at the algorithm stage. Definition 8 (normalized decimal form) Arealnumberp is said to be in normalized decimal form if p is written where 1 d 1 9 and 0 d j 9 for j>1. p = ±0.d 1 d 2 d 3...d k d k n (2) Example 9 The normalized decimal representation of is Example 10 The normalized decimal representation of is Remark 11 If a number is in normalized decimal representation, the number of significant digits is the number of digits after the decimal point. Since the computer can only use a finite number of digits to represent a number internally, some numbers cannot be represented entirely. Assuming that k is the maximum number of decimal digits a computer uses, then the digits d k+1,d k+2,... of number p shown in equation 2 will be ignored. For d k, there are two possibilities. Either p is chopped off, in which case d k is kept. Or, p is rounded off in which case d k is kept if 0 d k+1 < 5; d k is replaced by d k +1 if d k+1 5. Most computers used some form of rounding off. This problem may occur even when one does not expect it. Binary (see section below) numbers are used to represent numbers internally. Many decimal numbers may appear to have fewer significant digits than the computer allows. However, when they are translated to base 2, the new number may have too many digits, thus it will be approximated. Example 12 Let p = 22 = Assume a computer uses 7 six-digit representation. Then the normalized decimal form of p would be: if the computer uses chopping off if the computer uses round off. 2.6 Truncation Error This error occurs at the algorithm stage. It refers to errors introduced when a more complicated mathematical expression is replaced with a more elementary formula. Consider the problem of evaluating e x2 dx 7

8 Since we do not know an antiderivative for e x2, we can only approximate this integral. One method is to use a Taylor series representation for e x2. Such a representation is given by e x2 =1+x 2 + x4 2! + x6 3! x 2n = n! n= x2n n! +... To approximate the integral, we have to use a finite number of terms from this series, thus we truncate it. Suppose we use the first five terms for our approximation. Then, if we let p = e x2 dx and if we call ˆp the approximation for p, wehave ˆp = ) (1+x 2 + x4 2! + x6 3! + x8 dx 4! ) 1 = (x + x3 3 + x5 5 (2!) + x7 7(3!) + x9 2 9(4!) 0 = The exact value of this integral happens to be p = Tofind how many significant digits our approximation has, we solve: < 10 d < 5 10 d < 10 d ( 5 ) log 10 < d < d d< Thus, ˆp agrees with p to six significant digits, which we can tell if we look at the exact answer. 8

9 2.7 Loss of Significance This is a common problem when subtracting quantities which are almost identical. It occurs at the algorithm stage. Consider the two numbers p 1 = and p 2 = Both have 11 significant digits. However, p 1 p 2 = = = Thus, the answer only has five significant digits. significance or subtractive cancellation. Example 13 Let f (x) =x ( x +1 x ) and let g (x) = can see that f is algebraically equivalent to g. f (x) =x ( x +1 x ) = x ( x +1 x )( x +1+ x ) ( ) x +1+ x x (x +1 x) = x +1+ x x = x +1+ x = g (x) This is known as loss of x x +1+ x. We However, f contains a subtraction. If x is large, x +1 will be close to x, thus we should be able to verify loss of significance. If we use six digits and rounding, we have: ( ) f (500) = = 500 ( ) = and 500 g (500) = = The exact answer (of either computation) is which gives us g (500) as computed above, if we round it of to six significant digits. Thus, g gives us a better answer, there is no subtraction involved. 9

10 This is important, and should be remembered. If a quantity involves a subtraction between two quantities which are almost equal, identities should be used whenever possible to try to rewrite the quantity to eliminate the subtraction. This is illustrated in the problems at the end of the chapter. 3 Exercises Exercise 14 Write the numbers below in normalized decimal form, and find how many significant digits they have Exercise 15 Find the error E x and the relative error R x in each case below. In addition, determine the number of significant digits in the approximation. 1. x = and ˆx = x = and ˆx = x = and ˆx = Exercise 16 Loss of significance error can be avoided in some cases by rearranging terms in the function using a known identity. For each case below, explain why there might be a loss of significance, and rearrange the terms so that the loss of significance disappears. 1. ln (x +1) ln x for large x. 2. x 2 +1 x for large x. 3. cos 2 x sin 2 x for x π cosx 2 for x π. 4 Bibliography References [BF] Burden, L. Richard, & Faires, J. Douglas, Numerical Analysis, Fifth Edition, PWS Publishing Company, [HL] Hanselman, Duane & Littlefield, Bruce, The Student Edition of MATLAB Version 5, Prentice Hall,

11 [MF] Matthews, H. John, & Fink, D. Kurtis, Numerical Methods Using MAT- LAB, ThirdEdition, Prentice Hall, [S] Nakamura, Schoichiro, Numerical Analysis and Graphic Visualization with MATLAB, Prentice Hall,