Lecture notes Numerical Mathematics, First Course Autumn Sven-Åke Gustafson Høgskolen i Stavanger, Norway

Transcription

1 Lecture notes Numerical Mathematics, First Course Autumn 2003 Sven-Åke Gustafson Høgskolen i Stavanger, Norway August 22, 2003

2 Contents 1 On the representation of numbers in a computer Introduction Implementation of expressions on a computer Number systems in applied mathematical analysis Representing numbers in a computer Well-posedness and condition Gaussian elimination for Ax = b Introduction Trapezoidal systems Reduction of a general system of equations to trapezoidal form Linear spaces Dual systems in R n Dual systems Error analysis for linear systems of equations **Scalar products and orthogonality Iterative methods for Ax = b Introduction Jacobi and Gauss-Seidel iterations Least squares fit Normal equations On interpolation with polynomials Spaces of polynomials On polynomials as approximating functions The interpolation problem Interpolation formula with remainder On the choice of nodes Linear interpolation General interpolation formulas *Neville s scheme for calculating the value of the interpolating polynomial Q at a fixed point x **Romberg schemes **All powers **Even powers only

3 6 On approximation of functions Calculation of standard functions General remarks Division in simple computers The square root Elementary functions *Bernstein polynomials Taylor expansions Approximation by polynomials *General orthogonal expansions *Chebyshev expansions *Fourier series Splines Interpolation with piecewise polynomials Piecewise linear interpolation *Cubic spline interpolation *B-splines *Cubic Hermite interpolation On error propagation Absolute and relative errors in input data Error propagation during arithmetic operations Addition and subtraction Multiplication Division Monotonic functions of one variable Numerical example Estimate for a monotonic function with two continuous derivatives Mean-value theorem of integration Mean-value theorem of differentiation Error in estimate of extreme value of a function of one variable Numerical example of evaluating maximum value General error propagation formula for functions of several variables Example: addition Example: multiplication Example: division Numerical example Error propagation for functions of several variables Regularisation Numerical treatment of nonlinear equations Introduction Treatment of a single equation f(x) = 0, when f is continuous Determination of approximate roots. A general error bound The bisection method Newton-Raphson s method The fixed-point method

4 9 On the numerical evaluation of limit values Examples of limit values in computational mathematics Some standard definitions and results Order of magnitude O and o Arithmetic sequence Some measures of convergence speed of series and sequences Numerical summation of rapidly convergent series and sequences in the presence of round-offs Stability of term by term summation of series ** Stopping rule for term by term summation of rapidly converging sequences **Series which may be majorised by rapidly converging series Numerical integration with the trapezoidal and midpoint rules Introduction Constructing the trapezoidal rule and the midpoint rule with the method of undetermined coefficients Approximating general integrals with the trapezoidal and midpoint rules with step-size h Improving the accuracy of the trapezoidal rule by decreasing the step-size h **The Romberg scheme Numerical integration over general intervals The trapezoidal rule for a general table The trapezoidal rule for a finite interval, general functions and equidistant tables The trapezoidal rule for periodical functions, integral over one period The trapezoidal rule for the real line **Romberg extrapolation Numerical treatment of ordinary differential equations Introduction: Vector- and matrix-valued functions Initial- and boundary-value problems Numerical schemes based on discretisation for initial-value problems Euler s method ** Heun s method: ** Classical Runge-Kutta s ** The trapezoidal method: Examples in initial-value problems Euler s method **The trapezoidal method **Heun s method **The classical Runge-Kutta method **Difference methods for boundary-value problems

5 Abstract Numerical analysts study the algorithms of continuous mathematics, e. g. the solution of linear and nonlinear systems of equations, numerical evaluation of integrals and the treatment of differential equations. Hence linear and integer programming do not belong to numerical analysis. The goal of the numerical treatment of a mathematical problem is to determine a solution and to quantify the uncertainty of the calculated results. A major topic of numerical mathematics is the study of the issues arising by the use of computers. The result of this work is often represented in the form of packages of computer programs. Computers are used not only because the number of arithmetic operations is large but also because the computational schemes become large and complex. The user of the computer packages does not know all the details of the calculations to be performed, but (s)he needs to know the main principles. The research and development in applied mathematics often could be described as a three-phase process: Analysis of the problem under study. The result of this investigation may be represented in the form of theorems giving the mathematical properties of the solution sought Treatment of simple model problems, illustrating the theoretical results obtained Construction and documentation of computer programs solving the tasks above These lecture notes define the contents of Course TE149 which the author has taught in Stavanger most academic years in the period This is an introductory course and may even be looked upon as a user s course in the areas it introduces. It may not be sufficient for those aspiring to do professional work in the study and development of computational schemes. The course has not changed very much during this period reflecting the fact that knowledge of fundamental algorithms like the Gauss elimination for linear system of equations, iterative methods for linear and nonlinear systems of equations as well as classical methods for ordinary differential equations always will be needed for those using the results of computational mathematics. Note The contents of sections or subsections marked with * will not be tested in the exam for any students. Those marked with ** will not be tested in the exam for students, taking the shorter version of the course, for 5 European credits.

6 Chapter 1 On the representation of numbers in a computer 1.1 Introduction Computers may be used to process numbers in two different ways, namely as exact quantities which are processed by means of symbolic manipulations or as data which are subject to numerical evaluation, often involving approximation. Software packages called computer algebras perform symbolic manipulations. Examples of such packets are: Maple, Mathematica, Derive and Macsyma. There is a principal limit to the performance of these processors, since only a finite number of symbols may be stored in a given computer while it is possible to devise mathematical expressions of any length, e. g. s = ln(cos 2 ( n n)) e n. n=1 If we accept approximate values, then our calculations may be carried out by means of an algorithmic language like Matlab, Fortran77, Fortran90 or C. By writing a short program in any of these languages it would be easy to determine the sum above to any desired accuracy, say 8 decimal places, by direct summation. 1.2 Implementation of expressions on a computer Since a computer only may carry out a finite number of arithmetic operations and a finite number of logical choices the most general expression which may be evaluated by means of a computer is a piecewise rational function. The value of such an expression is affected by round-off errors, which may accumulate. This also means that e. g. all standard functions must be evaluated by piecewise rational functions. We now introduce Definition We distinguish between the following classes of errors: Round-off error, due to the approximation of a number to its computer representation and the fact that arithmetic operations are not carried out exactly. Data error if given parameters are not exactly known Truncation error: The replacement of an infinite sequence of operations by a finite one 1

7 Remark It is desirable to organise the computations such that the effects of round-offs and truncations are negligible in comparison to the effects of the uncertainties in given data. If e. g. data are given with an uncertainty of 1%, the calculations should be carried out such that the contributions of the round-offs and truncations are certainly less then 0.1%. 1.3 Number systems in applied mathematical analysis The following number systems are often encountered in applied mathematical work: integers rational numbers real numbers complex numbers We note that each of the listed classes of numbers is a subset of the following ones. Integers and rational numbers may be processed by means of symbolic manipulation, e. g. in Maple. Rational numbers are then represented in the form r = p/q where p and q are integers (q 0). Complex numbers are written in the form z = x + iy where x and y are reals. Our positional systems make it possible to represent any integer using a finite number of different symbols. This is a major difference from the Roman system which in principal requires infinitely many different symbols. Thus in the usual decimal system and in general any integer N is written 321 = N = sign(n)(a n a n 1... a 0 ) = sign(n) n a r 10 r r=0 where sign(n) is either + or, and each integer a r satisfies 0 a r 9, r = 0, 1,..., n. Hence we need only 10 different symbols and + and to define any integer N. It is customary to work with decimal fractions. Let u, 0 < u 1 be a real number. Then u may be represented as the sum of an infinite series, u = a r 10 r, 0 a r 9, a r integer. r=1 It is easily verified that this series converges. The numbers a r are uniquely determined provided that we only allow series with a finite number of nonzero terms. 2

8 Otherwise two distinct representations are indeed possible. We illustrate this with the example u = which has the two representations: u = 0.123, and u = It is also known, that if there are numbers n, p such that a r+p = a r, r n if and only if u is a rational number. Example: 4/7 = In practical work we approximate real numbers with decimal fractions of a finite length. There are the two possibilities fixed point numbers floating point numbers In fixed point numbers we work with a fixed number s of decimal places. Thus each number a is approximated by a number ā with s decimal places and by correct rounding we determine ā such that Example a ā s s = 3, a = 2, ā = s = 6, a = 2, ā = We next carry out our calculations with these rounded number and if necessary, the result of each operation is rounded as well. Example c = ab, a = π, b = e, c = ā b, c = c, correctly rounded. Thus: s = 3, ā = 3.142, b = c = c = s = 6, ā = , b = c = , c = Thus the rounding errors propagate in each step of the calculation and it is advisable to carry out intermediate calculations with one or two extra decimal places, so-called guarding digits. In the example above π e was not obtained with 6 correct decimals when each factor was rounded to this precision and the final result also was rounded to 6 decimals. In computational work one normally uses computers and calculators whose working accuracy is high, so that the effect of rounding errors can be ignored. Floating-point numbers are written in the form a = x 1 10 x2 where x 2 is an integer and x 1 a fixed-point number with s decimal places. Examples with s = 3: N A = , h = Here N A is Avogadro s number with the dimension atoms/k kmol and h is Planck s constant in J. If we choose the exponent x 2 such that 0.1 x 1 < 1 then the number a is said to be in normalised form. Thus we have for N A and h: N A = , h =

9 Definition Let ā be an approximation for a. We introduce: δa = ā a, the absolute error in ā as an approximation for a δa/ a = (ā a)/ a, the relative error in ā as an approximation for a, which is defined if a 0 Let now a satisfy a δa. Then we introduce the concepts a, the absolute uncertainty in ā as an approximation for a a/ a, the relative uncertainty in ā as an approximation for a, which is defined if a 0 Normally, the error itself δa is not known but an upper bound a i. e. the uncertainty is often available. Let now ā be an approximation to a given in normalised floating point format with s decimal places. Then a = a ā x2 s by correct rounding. Further we find: a/a = a ā a x2 s x 1 10 x2 = s x s The last inequality is a consequence of the fact that a is a normalised number and hence 0.1 x 1 < 1 Example N A = , N A /N A / h = , h/h / Representing numbers in a computer When working on a computer, several modifications to the presentation above are called for. The number s of decimal places is fixed for a given computer and software. Also the range of the exponent x 2 is limited such that there are two integers m, M such that m s M The number 10 which is the basis of the decimal system, is replaced by a positive integer B. The choices B = 2, 8 or 16 are most common. Thus for an integer N we have m N = sign(n)(b m b m 1 b 0 ) B = b r B r where the integers b r satisfy 0 b r < B. Thus the common decimal system has B = 10. Fixed-point and floating point numbers are introduced as before. Thus the result of correctly rounding to s B-mals gives the bound a ā 0.5 B s r=0 4

10 Example a = 1/3, B = 8, s = 4. We write 1 3 = b b b b ɛ where ɛ Multiplying both sides with 8 we get the relation: Here we take b 1 = 2 and obtain: 8 3 = b 1 + b b b ɛ = b b b ɛ 8 This relation is again multiplied by 8 giving: Now we put b 2 = 5 and arrive at 16 3 = b 2 + b b ɛ 82 Multiplying by 8 we find Thus b 3 = 2 and Finally we find This gives: 1 3 = b b ɛ = b 3 + b ɛ = b ɛ = b 4 + ɛ 8 4 b 4 = 5, ɛ = We next establish: We have namely: 1/3 = ( ) 8 ( ) = = 2 8 ( ) ( ) = (2/8 + 5/8 2 )(1/(1 1/64)) = 1/3 Remark If B = B p 0 with p an integer, then one digit in the B system corresponds to p numbers in the B 0 -system. Example B = 1000, p = 3, B 0 = 10 π = = 3 B B B B 3 We also find that one digit in the system with B = 16 corresponds to 4 digits for B = 2 We now introduce 5

11 Definition Two real numbers a and b are said to be computationally equivalent, if they have the same representation in a given computer We realise that whether or not two given numbers are computationally equivalent depends on the working accuracy of the computer used. The definition of computational equivalence is extended to vectors, matrices and functions in an obvious way. Example Consider a computer with working relative accuracy Define the functions f and g according to f(x) = exp(x), g(x) = 12 r=0 x r r!. 1 x 1 Then the exponential function f and the polynomial g are computationally equivalent on the interval [ 1, +1] We next observe that there are only finitely many different computer representations of a real number on a given computer with a pre-defined working accuracy but there are infinitely many real numbers. This means that there are infinitely many reals having an identical computer representation. This conclusion may be generalised to classes of computational problems as illustrated by Example Let A be a matrix, b, c and x compatible vectors. Consider the task of evaluating v = b T x, when Ax = c There are infinitely many different matrices and vectors A, b, c having identical computer representations and they could conceivably define different values of the answer v. This is not desirable. 1.5 Well-posedness and condition We introduce Definition A problem P is said to be well-posed in the sense of Hadamard if: It has a solution The solution is unique The solution depends continuously of input data If the problem P does not have the properties above, it is said to be ill-posed in the sense of Hadamard Remark If a problem P does not have a unique solution, this may be remedied by introducing additional conditions. If the example above has many solutions, one may introduce the further constraint that one should choose the vector x satisfying Ax = c which has the smallest norm. The input data belongs to a certain set and here continuity may be defined by introducing a distance function suitable for the problem at hand. Thus the well-posedness of a problem depends on the distance function chosen. 6

12 Within the class of well-posed problems some are more sensitive to changes in input data than others. To define this sensitivity we introduce the measure of the condition number which gives the quotient between the change of the output (answer) and the change of the input of a problem caused by a generally small perturbation of input data. The numerical value of the condition number depends on the measures chosen for the changes of input and output. Since the perturbation of the input is generally small, we may linearise the change of the output. We illustrate with Example We determine the condition number of calculating the function We have f(x) = x 2 f (x) = 2x and hence the change δf = 2xδx corresponds to the perturbation δx. Hence the condition number of the problem of evaluating f(x) is 2 x, if we measure in absolute errors. If we instead use relative errors, then the relative size of the perturbation is δx/x and the relative change of the output becomes δf/f = 2 δx/x and the condition number is thus 2 for all values of x 7

13 Chapter 2 Gaussian elimination for Ax = b 2.1 Introduction The Gauss elimination is based on systematic use of the following Lemma Let x = (x 1,..., x n ) T R n be a column vector, f 1 (x), f 2 (x) be functions having x as argument. Let finally c 0 be a constant. Then the following two systems of equations have the same solution sets: f 1 (x) = 0, f 2 (x) = 0 (2.1) and f 1 (x) = 0, f 1 (x) + c f 2 (x) = 0 (2.2) Proof: If x satisfies (2.1) then it immediately follows that x satisfies (2.2). On the other hand if f 1 (x) = 0, f 1 (x) + c f 2 (x) = 0 we first see that f 1 (x) = 0 giving that c f 2 (x) = 0 and since c 0 we find that f 2 (x) = 0 as well, proving the lemma. We illustrate an application of the lemma with the following Example Solve the linear system 3x + y = 1 (2.3) 2x y = 4 (2.4) Using the lemma above we conclude that the system (2.3),(2.4) has the same solution set as the system (2.5) below for all nonzero values of c: 3x + y = 1; (2 + 3c)x + (c 1)y = (4 + c) (2.5) We now determine c such that 2 + 3c = 0, i. e. c = 2/3 and hence arrive at the system 3x + y = 1; 5/3 y = 10/3 (2.6) This last system is of triangular form and we find easily y = 2 and subsequently x = 1 This procedure is applied to general system of equations to transform them into systems of a special structure whose solution set is easily determined. This scheme which is called Gaussian elimination with pivoting is described in section 2.3 8

14 2.2 Trapezoidal systems Definition Let A be a rectangular matrix with n rows and m columns, x and b column vectors with m and n elements respectively. Then Ax = b specifies a linear system of equations with n equations and m unknowns. b is the right hand side, A the table of coefficients and x is a solution vector to be determined. The system may be inconsistent, have a unique solution or have infinitely many solution vectors. If n = m, the matrix A is called square. The element in row r and column k is denoted a r,k The matrix A is called trapezoidal or in echelon form if it is such that: There is an integer k with the properties (1) and (2) below: (1) a i,i 0, a i,j = 0, j < i, i = 1,..., k. (2) If k < n then a i,j = 0, i > k Important special case: If m = n = k then A is said to be triangular. Example A = B = C = All the three matrices above are trapezoidal. Matrix A is even triangular. Consider now the three linear systems of equations Ax = b, By = b, Cz = b Then we easily verify: The system Ax = b has a unique solution for all right hand sides b. The system By = b has infinitely many solutions since y 4 may be chosen arbitrarily and then the remaining elements in the solution y may be expressed uniquely in terms of y 4. The system Cz = b is solvable only if b 3 = 0. In this case z 3 may be chosen arbitrarily and the remaining components of the solution vector may be expressed uniquely in terms of z 3. Definition The following two classes of square matrices are useful: square matrix A is termed symmetric if The a i,j = a j,i A is called tri-diagonal if a i,j = 0, i j > 1 Provided that the table of coefficients is of trapezoidal form, it is possible to determine all solutions of the system Ax = b by means of a finite number of arithmetic operations. 9

15 2.3 Reduction of a general system of equations to trapezoidal form If A is of trapezoidal form, then the system Ax = b is such that the coefficient a i,j in front of the unknown x j in equation number i is 0 if i > j. The Gaussian elimination scheme may be used to reduce a general system of equations on a system of the trapezoidal form which has the same solution set as the original system. To secure stability, pivoting, i.e. reordering of the equations, is performed according to certain rules to be described below. For certain classes of matrices pivoting is not required. Also, if the elimination is carried out exactly, as e.g. in theoretical analysis, pivoting is only carried out to avoid division by 0. We describe now the Gaussian elimination for a general system Ax = b where A has n rows and m columns. The main idea is that by performing elimination step number k one eliminates the coefficient in front of the unknown x k in equations i > k for k = 1,..., min(m, n). It is essential that the coefficient in front of x k in equation k remains different from 0. The algorithm may be described as follows: (In order to simplify the notation we denote by a i,j the contents of the place in row i and column j, which could be thought of as a space in the computer. Step 1: Elimination of coefficients in front of x 1 : Substep 1a: Exceptional case: If all element in column 1 are 0, find another column which has a nonzero element and interchange it with the first column. (Observe that this interchange means a reordering of the unknowns, which should be recorded) If there is no such column, A is already in the trapezoidal form (with all elements zero) and the algorithm is stopped. Substep 1b: Let a l,1 be an element in column 1 with the largest absolute value. (In case of tie any of the elements with largest absolute values will do) Equation number l is called the pivot equation. Substep 1c: If l 1, interchange equations number 1 and l, including the right hand side. (This interchange is called pivoting) Substep 1d: Eliminate the coefficients in front of x 1 in equations number 2,..., n by subtracting suitable multiples of the pivot equation from the other equations. The transformation of the coefficients in equation number i is performed according to the formulas: m i = a i,1 /a 1,1, a new i,j = a old i,j m i a 1,j An analogous formula gives the transformation of the right hand side. This operation is carried out for equations 2, 3,..., n Step number k: The coefficients in front of x k are eliminated in equations number k + 1,..., n by means of the following substeps: Substep k(a): Exceptional case: If all element in column k and rows number k,..., n are 0, find another column which has a nonzero element in one of these rows and interchange it with the first column. (Observe that this interchange means a reordering of the unknowns, which should be recorded). If there is no such column, A is already in the trapezoidal form, and the algorithm is stopped. Substep k(b): Let a l,k be an element in column k with l k and having the largest absolute value. (In case of tie any of the elements with largest absolute values will do) Equation number l is called the pivot equation. Substep k(c): If l k interchange equations number k and l, including the right hand side. (This interchange is called pivoting) Substep k(d): Eliminate the coefficients in front of x k in equations number k + 1,..., n by subtracting suitable multiples of the pivot equation from the other equations. The transformation of the coefficients in equation number i is performed 10

16 according to the formulas: m i = a i,k /a k,k, a new i,j = a old i,j m i a k,j An analogous formula gives the transformation of the right hand side. This operation is carried out for equations k + 1, k + 2,..., n Example We want to determine the polynomial of degree < 4 which interpolates the function exp(t) at the 4 points 1, 0.5, 0.5, 1. Write the polynomial Q(t) = a 1 + a 2 t + a 3 t 2 + a 4 t 3 The interpolation conditions give the following linear set of equations with the coefficients a 1,..., a 4 as unknowns: We eliminate the coefficients in front of a 1 in the second and following equations by subtracting the first equation from the others. The system then takes the form The coefficient in front of a 2 has the largest absolute value in equation 4 and hence this equations is interchanged with the second one, i. e. we perform a pivoting operation before carrying out the elimination: After the interchange we eliminate the coefficient in front of a 2 in the third and fourth equations. Thus we subtract the second equation multiplied by 1.5/2 = 0.75 from the third equation and by 0.5/2 = 0.25 from the fourth equation Before the last elimination step no pivoting is required and we obtain a triangular system after subtracting the third equation from the fourth Now back-substitution gives the solution: a 1 = , a 2 = , a 3 = , a 4 =

17 2.4 Linear spaces We recall that a set of numbers S such that the operations of addition, subtraction, multiplication and division are defined and satisfying the laws we are used to from real numbers is called a set of scalars. Other examples of scalars are the complex numbers and the rational numbers. Definition E is called a vector space (or linear space) over the set of scalars F, if vector addition and multiplication by scalars are defined, satisfying the laws we remember from the familiar example of R n, the space of ordered n-tuples Example Let S be a set, E the set of real-valued functions defined on S We next define linear combinations as follows (af 1 + bf 2 )(s) = af 1 (s) + bf 2 (s) where a, b are scalars. Thus f 1, f 2 are elements of E (vectors) and the new vector (af 1 + bf 2 ), the linear combination of f 1 and f 2 is defined by using the laws of real numbers. This general example may be specialised in different ways to give familiar instances of linear spaces. If S consists of n distinct elements, we get a space like R n, the space of n-tuples. We may also consider spaces of continuous functions defined on an interval. A particular instance is the linear space of polynomials. Definition Let E be a linear space and let the subset B E be such that B is linearly independent and each element x E is a linear combination of elements in B. Then B is called a basis of E It can be shown that all linear spaces have a basis. In general, a given linear space has more than one basis. If a linear space has a basis with n elements, where n is a finite number, then all bases have the same number n of elements and n is called the dimension of the space. We next consider functions defined on linear spaces: Definition Let X, Y be linear spaces, f a function mapping elements of X on elements of Y. We write this which may also be written f : X Y f(x) Y, x X Instead of the word function we may use one of the words mapping, or operator. In particular, if the space Y is a a set of scalars we use the word functional. f is said to be linear, if f(ax 1 + bx 2 ) = af(x 1 ) + bf(x 2 ), x 1 X, x 2 X, a, b scalars Example Let X be C[ 1, 1] the linear space of functions, continuous on [ 1, 1]. Define I, T, δ and F by ( ) 1 (f( 1) + f(1)) 19 I(f) = f(t) dt, T (f) = f( 1 + r 0.1) 2 1 δf = f(0), F (f)(x) = 1 1 r=1 exp(xt)f(t) dt Then all of I, T, δ and F could be said to be linear operators, in particular I, T, δ are linear functionals 12

18 Definition Let E be a linear space. The functional f is called a norm on E, if for all f, g E and scalars a f 0, f E f = 0 implies f = 0 af = a f f + g f + g Example Let X be C[ 1, 1] the linear space of real-valued functions, continuous on [ 1, 1]. Then we often use the following norms f = max 1 t 1 f(t) f 2 2 = 1 1 (f(t))2 dt f 1 = 1 f(t) dt 1 If we now consider the special example f(t) = t, we find 2.5 Dual systems in R n Dual systems f = 1, f 2 = 2/3, f 1 = 2 We consider R n, the space of n-tuples. In this space linear functionals are represented by scalar products n l(x) = c T x = c r x r and operators by square matrices. Definition (Dual systems) Let A be a square matrix and b, c, x, y be vectors. We introduce the two problems: Determine b T x when Ax = c and Determine r=1 c T y when A T y = b We will only consider the case when the two linear systems have unique solutions for all right hand sides c, b. Then we have the following duality result Theorem Let A be a square matrix and b, c, x, y be vectors. If A is such that the system Ax = c has a unique solution x for each c, we define the linear functional d(c) = b T x. If b allows the representation we have b = A T y, d(c) = c T y 13

19 Proof: d(c) = b T x = (A T y) T x = y T Ax = y T c This proof may also be carried out by calculating componentwise: We have and Hence d(c) = n b i x i = i=1 n d(c) = b i = n x i i=1 r=1 n b i x i i=1 n a ri y r r=1 a ri y r = n n y r r=1 i=1 x i a ri = n y r c r Remark This simple result is very useful in many contexts and allows generalisations in various directions. In R n we consider the two linear systems Ax = c, A T y = b and efficient methods for solving both at the same time are available. We note that evaluating the functional d(c) for various values of of c may be looked upon as a generalisation of the task to tabulate d. We also establish Theorem Let c = c + ɛ. Then we have Proof: We have d( c) d(c) = y T ɛ d( c) = b T x, A x = c + ɛ. d(c) = b T x,, Ax = c Subtracting the last relation from the preceding one we get d( c) d(c) = b T ( x x) = (A T y) T ( x x) = y T A( x x) = y T ( c c) = y T ɛ. Thus we may estimate the change in the value of the functional d(c) caused by a perturbation of c without studying the inverse of A Error analysis for linear systems of equations In many practical situations the numerical treatment of a linear system of equations gives an approximate solution vector which is far from the exact solution, but the residual is small. However, we do not seek the solution vector itself, but need to enter it into some formula to obtain the end result. If we need to evaluate a linear functional to find the desired result, we may use the theorem above to estimate the error in our final result which may be much smaller than the error in the calculated solution vector. We note that we may estimate the error in component x i by putting b = (0, 0,..., 1, 0,..., 0) T, i. e. by putting the i-th component equal to 1 and all the others to 0. It is worth mentioning that these error bounds are obtained without estimating the norm of the inverse of the square matrix A. However, if we need to estimate the errors in all the components of the calculated solution vector, then we need to perform a computational effort equivalent to inverting A r=1 14

20 2.6 **Scalar products and orthogonality Definition let A, B be sets. Then we define the mew set C = A B = {(a, b) a A, b B} The set C is called the Cartesian product of the sets A, B We next introduce Definition Let E be a linear space over the real numbers. A scalar product on E is a functional < x, y > on E E with the properties < ax 1 + bx 2, y >= a < x 1, y > +b < x 2, y >, a, b scalars < x, y >=< y, x > < x, x > 0 < x, x >= 0 = x = 0 Thus < x, y > is linear in each of its arguments x, y. Definition We often use the norm x 2 = < x, x > Example Let x R n, y R n, d r > 0, r = 1,..., n. Then we define We also give < x, y >= n x r y r d r r=1 x 2 2 =< x, x > Example Let E = C([ 1, 1]) be the linear space of functions, which are continuous on [ 1, 1]. Let further w E be such that w(t) > 0, 1 t 1 We may now introduce the scalar product < f, g >= 1 1 f(t)g(t)w(t) dt and norm f 2 2 = 1 1 (f(t)) 2 w(t) dt Definition Let E be a linear space with scalar product. Then x is said to be orthogonal to y (written x y), if < x, y >= 0. Let further B E be a basis such that x B, y B, x y = < x, y >= 0, then B is said to be an orthogonal basis to E. If in addition < x, x >= 1, x B, then B is termed an orthonormal basis to the linear space E 15

21 Chapter 3 Iterative methods for Ax = b 3.1 Introduction We want to find an approximation to the smallest positive root s of the equation x = x 6 (3.1) Assuming the 0 < s < 1 we get the first approximation x 0 = 0.2 Entering this into the right hand side we get the next approximation 6 x 1 = x 0 and the next and hence the general relation x 2 = x 1 6 x r+1 = x r 6 which generates the infinite sequence x 0, x 1,... which can be shown to converge towards the root s. Using Maple and working with 10 decimal places we find the numerical values x 0 = 0.2, x 1 = , x 2 = , x 3 = and hence we accept x 3 as our approximation for s. This an example of an iterative scheme. In principle, the sequence of approximation is infinite but we want to find a good approximation to the limit value after a finite number of steps. It is then necessary to estimate the error in the accepted approximation. Iterative schemes have been developed for many classes of problems, including systems of linear and nonlinear equations. The present chapter will be devoted to describing two classical schemes for systems of linear equations. 3.2 Jacobi and Gauss-Seidel iterations We consider again the linear system Ax = b which we write on the form x = Bx + f. 16

22 The latter is solved by iterations. We discuss only two different methods, namely the Jacobi and the Gauss-Seidel schemes. By Jacobi iterations we form the sequence: x l+1 = Bx l + f, l = 0, 1,... where the starting approximation may be arbitrary. Often one takes x 0 = 0. The elements of x l are updated according to x l+1 i = n b i,j x l j + f i j=1 This process is easily parallelized. The Gauss-Seidel iterations differs from the Jacobi ones, in that one uses the latest available components of x to make the next update. Thus one updates the components of x one at a time beginning with x 1. When x 2 is updated one uses the just calculated value of x 1 together with the earlier values of the remaining components. To update x 3 one uses the recent values of x 1, x 2 together with earlier values for x 4,... and so on. The two iteration schemes do not converge for all matrices B but the following condition is sufficient for convergence of both methods: max i n b i,j < 1 j=1 Example We solve the system Ax = b where A = b = We write this system in the form x = Bx + f by solving the first equation with respect to x 1, the second with respect to x 2 and the third with respect to x 3. Then we find B = b = We compare the results of Jacobi and Gauss-Seidel iterations. The following output was obtained from simple Fortran programs implementing these methods and using the null vector as starting vector. given system starting vector Jacobi approximates

23 given system starting vector Gauss-Seidel approximates

24 Chapter 4 Least squares fit 4.1 Normal equations Definition Let A be a rectangular matrix with n rows and m columns, x and b column vectors with m and n elements respectively. Then Ax = b specifies a linear system of equations with n equations and m unknowns. b is the right hand side, A the table of coefficients and x is a solution vector to be determined. The system may be inconsistent, have a unique solution or have infinitely many solution vectors. We discuss here the case when n > m. Then the system is, as a rule, inconsistent, i. e. one cannot find any vector x such that Ax = b exactly. Instead, one seeks a vector x which minimizes the square of the length of the deviation vector. Hence we want to minimize (Ax b) T (Ax b). This problem always has a solution, which satisfies the normal equations, A T Ax = A T b. This system may be solved using Gaussian elimination, and it can be verified that pivoting is not required for stability. We describe next how to form the elements in the matrix C = A T A and vector f = A T b. Using the definitions of matrix by matrix product and matrix by vector product we find the formulas: c i,j = n a l,i a l,j f i = l=1 n a l,i b l. Thus the element c i,j is the scalar product of columns number i and j of matrix A, while f i is the scalar product of column number i of matrix A and the original right hand side b. Since the resulting system A T Ax = A T b, often called the normal equations, always is consistent, it may have a unique solution or infinitely many solution vectors. Example We consider the same example as in Chapter 2, namely the task to approximate exp(t) with a polynomial of degree less than 4. Now we do least squares fit with respect to the set of the five points 1, 0.5, 0, 0.5, 1. Thus we get an over-determined linear system with 5 equations and 4 unknowns. l=1 19

25 The following output was obtained given system normal equations: triangular system solution vector

26 Chapter 5 On interpolation with polynomials 5.1 Spaces of polynomials On polynomials as approximating functions In this chapter we will discuss some linear spaces often encountered in numerical work. As written earlier, we need to approximate a given function f with an expression g, which may be evaluated with a finite number of arithmetic operations and logical choices. Sometimes we want to perform further operations, like integration or derivation on g and it is an advantage if these latter operations can be carried out easily, or even analytically. As an example, assume that we have constructed g such that f(t) g(t), 1 t 1 We seek an approximation for the integral of f and make the approximation 1 1 f(t) dt 1 1 g(t) dt. Hence it is desirable that the integral on the right hand side may be evaluated more easily than that on the left hand side. A common choice it to construct a polynomial g to approximate f. In the sequel we will present some systematic methods for doing this. We will need to describe some useful properties of polynomials. We begin with Example Let E 5 be the linear space of polynomials of degree < 5. We want to give some examples of bases to this space which has the dimension 5. Let now {t 1, t 2, t 3, t 4, t 5 } be 5 distinct real points such that Put and t 1 < t 2 < t 3 < t 4 < t 5 L i (t) = P (t) = 5 (t t i ) i=1 P (t) (t t i ), i = 1,..., 5 Then the following three sets of functions are bases for the space E 5 : 21

27 1, t, t 2, t 3, t 4 L 1, L 2, L 3, L 4, L 5 where See [5], page 104. T 0, T 1, T 2, T 3, T 4 T 0 (t) = 1, T 1 (t) = t, T r+1 = 2 t T r (t) T r 1, r = 1, 2, The interpolation problem We next establish the following result: Theorem Let n distinct real points t i, and n numbers y i, i = 1,..., n be given. Then there is a unique polynomial Q of degree < n, such that Q(t i ) = y i, i = 1,..., n (5.1) Proof: Existence of interpolating polynomial Q: Form the sequence of polynomials according to N 1 (t) = 1, N i+1 = (t t i ) N i (t), i = 1,..., n 1 Next put Q(t) = n c i N i (t) i=1 Next determine the coefficients c i such that Q(t i ) = y i, i = 1,..., n We arrive at a linear system of equations of triangular form and the terms on the diagonal are different from 0. Thus this system may be solved, defining an interpolating polynomial Q. We next prove uniqueness: Let Q, R be two interpolating polynomials of degree < n Next set S(t) = P (t) R(t) Thus S is a polynomial of degree < n such that S(t i ) = 0, i = 1,..., n Thus S is of degree < n having n distinct zeros. identically 0, establishing uniqueness Interpolation formula with remainder This is only possible, if S is Theorem Let I = [a, b] be a real interval, f be a function defined on I having n continuous derivatives there and let t 1 < t 2 <... < t n be n distinct points in I. Let also Q be the polynomial of degree < n satisfying Then we establish where ξ depends on x Q(t i ) = f(t i ), i = 1, 2,..., n f(x) = Q(x) + K(x)P (x), P (x) = n (x t i ), K(x) = f n (ξ), n! i=1 22

28 Proof: If x coincides with one of the points t i, it is seen at once that the statement is true. We treat next the case when x is different from all of these n points. Let now x be fixed. Then we have f(x) = Q(x) + K(x)P (x), K(x) = Next form the new function F, given by F (t) = f(t) Q(t) K(x)P (t) f(x) Q(x) P (x) We verify that F is 0 at t 1,..., t n, x and hence F has in total n+1 zeros. By Rolle s theorem F (n) has a zero at a point which we denote ξ. Differentiating n times with respect to t we arrive at F (n) (t) = f (n) (t) K(x) n! Using the fact that F (n) (ξ) = 0 we finally get as claimed. K(x) = f (n) (ξ), n! Remark Only in special cases are the higher order derivatives available as useful formulas, a fact which often causes difficulties in practical situations. 5.2 On the choice of nodes If t i in (5.1) are the scaled Chebyshev points given by then t i = (a + b) 2 + (b a) 2 f(t) Q(t) 2 cos(θ i ), θ i = π(i 1/2), i = 1, 2,..., n. n (b a)n 4 n max a t b f (n) (t) /n! The numerical interpolation problem generally becomes more stable if a change of variable is performed such that the interpolation interval becomes [ c, c]. Often c is taken to 1. Thus if t is the original variable which is in the interval [a, b] we introduce the new variable u in the interval [ 1, 1] by means of the formula t = (a + b) Linear interpolation + (b a) u. 2 Take n = 2 in (5.1). One verifies directly that Q may be written in one of the two equivalent forms and the remainder becomes: Q(t) = f(t 1 ) + (t t 1 ) f(t 2) f(t 1 ) (t 2 t 1 ) Q(t) = f(t 1 ) (t 2 t) (t 2 t 1 ) + f(t 2) (t t 1) (t 2 t 1 ) (5.2) (5.3) R(t) = (t t 1 )(t t 2 )f (ξ)/2, t 1 ξ t 2. 23

29 Using the fact that (t t 1 )(t t 2 ) (t 2 t 1 ) 2 /4 we get the error bound for linear interpolation R(t) (t 2 t 1 ) 2 /8 max f (t), if t 1 t t 2 t 1 t t General interpolation formulas The interpolation problem may always be solved by using the interpolation conditions to formulate a linear system of equations whose solution defines the interpolating formulas sought. This approach is illustrated by Example Since the structure of this system is special its solution may also be obtained using special formulas, associated with the names of Lagrange and Newton. The use of these formulas is illustrated in Example below. Theorem (Lagrange s formula) Let the n nodes t i be distinct. Then Lagrange s formula for the interpolating polynomial Q reads Proof. Set Q(t) = n P (t) n f(t i ) (t t i )P (t i ), P (t) = (t t i ). (5.4) i=1 Using L Hôpital s rule we find p i (t) = P (t) t t i. lim p i (t) = P (t i ). t t i For k i, we conclude, that p i (t k ) = 0. Since the points t i are distinct, we have that P (t i ) 0. Also, p i is a polynomial of exact degree n 1. Thus (5.4) defines the interpolating polynomial as claimed. Divided differences: One argument: Two arguments: k arguments: f[t i ] = f(t i ), i = 1, 2,..., n. f[t i, t j ] = f[t j] f[t i ]. (t j t i ) i=1 f[t i1,..., t ik ] = f[t i 2,..., t ik ] f[t i1,..., t ik 1 ] t ik t i1. Theorem (Newton s formula) Let f be defined on [a, b] and let a t 1 < t 2 <... < t n b. Define Q by (5.1). Then the following relations hold: Q(t) = f[t 1 ] + (t t 1 )f[t 1, t 2 ] + (t t 1 )(t t 2 )f[t 1, t 2, t 3 ] (t t 1 )(t t 2 )... (t t n 1 )f[t 1,..., t n ]. f(t) = Q(t) + P (t)f[t, t 1, t 2,..., t n ]. Proof: Let t be a fixed point in [a, b]. We will use the definition above of divided differences and also the fact, that a reordering of the arguments does not change the value of a given divided difference. Since f[t 1 ] = f(t 1 ), f[t 1, t] = f[t] f[t 1]. (t t 1 ) 24

30 we have Next giving and We now write to obtain resulting in f(t) = f[t 1 ] + (t t 1 )f[t 1, t]. f[t 1, t 2, t] = f[t 1, t] f[t 1, t 2 ] t t 2 f[t 1, t] = f[t 1, t 2 ] + (t t 2 )f[t 1, t 2, t] f(t) = f[t 1 ] + (t t 1 ]f[t 1, t 2 ] + (t t 1 )(t t 2 )f[t 1, t 2, t]. f[t 1, t 2, t 3, t] = f[t 1, t 2, t] f[t 1, t 2, t 3 ] t t 3 f[t 1, t 2, t] = f[t 1, t 2, t 3 ] + (t t 3 )f[t 1, t 2, t 3, t] f(t) = f[t 1 ] + (t t 1 )f[t 1, t 2 ] + (t t 1 )(t t 2 )f[t 1, t 2, t 3 ] + (t t 1 )(t t 2 )(t t 3 )f[t 1, t 2, t 3, t] These operations are continued in an analogous manner to give the desired result. Example For the special function f(t) = 1/(1 xt), x constant, we have for general t i : 1 1 xt = Q(t) + P (t) (1 xt)p (1/x). We give now a numerical example illustrating the use of Lagrange s and Newton s interpolation formulas: Example The following table of the function f is given x f(x) Construct the polynomial of degree < 4 which interpolates f at the 4 points a) Use Lagrange s interpolation formula b) Use Newton s interpolation formula with divided differences. a) Lagrange s formula We get: Q(x) = x(x 1)(x 2) 4.6 ( 2 0)( 2 1)( 2 2) + (x + 2)x(x 2) 4.0 (1 + 2)1(1 2) Q(x) = 4.6 x(x2 3x + 2) x(x2 4) (x + 2)x(x 1) (2 + 2)2(2 1) + 3.0(x + 2)(x 1)(x 2) (0 + 2)(0 1)(0 2) (x 1)(x2 4) x(x2 + x 2) 8 = 0.6x x