Iterative solvers for nonlinear equations

Transcription

1 I Math Institute of Mathematics Iterative solvers for nonlinear equations Dr. Svetlana Tokareva 10/03/2016

2 Motivation Example (State equation of a gas) We want to determine the volume V occupied by a gas at temperature T and pressure p. The state equation that relates V, p and T is ( ( N ) ) 2 p + a (V Nb) = knt, V where a and b are gas-specific constants, N is the number of molecules in V and k is the Boltzmann constant. We need to solve the nonlinear equation with respect to V. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 2

3 Example (Population dynamics) The equation x + = xr(x) links the number of individuals in a generation x and in the following generation. Function R(x) models the variation rate of the population and can be modeled e.g. as: r R(x) =, r > 0, K > 0; 1 + xk rx R(x) = 1 + (x/k ) 2 etc. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 3

4 Nonlinear equations Model problem We consider a scalar nonlinear equation f (x) = 0, x [a, b] R. (1) Assume that f is continuous on [a, b], denoted f C[a, b]. Denote a solution of (1) by x. Solution x is called root of the equation (1), or zero of function f. Geometric interpretation: the roots of equation (1) are intersections of the graphs of y = f (x) and y = 0. In general, equation f (x) = 0 might have more than one root or no roots at all! University of Zurich, I Math 10/03/2016 Nonlinear equations Page 4

5 Example 1 We solve x 1 = 0. f (x) = x 1, x [0, 2]. This is a linear equation with one root: x = f(x) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 5

6 Example 2 f (x) = sin(x), x R. We solve sin(x) = 0. This is a nonlinear equation. Since sin(πn) = 0 for any integer n, we have on the interval [ π 2, 3π ] there is one root; 2 on the interval [0, 4π] there are five roots f(x) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 6

7 Example 3 f (x) = x 3 30x , 0 x 20. We solve x 3 30x = 0. f (x) is a cubic polynomial. In general, there are at three complex roots. How many real roots? From the plot we see that x is the only real root on the interval [0, 20] f(x) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 7

8 Example 4 f (x) = 10 cosh(x/4) x, 10 x 10, where cosh(x) = ex +e x 2 is a hyperbolic cosine. We solve 10 cosh(x/4) x = 0. From the plot we see that there are no roots in the interval [ 10, 10] f(x) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 8

9 Roots localization Definitions The root x of equation f (x) = 0 is called simple, if f ( x) = 0 and f ( x) 0. Otherwise, if f (k) = 0 for k = 1, 2,..., m 1 and f (m) 0, the root x is called multiple with multiplicity m. The interval [a, b] containing only one root x of (1) is called the localization interval of x. Theorem Let f be a continuous function on [a, b] and f (a) f (b) < 0. Then the interval [a, b] contains at least one root of the equation f (x) = 0. Remark Theorem applies for simple roots and roots with odd multiplicity (m = 1, 3, 5,... ). Unfortunately, for roots with even multiplicity (m = 2, 4, 6,... ) it is not possible to localize the roots using this theorem. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 9

10 Example Localize the roots for the quadratic equation a 2 x 2 + a 1 x + a 0 = 0 with different a 0, a 1, a 2 for x [ 1, 1]. x 2 = 0 x 2 x 1 = 0 x = one root with m = 2 one root on [ 1, 1] two roots x 1 = x 2 = 0 x x 1,2 ± no localization possible I loc = [ 0.71, 0.52] I (1) loc = [ 0.73, 0.65], = [0.67, 0.78] I (2) loc University of Zurich, I Math 10/03/2016 Nonlinear equations Page 10

11 Numerical methods For a general nonlinear equation, the roots cannot be found in a finite number of steps and therefore iterative methods are widely adopted. bisection method fixed-point iterations method Newton s method modifications of Newton s method Newton-Hörner method for polynomial roots Problems: Choice of initial guess x (0)!!! Convergence criteria. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 11

12 Bisection method Let f be a continuous function on [a, b] satisfying f (a) f (b) < 0 and assume that f has a unique root on [a, b]. The strategy of the bisection method is to halve the given root localization interval and proceed with the subinterval for which f has different signs at endpoints x 2 x 1 x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 12

13 Denote a (0) = a, b (0) = b, I (0) = [a (0), b (0) ] and choose the mid-point x (0) = (a (0) + b (0) )/2 as the first guess value for x. The approximation error of x (0) doesn t exceed half of the interval length: x (0) x (b (0) a (0) )/2. x a = a (0) x (0) b = b (0) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 13

14 Next, choose I (1) = [a (1), b (1) ], where I (1) is either [a (0), x (0) ] or [x (0), b (0) ], satisfying the condition f (a (1) ) f (b (1) ) 0. Choose the mid-point x (1) = (a (1) + b (1) )/2 as the next approximation to the root with accuracy x (1) x (b (1) a (1) )/2 = (b a)/2 2. x a (1) = a (0) x (1) b (1) = x (0) b = b (0) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 14

15 Algorithm At each step k 1 we select the subinterval I (k) = [a (k), b (k) ] of the interval I (k 1) = [a (k 1), b (k 1) ] as follows. Given x (k 1) = (a (k 1) + b (k 1) )/2, if f (x (k 1) ) = 0 then x = x (k 1) and the method terminates; otherwise, if f (a (k 1) ) f (x (k 1) ) < 0, set a (k) = a (k 1) and b (k) = x (k 1) ; if f (x (k 1) ) f (b (k 1) ) < 0, set a (k) = x (k 1) and b (k) = b (k 1). Then define x (k) = (a (k) + b (k) )/2 and proceed to the next iteration until the required accuracy is reached. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 15

16 Accuracy The mid-point x (k) of the k-th subinterval I (k) gives an approximation to the root x with the error estimate x (k) x (b (k) a (k) )/2 = (b a)/2 k+1, (2) i.e. the iterative process converges as the geometric progression with the common ratio r = 1/2. In order to guarantee that the error e (k) = x (k) x < ε it is sufficient to do k min iterations with ( ) b a k min > log 2 1. (3) ε University of Zurich, I Math 10/03/2016 Nonlinear equations Page 16

17 Fixed point iterations The fixed point iterations method is an iterative method for finding the roots of the nonlinear equation x = ϕ(x). (4) If the solution x of (4) exists it is called a fixed point of ϕ and it could be computed by the following iterative algorithm: with x (0) being the initial guess. x (k+1) = ϕ(x (k) ), k 0, (5) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 17

18 Geometric interpretation M (0) M (2) K (2) K (1) M (1) y K (0) x The solution x of x = ϕ(x) is clearly the intersection point of two graphs: y = ϕ(x) and y = x. This plot is for x = 1 exp(x)/4. The method converges to the fixed point x. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 18

19 Attempting to solve x = (x 0.5) 2 1 using fixed-point iterations leads to divergence M (5) M (1) K (2) K (6) 0.5 M (3) M (9) K (4) y 0 M (7) K (8) K (0) -0.5 K (7) M (8) M (6) What is the reason? M (2) K (3) K (9) M (0) -1 K (1) M (4) K (5) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 19

20 Convergence of fixed-point iterations Definition A σ-neighborhood of the point x is an interval (x σ, x + σ). Theorem Assume that in some σ-neighborhood of the root x the function ϕ(x) is differentiable and ϕ (x) q (6) with 0 q < 1. Then for any initial guess x (0) ( x σ, x + σ) the fixed point iterations converge and x (k) x q k x (0) x. (7) Formula (7) indicates linear convergence of the fixed-point iterations. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 20

21 Proof. For the fixed point we have x = ϕ( x) and the iterations are given by x (k+1) = ϕ(x (k) ), k 0. Taking the difference of these two expressions we get The mean value theorem gives x (k+1) x = ϕ(x (k) ) ϕ( x). ϕ(x (k) ) x = ϕ (ξ)(x (k) x) for some ξ between x and x (k), therefore x (k+1) x = ϕ(x (k) ) x = ϕ (ξ)(x (k) x) = ϕ (ξ)(x (k) x). Since ϕ (ξ) q we get x (k+1) x = ϕ(x (k) ) x = ϕ (ξ) x (k) x q x (k) x. Continuing the latter inequality to k = 0 we finally obtain x (k) x q k x (0) x and therefore iterations converge provided q < 1. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 21

22 Theorem Under the assumptions of the previous Theorem, for any x (0) ( x σ, x + σ) the following a-posteriori error estimate holds x (k) x q 1 q x(k) x (k 1), k 1. (8) Proof. Do it as an exercise! The inequality (8) can be used as a reliable criterion to terminate the iterations. For the target tolerance ε we can stop the process if x (k) x (k 1) < 1 q ε. (9) q University of Zurich, I Math 10/03/2016 Nonlinear equations Page 22

23 In our examples: 1 ϕ(x) = 1 exp(x)/4 and ϕ exp(x) (x) = 8 1 exp(x)/4 x [0, 1], ϕ (x) < 1 on [0, 1], x (0) [0, 1] = convergence dϕ(x)/dx x (0) x x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 23

24 2 ϕ(x) = (x 0.5) 2 1 and ϕ (x) = 2(x 0.5) x [ 0.5, 0], ϕ (x) 1 on [ 0.5, 0] = divergence dϕ(x)/dx x x (0) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 24

25 The Newton method We consider a nonlinear equation f (x) = 0. (10) The Newton method is one of the most efficient methods for the solution of various nonlinear equations: it is based on the linearization of the equation (10); it exploits the information about function f and derivative f ; it has quadratic convergence; however, choice of x (0) is still crucial. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 25

26 Geometric interpretation Newton s method can be derived from purely geometric considerations using the tangent lines M (0) 5 4 y M (2) M (1) x (2) x (1) x (0) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 26

27 Let x (0) be a guess value for the root x. The equation of the tangent line to the curve y = f (x) at the point (x (k), f (x (k) )) is y = f (x (k) ) + f (x (k) )(x x (k) ). (11) Assume that f (x (k) ) 0 and setting y = 0 in (11) we get the equation for the abscissa x (k+1) of the intersection point of the tangent line and the x-axis: 0 = f (x (k) ) + f (x (k) )(x (k+1) x (k) ). (12) From (12) we find x (k+1) = x (k) f (x(k) ), k 0. (13) f (x (k) ) Due to such geometric interpretation the Newton method is sometimes called also method of tangents. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 27

28 Linearization The Newton method can be also derived using the linearization of the original equation. Assume that we have computed the approximation x (k). In the neighbourhood of the point x (k) the function f (x) can be represented according to the Taylor formula as f (x) = f (x (k) ) + f (x (k) )(x x (k) ) + f (ξ) 2 (x x(k) ) 2, (14) where ξ is some point in the interval (x, x (k) ). Neglecting the second-order term in equation (14) we obtain a linear equation f (x (k) ) + f (x (k) )(x (k+1) x (k) ) = 0. (15) Choosing the solution of the equation (15) as the next approximation x (k+1), we get the same formula (13). University of Zurich, I Math 10/03/2016 Nonlinear equations Page 28

29 Convergence Theorem Let x be a simple root of the equation f (x) = 0 and assume that in some neighbourhood of x the function f is continuously differentiable up to its second derivative. Then there exists a σ-neighbourhood of x such that with any choice of the initial value x (0) from this neighbourhood the Newton method has quadratic convergence: or, equivalently, x (k+1) x σ 1 x (k) x 2, k 0, (16) x (k+1) x lim k (x (k) x) = f ( x) 2 2f ( x). (17) A consequence of (16) is an a-priori error estimate where q = σ 1 x (0) x. x (k) x σq 2k, k 0, (18) Remark Note that the quadratic convergence is local: it can be observed only in the neighbourhood of the exact root x, but might not hold globally. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 29

30 The following theorem gives an a-posteriori error estimate for the Newton method. Theorem If the assumptions of the previous theorem are satisfied and x (0) x < σ/2, then for all k 1 x (k) x x (k) x (k 1). (19) This theorem provides a reliable termination criterion which is used in practice for the Newton method. Tor the target tolerance ε the iterations can be stopped as soon as x (k) x (k 1) < ε. (20) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 30

31 Example Using the Newton method, construct an iterative process to compute p a for a > 0 and p = 1, 2, 3,.... By definition, x = p a is a nonnegative number satisfying the equation x p = a. Hence, the problem reduces to the estimation of the positive root of the nonlinear equation f (x) = 0 with f (x) = x p a. (21) The Newton method (13) applied to the equation f (x) = 0 with f (x) given by (21) takes the form x (k+1) = x (k) (x(k) ) p a p(x (k) ) = p 1 a p 1 p x(k) + p(x (k) ). p 1 For p = 2 this formula gives an iterative method to compute a: x (k+1) = 1 ( x (k) + a ). (22) 2 x (k) With x (0) = 2 the Newton method (22) gives in 4 iterations with tolerance ε = University of Zurich, I Math 10/03/2016 Nonlinear equations Page 31

32 Roots with multiplicity m > 1 Note that if the exact root x has multiplicity m > 1, i.e. if f ( x) = 0,..., f (m 1) ( x), then the Newton method converges with first order only for properly chosen x (0) and only if f (x) 0 for all x in the neighbourhood of x. In this case one can recover the second order of convergence by modifying the original method to Problem: m might be unknown a-priori. x (k+1) = x (k) m f (x(k) ), k 0. (23) f (x (k) ) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 32

33 Drawbacks of Newton s method 1 The Newton method in general does not converge for all possible choices of x (0), but only for those which are sufficiently close to x. Therefore an inaccurate choice of the initial guess might lead to a divergent sequence of approximations. In practice, a suitable guess x (0) can be obtained by implementing a few iterations of the bisection method, or, alternatively, through an investigation of the graph of f. 2 The Newton method might stop converging if at the k-th iteration f (x (k) ) 0. 3 Another difficulty consists of the computation of the derivative f (x) which is needed in formula (13). Very often it is simply impossible to find an analytic expression for f (x), while the approximate computation of the derivative with high accuracy might be, in general, unstable. The computation of the derivative, if possible, might as well be very expensive. In this cases a common practice is to use the modifications of the Newton method which avoid the direct computation of f (x). University of Zurich, I Math 10/03/2016 Nonlinear equations Page 33

34 Simplified Newton s method If the derivative f (x) is continuous, then in the neighbourhood of x f (x) f (x (0) ). Therefore one can replace the values f (x (k) ) in the formula (13) by f (x (0) ). This leads to a simplified Newton s method x (k+1) = x (k) f (x(k) ), k 0. (24) f (x (0) ) M (0) 5 4 y M (2) M (1) 0 x (2) x (1) x (0) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 34 x

35 Fault location method The basis for this modification of the Newton method is an approximate equality f (x (k) ) f (z(k) ) f (x (k) ) z (k) x (k), (25) which is valid for z (k) x (k) and is a consequence of the definition of the derivative f f (z) f (x) (x) = lim. z x z x Let c be some fixed point in the neighbourhood of the root x. Replacing f (x (k) ) in (13) by the right-hand side of (25) with z (k) = c we obtain the formula for the fault location method: x (k+1) = x (k) c x(k) f (c) f (x (k) ) f (x(k) ), k 0. (26) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 35

36 Geometric interpretation 10 9 M M (0) 5 y M (1) 1 M (2) 0 x (2) x (1) x (0) c x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 36

37 The secant method Setting in Newton s method (13) we obtain the secant method x (k+1) = x (k) f (x (k) ) f (x(k 1) ) f (x (k) ) x (k 1) x (k), x(k 1) x (k) f (x (k 1) ) f (x (k) ) f (x(k) ), k 0. (27) Note that the secant method needs two previous approximations x (k) and x (k 1) to compute the next approximation x (k+1) and therefore requires two guess values x (0) and x (1) to start the iterative process. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 37

38 Geometric interpretation M (0) y M (1) 2 1 M (2) M (3) 0 x (3) x (2) x (1) x (0) x University of Zurich, I Math 10/03/2016 Nonlinear equations Page 38

39 Convergence The following convergence result is valid for the secant method. Theorem Let x be a simple root of the equation f (x) = 0 and assume that f is continuously differentiable twice in some neighbourhood of x and f ( x) 0. Then there exists a σ-neighbourhood of x such that for any choice of the guess values x (0) and x (1) from this neighbourhood the secant method converges with order p = , that is, 2 x (k+1) x C x (k) x p 5 + 1, p =, k 1. (28) 2 University of Zurich, I Math 10/03/2016 Nonlinear equations Page 39

40 Quadratic convergence of Newton s method Theorem Let x be a simple root of the equation f (x) = 0 and assume that in some neighbourhood of x the function f is continuously differentiable up to its second derivative. Then there exists a σ-neighbourhood of x such that with any choice of the initial value x (0) from this neighbourhood the Newton method has quadratic convergence: or, equivalently, x (k+1) x σ 1 x (k) x 2, k 0, (29) x (k+1) x lim k (x (k) x) = f ( x) 2 2f ( x). (30) A consequence of (16) is an a-priori error estimate where q = σ 1 x (0) x. x (k) x σq 2k, k 0, (31) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 40

41 Proof. Since x is a simple root of f (x) = 0, we have f ( x) 0 (by definition of a simple root). Since f and f are continuous, δ 0 -neighborhood of x such that for some constant α R and β R we can write for x ( x δ 0, x + δ 0 ): 0 < α f (x) (f continuous and f ( x) 0) f (x) β (f continuous and hence bounded) Assume x (k) ( x σ, x + σ), where σ = min{δ 0, 2α/β}. Taylor expansion of f (x) at x (k) : f (x) = f (x (k) ) + f (x (k) )(x x (k) ) + f (ξ) 2 (x x(k) ) 2, (32) where ξ ( x σ, x + σ). Substituting x = x in (32) gives 0 = f (x (k) ) + f (x (k) )( x x (k) ) + f (ξ) 2 ( x x(k) ) 2. (33) Recall that Newton s method was defined from the linearized equation 0 = f (x (k) ) + f (x (k) )(x (k+1) x (k) ). (34) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 41

42 Proof (Cont.) Taking the difference between (33) and (34) we obtain f (x (k) )(x (k+1) x) = f (ξ) 2 ( x x(k) ) 2. (35) Estimating the modulus of left- and right-hand sides of (35) and taking into account the bounds on f and f we get and hence α x (k+1) x β 2 x x(k) 2 (36) x (k+1) x β 2α x x(k) 2 σ 1 x x (k) 2, (37) which means that the Newton method has a local quadratic convergence. The condition x (k+1) ( x σ, x + σ) is satisfied since x (k+1) x β 2α x x(k) 2 σ 1 σ 2 = σ, (38) meaning that the next approximation x (k+1) remains in the same neighborhood of x. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 42

43 Steffensen s method Iterations given by where x (k+1) = x (k) f (x(k) ), k 0, (39) g(x (k) ) g(x) = f ( x + f (x) ) f (x) f (x) f (x). This method can be interpreted as a fault location method with c = x (k) + f (x (k) ). It has quadratic convergence. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 43

44 Halley s method with cubic convergence We solve f (x) = 0 and assume x simple root. Halley s method: x (k+1) = x (k) u(x (k) 1 ), v(x(k) ) k 0, (40) where u(x (k) ) = f (x(k) ) f (x (k) ), v(xk ) = f (x(k) )f (x (k) ) [f (x (k) )] 2. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 44

45 Derivation. Assume for simplicity f (x) > 0 and consider the function g(x) = f (x) f (x). Since x is a simple root, f ( x) = 0 and f ( x) 0, therefore g( x) = 0. We can therefore apply Newton s method to the function g(x): g (x) = then the iterative process is f (x) f (x) f (x) f (x) 2 f (x) f (x) = 2[f (x)] 2 f (x)f (x) 2f (x), f (x) x (k+1) = x (k) g(x(k) ) g (x (k) ) = = 2f (x (k) )f (x (k) ) x(k) 2[f (x (k) )] 2 f (x (k) )f (x (k) ), which is equivalent to (40). University of Zurich, I Math 10/03/2016 Nonlinear equations Page 45

46 Householder s method We solve f (x) = 0 assuming that f is (p + 1) times continuously differentiable, and that x is a simple root. Householder s method of order p > 0 reads: and it has a convergence rate p + 1, i.e. x (k+1) = x (k) + p (1/f )(p 1) (x (k) ) (1/f ) (p) (x (k) ), (41) x (k+1) x C x (k) x p+1, k 0. (42) Special cases: p = 1 Newton s method; p = 2 Halley s method. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 46

47 Algebraic polynomials Consider a polynomial of degree n: p n(x) = a 0 + a 1 x + a 2 x a nx n. (43) Equivalent nested representation of p n(x): p n(x) = a 0 + x(a 1 + a 2 x + a 3 x a nx n 1 ) = a 0 + x ( a 1 + x(a 2 + a 3 x + + a nx n 2 ) ) =... ( = a 0 + x a 1 + x ( a x(a n 1 + a )) nx). (44) Form (44) gives an algorithm (Hörner s algorithm) to evaluate p n at some point x = z: b n := a n, b k := a k + b k+1 z, k = n 1, n 2,..., 0, p n(z) := b 0. Formula (43) requires n sums and 2n 1 products while formula (44) only n sums and n products. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 47

48 Finding the roots of p n Introduce the associated polynomial q n 1 (x; z) of degree (n 1): depending on z via b k coefficients. Division of p n(x) by (x z) gives q n 1 (x; z) = b 1 + b 2 x + + b nx n 1, p n(x) = b 0 + (x z)q n 1 (x; z). (45) If z is a root of p n, i.e. p n(z) = 0, then b 0 = p n(z) = 0 and In this case the algebraic equation p n(x) = (x z)q n 1 (x, z). q n 1 (x, z) = 0 provides the n 1 remaining roots of the polynomial p n. University of Zurich, I Math 10/03/2016 Nonlinear equations Page 48

49 Newton-Hörner method Algorithm for finding all roots of p n : for m = n, n 1,..., 1 find an approximation of the root r m for p m with Newton s method; compute q m 1 (x; r m) using Hörner s algorithm; set p m 1 = q m 1. For given m, if q m 1 is the polynomial associated with p m, then p m(x) = ( (x z)q m 1 (x, z) ) = q m 1 (x, z) + (x z)q m 1(x, z) (46) and with x = z Thanks to (47) the Newton method takes the form: r (k+1) j p m(z) = q m 1 (z, z). (47) = r (k) j pm(r (k) ) p m(r (k) ) = r (k) p m(r (k) ) j, k 0. (48) q m 1 (r (k), r (k) ) University of Zurich, I Math 10/03/2016 Nonlinear equations Page 49