Numerical Methods for Differential Equations

Numerical Methods for Differential Equations Chapter 1: Initial value problems in ODEs Gustaf Söderlind and Carmen Arévalo Numerical Analysis, Lund University Textbooks: A First Course in the Numerical Analysis of Differential Equations, by Arieh Iserles and Introduction to Mathematical Modelling with Differential Equations, by Lennart Edsberg c Gustaf Söderlind, Numerical Analysis, Mathematical Sciences, Lund University, 2008-09 Numerical Methods for Differential Equations p. 1/52

Chapter 1: contents Course contents Introduction to initial value problems The explicit Euler method Convergence Order of consistency The trapezoidal rule Theta methods Numerical tests The linear test equation and numerical stability Stiff equations Numerical Methods for Differential Equations p. 2/52

What is Numerical Analysis? Categories of mathematical problems Category Algebra Analysis linear computable not computable nonlinear not computable not computable Algebra: Only finite constructions Analysis: Limits, derivatives, integrals etc. (transfinite) Computable: the exact solution can be obtained in a finite number of operations Numerical Methods for Differential Equations p. 3/52

What is Numerical Analysis? Category Algebra Analysis linear computable not computable nonlinear not computable not computable With numerical methods, problems from all four categories can be solved: Numerical analysis aims to construct and analyze quantitative methods for the automatic computation of approximate solutions to mathematical problems. Goal: Construction of mathematical software Numerical Methods for Differential Equations p. 4/52

0. What will we study in this course? To solve a differential equation analytically we look for a differentiable function that satisfies the equation Large, complex and nonlinear systems cannot be solved analytically Numerical Methods for Differential Equations p. 5/52

0. What will we study in this course? To solve a differential equation analytically we look for a differentiable function that satisfies the equation Large, complex and nonlinear systems cannot be solved analytically Instead, we compute numerical solutions with standard methods and software To solve a differential equation numerically we generate a sequence {y k } N k=0 analytical solution: y(t k ) y k of pointwise approximations to the Numerical Methods for Differential Equations p. 5/52

Some differential equations we will solve Initial value problems (IVP) first-order equations; higher-order equations; systems of differential equations Numerical Methods for Differential Equations p. 6/52

Some differential equations we will solve Initial value problems (IVP) first-order equations; higher-order equations; systems of differential equations Boundary value problems (BVP) two-point boundary value problems; Sturm-Liouville eigenvalue problems Numerical Methods for Differential Equations p. 6/52

Some differential equations we will solve Initial value problems (IVP) first-order equations; higher-order equations; systems of differential equations Boundary value problems (BVP) two-point boundary value problems; Sturm-Liouville eigenvalue problems Partial differential equations (PDE) the diffusion equation; the advection equation; the wave equation Numerical Methods for Differential Equations p. 6/52

Some differential equations we will solve Initial value problems (IVP) first-order equations; higher-order equations; systems of differential equations Boundary value problems (BVP) two-point boundary value problems; Sturm-Liouville eigenvalue problems Partial differential equations (PDE) the diffusion equation; the advection equation; the wave equation Applications in all three areas Numerical Methods for Differential Equations p. 6/52

Initial value problems: examples A first-order equation: a simple equation without a known analytical solution dy dt = y e t2, y(0) = y 0 Numerical Methods for Differential Equations p. 7/52

Initial value problems: examples A first-order equation: a simple equation without a known analytical solution A second-order equation: motion of a pendulum θ (t) + g L sin θ(t) = 0, θ(0) = θ 0, θ (0) = θ 0 where θ(t) is the angle, g is the gravitational constant and L is the pendulum length Numerical Methods for Differential Equations p. 7/52

Initial value problems: examples A first-order equation: a simple equation without a known analytical solution A second-order equation: pendulum equation A system of equations: the predator prey model y 1(t) = k 1 y 1 (t) k 2 y 1 (t)y 2 (t) y 2(t) = k 3 y 1 (t)y 2 (t) k 4 y 2 (t) where y 1 (t) is the population of the prey species and y 2 (t) is the population of the predator species at time t Numerical Methods for Differential Equations p. 7/52

Boundary value problems: examples Second-order two-point BVP: the electrostatic potential u(r) between two concentric metal spheres satisfies d 2 u dr 2 + 2 r du dr = 0, u(r 1) = V 1, u(r 2 ) = 0 at the distance r from the center; R 1 and R 2 are the radii of the two spheres Numerical Methods for Differential Equations p. 8/52

Boundary value problems: examples Second-order two-point BVP: the electrostatic potential u(r) between two concentric metal spheres satisfies d 2 u dr 2 + 2 r du dr = 0, u(r 1) = V 1, u(r 2 ) = 0 at the distance r from the center; R 1 and R 2 are the radii of the two spheres A Sturm-Liouville eigenproblem: Euler buckling of column y + λy = 0, y (0) = 0, y(1) = 0 Find eigenvalues λ and eigenfunctions y Numerical Methods for Differential Equations p. 8/52

Partial differential equations: examples The heat equation u t (x,t) = u xx (x,t), x [0,a), t (0,b) u(x, 0) = f(x), x [0,a] u(0,t) = c 1, u(a,t) = c 2, t [0,b] is a parabolic PDE modelling e.g. the temperature in an insulated rod with constant temperatures c 1 and c 2 at its ends, and initial temperature distribution f(x) Numerical Methods for Differential Equations p. 9/52

Partial differential equations: examples The heat equation The wave equation u tt (x,t) = u xx (x,t), x (0,a), t (0,b) u(0,t) = 0, u(a,t) = 0 for t [0,b] u(x, 0) = f(x) u t (x, 0) = g(x) for x [0,a] for x (0,a) is a hyperbolic PDE e.g. modelling the displacement u of a vibrating elastic string fixed at x = 0 and x = a Numerical Methods for Differential Equations p. 9/52

Some applications in ODEs Initial value problems mechanics M q = F(q) electrical circuits C v = I(v) chemical reactions ċ = f(c) Boundary value problems materials u = M(u)/EI microphysics 2m ψ = E eigenmodes u + λu = 0 Numerical Methods for Differential Equations p. 10/52

1. Initial value problems Standard formulation y = f(t,y) ; y(0) = y 0 Scalar equation: y R; f scalar; Systems of ODEs: y R m ; f vector-valued. Theorem (Cauchy & Peano) A solution to y = f(t,y) exists if f is continuous. Numerical Methods for Differential Equations p. 11/52

Existence Note: Continuity is not sufficient to guarantee uniqueness! Example: ẏ = 2 y; y(0) = 0 with solution y(t) = 0, t τ and y(t) = (t τ) 2, t > τ This may happen in real physical models! Consider the energy of a falling particle of mass m: 1 2 mẏ2 + mgy = E := 0 This yields ẏ = 2g y with y(0) = 0 Numerical Methods for Differential Equations p. 12/52

Existence and uniqueness Theorem If f(t,y) is continuous on [0,T] and satisfies the Lipschitz condition f(t,u) f(t,v) L[f] u v for all u,v and some Lipschitz constant L[f] <, there exists a unique solution to the initial value problem y = f(t,y) on [0,T] for every initial value y(0) = y 0 Numerical Methods for Differential Equations p. 13/52

Existence and uniqueness... Note Most problems do not satisfy a Lipschitz condition on all of R m Example The problem has solution ẏ = y 2 ; y(0) = y 0 > 0 y(t) = y 0 1 y 0 t The solution blows up at t = 1/y 0 ( Finite escape time ) Numerical Methods for Differential Equations p. 14/52

Existence and uniqueness... Other problems always satisfy Lipschitz conditions on R m Example Lipschitz constant: ẏ = Ay; y(0) = y 0 L[f] = max u =v Au Av u v = max y 0 Ay y = A The matrix norm A is a Lipschitz constant for f(y) = Ay Numerical Methods for Differential Equations p. 15/52

Standard form: x = F(t,x) Example y = f(t,y,y ); y(0) = y 0 ; y (0) = y 0 Standard substitution Introduce new variables x 1 = y x 2 = y Then we get a system of first order equations x 1 = x 2 x 2 = f(t,x 1,x 2 ) with x 1 (0) = y 0 and x 2 (0) = y 0 Numerical Methods for Differential Equations p. 16/52

2. The Explicit Euler method (1768) y = f(t,y); y(t 0 ) = y 0 Replace derivative by finite difference approximation y (t n ) y(t n + h) y(t n ) h Let {u n } denote the numerical approximation to {y(t n )} and h = t n+1 t n denote the time-step. Numerical Methods for Differential Equations p. 17/52

2. The Explicit Euler method (1768) y = f(t,y); y(t 0 ) = y 0 Replace derivative by finite difference approximation y (t n ) y(t n + h) y(t n ) h Let {u n } denote the numerical approximation to {y(t n )} and h = t n+1 t n denote the time-step. Compute u n from u n+1 u n h = f(t n,u n ), u 0 = y 0 Numerical Methods for Differential Equations p. 17/52

2. The Explicit Euler method (1768) y = f(t,y); y(t 0 ) = y 0 Replace derivative by finite difference approximation y (t n ) y(t n + h) y(t n ) h Let {u n } denote the numerical approximation to {y(t n )} and h = t n+1 t n denote the time-step. Compute u n from u n+1 u n h = f(t n,u n ), u 0 = y 0 The Explicit Euler method Given u 0, t 0 and h, compute u n+1 = u n + hf(t n,u n ) t n+1 = t n + h Numerical Methods for Differential Equations p. 17/52

Explicit Euler: Taylor series expansion Use Taylor series expansion y (t) = f(t,y), y(t 0 ) = y 0 y(t + h) = y(t) + hy (t) + h2 2! y (ζ) y(t + h) = y(t) + hf(t,y(t)) + O(h 2 ) y(t) + hf(t,y(t)) Numerical Methods for Differential Equations p. 18/52

Explicit Euler: Taylor series expansion Use Taylor series expansion y (t) = f(t,y), y(t 0 ) = y 0 y(t + h) = y(t) + hy (t) + h2 2! y (ζ) y(t + h) = y(t) + hf(t,y(t)) + O(h 2 ) y(t) + hf(t,y(t)) Construct the numerical method (drop higher order terms) u n+1 = u n + hf(t n,u n ); u 0 = y 0 t n+1 = t n + h Numerical Methods for Differential Equations p. 18/52

Explicit Euler: Taylor series expansion Use Taylor series expansion y (t) = f(t,y), y(t 0 ) = y 0 y(t + h) = y(t) + hy (t) + h2 2! y (ζ) y(t + h) = y(t) + hf(t,y(t)) + O(h 2 ) y(t) + hf(t,y(t)) Construct the numerical method (drop higher order terms) u n+1 = u n + hf(t n,u n ); u 0 = y 0 t n+1 = t n + h Explicit Euler! Numerical Methods for Differential Equations p. 18/52

Explicit Euler: graphic interpretation Take a step of size h in the direction of the tangent 1.2 1.15 1.1 1.05 1 y 0.95 0.9 0.85 0.8 0.75 0.7 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t Each step introduces an error and ends up on a different solution trajectory (dashed curves) Numerical Methods for Differential Equations p. 19/52

Explicit Euler: an example Compute numerical solution to y = y cos t; y(0) = 1; t [0,2π] Solution Choose a stepsize h = 2π/N; N = 24 h = π/12 Recursion u n+1 = u n + h ( u n cos t n ); u 0 = 1 t n+1 = t n + h; t 0 = 0 1.8 1.8 1.6 1.6 1.4 1.4 1.2 1.2 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0 1 2 3 4 5 6 7 0.2 0 1 2 3 4 5 6 7 The numerical solution is a sequence of points (t n,u n ) Numerical Methods for Differential Equations p. 20/52

3. Convergence Analytical and numerical solutions for h = π/8 and π/128 3 N=64 3 N=64*16 2.5 2.5 2 2 1.5 1.5 1 1 0.5 0.5 0 0 5 10 15 20 25 0 0 5 10 15 20 25 As h 0 the numerical solution approaches the exact solution Numerical Methods for Differential Equations p. 21/52

3. Convergence Analytical and numerical solutions for h = π/8 and π/128 3 N=64 3 N=64*16 2.5 2.5 2 2 1.5 1.5 1 1 0.5 0.5 0 0 5 10 15 20 25 0 0 5 10 15 20 25 As h 0 the numerical solution approaches the exact solution A method is convergent if, for every ODE with a Lipschitz function f and every fixed T, with T = N h, it holds that lim y N,h y(t) = 0 N Numerical Methods for Differential Equations p. 21/52

Local and global errors 1.15 1.1 y n y n+1 y 1.05 ŷ n+1 1 y(t n ) y(t n+1 ) y(t) 0.95 1.3 1.35 1.4 1.45 1.5 1.55 1.6 1.65 1.7 1.75 1.8 t Global error e n = y n y(t n ) and e n+1 = y n+1 y(t n+1 ) Local error l n+1 = ŷ n+1 y(t n+1 ) Numerical Methods for Differential Equations p. 22/52

The local error of the explicit Euler method Insert exact data ŷ n+1 = y(t n ) + hf(t n,y(t n )) Local error definition l n+1 = ŷ n+1 y(t n+1 ) implies residual y(t n+1 ) = y(t n ) + hf(t n,y(t n )) l n+1 Taylor series y(t n+1 ) = y(t n ) + hy (t n ) + h2 2 y (t n ) +... Local error for explicit Euler (along exact solution, solid curve) l n+1 h2 2 y (t n ) Numerical Methods for Differential Equations p. 23/52

Error propagation Explicit Euler (numerical solution) y n+1 = y n + hf(t n,y n ) Subtract Taylor series expansion of exact solution y(t n+1 ) = y(t n ) + hf(t n,y(t n )) + h2 2 y (t n ) +... Global error recursion e n+1 = e n + hf(t n,y(t n ) + e n ) hf(t n,y(t n )) + l n+1 Numerical Methods for Differential Equations p. 24/52

Error propagation... e n+1 = e n + hf(t n,y(t n ) + e n ) hf(t n,y(t n )) + l n+1 Take norms and use Lipschitz condition: e n+1 e n + hl[f] e n + l n+1 Lemma If {a n }, a 0 = 0, is a sequence of non-negative numbers satisfying a n+1 (1 + hµ)a n + ch 2 for µ > 0, then a n c µ h[(1 + hµ)n 1], n = 0, 1,... Numerical Methods for Differential Equations p. 25/52

Convergence of Euler s method: Theorem Theorem The explicit Euler method is convergent Numerical Methods for Differential Equations p. 26/52

Convergence of Euler s method: Theorem Theorem The explicit Euler method is convergent Proof Suppose f sufficiently differentiable. Given h and a fixed T = Nh, let e n,h = y n,h y(t n ) Numerical Methods for Differential Equations p. 26/52

Convergence of Euler s method: Theorem Theorem The explicit Euler method is convergent Proof Suppose f sufficiently differentiable. Given h and a fixed T = Nh, let e n,h = y n,h y(t n ) Apply the lemma to global error recursion, to get e n,h c L[f] h[(1 + hl[f])n 1], n = 0, 1,... with c = max n l n /h 2 max t y /2 Numerical Methods for Differential Equations p. 26/52

Convergence of Euler s method... As (1 + hl[f]) n < e nhl[f] e TL[f], we have for n N, e n,h c L[f] h(etl[f] 1) So e n,h C(T) h implies convergence because lim e n,h = 0 h 0 Note 1) The global error can be made arbitrarily small! 2) The error is way too large for practical purposes! 3) Better bounds can be obtained Numerical Methods for Differential Equations p. 27/52

Theoretical error bound Example y = 100y, y(0) = 1. Then L[f] = 100 and the exact solution is y(t) = e 100t with y (t) = 100 2 e 100t, so c = 100 2 /2, with bound e n,h 1002 2 100 h (e100t 1) Numerical Methods for Differential Equations p. 28/52

Theoretical error bound Example y = 100y, y(0) = 1. Then L[f] = 100 and the exact solution is y(t) = e 100t with y (t) = 100 2 e 100t, so c = 100 2 /2, with bound e n,h 1002 2 100 h (e100t 1) Error estimate at T = 1 is e n,h 50he 100 1.4 10 45 h! Numerical Methods for Differential Equations p. 28/52

Theoretical error bound Example y = 100y, y(0) = 1. Then L[f] = 100 and the exact solution is y(t) = e 100t with y (t) = 100 2 e 100t, so c = 100 2 /2, with bound e n,h 1002 2 100 h (e100t 1) Error estimate at T = 1 is e n,h 50he 100 1.4 10 45 h! But y n = (1 100h) n, so for h < 1/50, at T = 1, Actual error is e n,h = (1 100/N) N e 100 3.7 10 44 h! Numerical Methods for Differential Equations p. 28/52

Computational test ẏ = λ(y sint) + cos t λ = 0.2, with initial condition y(π/4) = 1/ 2 1.2 1.2 1.1 1.1 1 1 0.9 0.9 y y 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t h = π/10 h = π/20 Note Local error O(h 2 ), global error O(h)! Numerical Methods for Differential Equations p. 29/52

4. Order of consistency Given a method in the form y n+1 = Φ n (f,h,y 0,y 1,...,y n ) Insert exact data y(t 0 ),y(t 1 ),...,y(t n ) Definition The order of consistency is p if y(t n+1 ) Φ n (f,h,y(t 0 ),y(t 1 ),...,y(t n )) = O(h p+1 ), n as h 0, for every analytic f Theorem The local error is then O(h p+1 ) Numerical Methods for Differential Equations p. 30/52

4. Order of consistency Given a method in the form y n+1 = Φ n (f,h,y 0,y 1,...,y n ) Insert exact data y(t 0 ),y(t 1 ),...,y(t n ) Definition The order of consistency is p if y(t n+1 ) Φ n (f,h,y(t 0 ),y(t 1 ),...,y(t n )) = O(h p+1 ), n as h 0, for every analytic f Theorem The local error is then O(h p+1 ) Alternative The order of consistency is p if the formula is exact for all polynomials y = P(t) of degree p or less Numerical Methods for Differential Equations p. 30/52

Order of consistency of Euler s method Example Explicit Euler Φ n (f,h,y 0,...,y n ) = y n + hf(t n,y n ) Numerical Methods for Differential Equations p. 31/52

Order of consistency of Euler s method Example Explicit Euler Φ n (f,h,y 0,...,y n ) = y n + hf(t n,y n ) Expanding in Taylor series, y(t n+1 ) [y(t n ) + hf(t n,y(t n ))] = O(h 2 ) so the method s consistency order is one Numerical Methods for Differential Equations p. 31/52

Order of consistency of Euler s method Example Explicit Euler Φ n (f,h,y 0,...,y n ) = y n + hf(t n,y n ) Expanding in Taylor series, y(t n+1 ) [y(t n ) + hf(t n,y(t n ))] = O(h 2 ) so the method s consistency order is one Alternatively, suppose y(t) = 1 with f = y = 0. Then 1 = y(t n+1 ) = y(t n ) + hf(t n,y(t n )) = 1 + h 0 = 1 exact! Numerical Methods for Differential Equations p. 31/52

Order of consistency of Euler s method Example Explicit Euler Φ n (f,h,y 0,...,y n ) = y n + hf(t n,y n ) Expanding in Taylor series, y(t n+1 ) [y(t n ) + hf(t n,y(t n ))] = O(h 2 ) so the method s consistency order is one Alternatively, suppose y(t) = 1 with f = y = 0. Then 1 = y(t n+1 ) = y(t n ) + hf(t n,y(t n )) = 1 + h 0 = 1 exact! Next take 1st degree polynomial y(t) = t with f = y = 1, then h = y(t n+1 ) = y(t n ) + hf(t n,y(t n )) = 0 + h 1 = h exact! Numerical Methods for Differential Equations p. 31/52

Order of consistency of Euler s method Example Explicit Euler Φ n (f,h,y 0,...,y n ) = y n + hf(t n,y n ) Expanding in Taylor series, y(t n+1 ) [y(t n ) + hf(t n,y(t n ))] = O(h 2 ) so the method s consistency order is one Alternatively, suppose y(t) = 1 with f = y = 0. Then 1 = y(t n+1 ) = y(t n ) + hf(t n,y(t n )) = 1 + h 0 = 1 exact! Next take 1st degree polynomial y(t) = t with f = y = 1, then h = y(t n+1 ) = y(t n ) + hf(t n,y(t n )) = 0 + h 1 = h exact! For a 2nd degree polynomial, y(t) = t 2 with f = y = 2t. Then h 2 = y(t n+1 ) = y(t n )+hf(t n,y(t n )) = 0+h 0 = 0 h 2 p = 1! Numerical Methods for Differential Equations p. 31/52

5. The trapezoidal rule Explicit Euler linearizes y at t n with a slope of y (t n ) Numerical Methods for Differential Equations p. 32/52

5. The trapezoidal rule Explicit Euler linearizes y at t n with a slope of y (t n ) Instead, linearize with a slope equal to the average of y (t n ) and y (t n+1 ) y(t) y(t n ) + (t t n ) 1 2 [f(t n,y(t n )) + f(t n+1,y(t n+1 ))] Numerical Methods for Differential Equations p. 32/52

5. The trapezoidal rule Explicit Euler linearizes y at t n with a slope of y (t n ) Instead, linearize with a slope equal to the average of y (t n ) and y (t n+1 ) y(t) y(t n ) + (t t n ) 1 2 [f(t n,y(t n )) + f(t n+1,y(t n+1 ))] The numerical method is the trapezoidal rule t n+1 = t n + h y n+1 = y n + h 2 [f(t n,y n ) + f(t n+1,y n+1 )] Numerical Methods for Differential Equations p. 32/52

5. The trapezoidal rule Explicit Euler linearizes y at t n with a slope of y (t n ) Instead, linearize with a slope equal to the average of y (t n ) and y (t n+1 ) y(t) y(t n ) + (t t n ) 1 2 [f(t n,y(t n )) + f(t n+1,y(t n+1 ))] The numerical method is the trapezoidal rule t n+1 = t n + h y n+1 = y n + h 2 [f(t n,y n ) + f(t n+1,y n+1 )] The method is implicit. Nonlinear equation solving required Numerical Methods for Differential Equations p. 32/52

Trapezoidal rule: Order and convergence Order of consistency (sketch): y(t n+1 ) {y(t n ) + h 2 [f(t n,y(t n )) + f(t n+1,y(t n+1 ))]} = y(t n ) + hy (t n ) + h2 2 y (t n ) + O(h 3 ) {y(t n ) + h 2 (y (t n ) + [y (t n ) + hy (t n ) + O(h 2 )])} = O(h 3 ) Numerical Methods for Differential Equations p. 33/52

Trapezoidal rule: Order and convergence Order of consistency (sketch): y(t n+1 ) {y(t n ) + h 2 [f(t n,y(t n )) + f(t n+1,y(t n+1 ))]} = y(t n ) + hy (t n ) + h2 2 y (t n ) + O(h 3 ) {y(t n ) + h 2 (y (t n ) + [y (t n ) + hy (t n ) + O(h 2 )])} = O(h 3 ) The method is of order two Numerical Methods for Differential Equations p. 33/52

Trapezoidal rule: Order and convergence Order of consistency (sketch): y(t n+1 ) {y(t n ) + h 2 [f(t n,y(t n )) + f(t n+1,y(t n+1 ))]} = y(t n ) + hy (t n ) + h2 2 y (t n ) + O(h 3 ) {y(t n ) + h 2 (y (t n ) + [y (t n ) + hy (t n ) + O(h 2 )])} = O(h 3 ) The method is of order two Theorem The trapezoidal rule is convergent (No proof given here) Numerical Methods for Differential Equations p. 33/52

Trapezoidal rule: the dramatic effect of 2nd order Compute numerical solution to y = y cos t; y(0) = 1; t [0,8π]. Choose h = π/12. Note that f is linear in y. The recursion is u n+1 = 1 h 2 cos t n 1 + h 2 cos t n+1 u n ; u 0 = 1 3 N=96 Trapezoidal rule 3 N=96 Explicit Euler 2.5 2.5 2 2 1.5 1.5 1 1 0.5 0.5 0 0 5 10 15 20 25 0 0 5 10 15 20 25 Solutions with Trapezoidal rule and Explicit Euler Numerical Methods for Differential Equations p. 34/52

The implicit midpoint rule We can also approximate the derivative y (t) by taking the average of t n and t n+1 as well as y n and y n+1 : y (t) f( t n + t n+1 2, y n + y n+1 2 ), t [t n,t n+1 ] Numerical Methods for Differential Equations p. 35/52

The implicit midpoint rule We can also approximate the derivative y (t) by taking the average of t n and t n+1 as well as y n and y n+1 : y (t) f( t n + t n+1 2, y n + y n+1 2 ), t [t n,t n+1 ] The resulting method is the 2nd order implicit midpoint method t n+1 = t n + h y n+1 = y n + hf( t n + t n+1 2, y n + y n+1 2 ) Numerical Methods for Differential Equations p. 35/52

6. Theta methods We construct methods that linearize y at t n with a slope equal to a convex combination of y (t n ) and y (t n+1 ): y n+1 = y n + h [θf(t n,y n ) + (1 θ)f(t n+1,y n+1 )], θ [0,1] Explicit Euler: θ = 1 Trapezoidal rule (implicit): θ = 1/2 Implicit Euler: θ = 0 y n+1 = y n + hf(t n+1,y n+1 ) Numerical Methods for Differential Equations p. 36/52

Theta methods: Order and convergence Use Taylor expansion to get y(t n+1 ) y(t n ) h [θf(t n,y(t n )) + (1 θ)f(t n+1,y(t n+1 ))] = (θ 1 2 )h2 y (t n ) + 1 2 (θ 2 3 )h3 y (t n ) + O(h 4 ) Numerical Methods for Differential Equations p. 37/52

Theta methods: Order and convergence Use Taylor expansion to get y(t n+1 ) y(t n ) h [θf(t n,y(t n )) + (1 θ)f(t n+1,y(t n+1 ))] = (θ 1 2 )h2 y (t n ) + 1 2 (θ 2 3 )h3 y (t n ) + O(h 4 ) If θ = 1/2 the method is of order 2; otherwise it is of order 1 Numerical Methods for Differential Equations p. 37/52

Theta methods: Order and convergence Use Taylor expansion to get y(t n+1 ) y(t n ) h [θf(t n,y(t n )) + (1 θ)f(t n+1,y(t n+1 ))] = (θ 1 2 )h2 y (t n ) + 1 2 (θ 2 3 )h3 y (t n ) + O(h 4 ) If θ = 1/2 the method is of order 2; otherwise it is of order 1 Theorem (without proof) The θ methods are convergent Numerical Methods for Differential Equations p. 37/52

Implicit methods Implicit Euler y n+1 = y n + hf(t n+1,y n+1 ) We need to solve a nonlinear equation to compute y n+1 The extra cost is motivated if we can take larger steps There are some problems where implicit methods can take enormous time steps without losing accuracy! We will return to how to solve nonlinear equations Numerical Methods for Differential Equations p. 38/52

7. Tests: Explicit Euler ẏ = λ(y sint) + cost λ = 0.2, with initial condition y(π/4) = 1/ 2 1.2 1.2 1.1 1.1 1 1 0.9 0.9 y y 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t h = π/10 h = π/20 Local error O(h 2 ), global error O(h)! Numerical Methods for Differential Equations p. 39/52

Tests: Implicit Euler ẏ = λ(y sint) + cost λ = 0.2, with initial condition y(π/4) = 1/ 2 1.2 1.2 1.1 1.1 1 1 0.9 0.9 y y 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t h = π/10 h = π/20 Local error O(h 2 ), global error O(h)! Numerical Methods for Differential Equations p. 40/52

Tests: Implicit Euler ẏ = λ(y sint) + cost λ = 10, with initial condition y(π/4) = 1/ 2 1.2 1.2 1.1 1.1 1 1 0.9 0.9 y y 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t h = π/10 h = π/20 Local error = O(h 2 ), global error = O(h), but difficult to see! Numerical Methods for Differential Equations p. 41/52

Tests: Explicit Euler ẏ = λ(y sint) + cost λ = 10, with initial condition y(π/4) = 1/ 2 1.2 1.1 1 0.9 y 0.8 0.7 0.6 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t h = π/20 Numerical Methods for Differential Equations p. 42/52

Tests: Explicit Euler ẏ = λ(y sint) + cost λ = 10, with initial condition y(π/4) = 1/ 2 1.2 1.2 1.1 1.1 1 1 0.9 0.9 y y 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t 0.5 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t h = π/10 h = π/20 NUMERICAL INSTABILITY! Stability for h small Numerical Methods for Differential Equations p. 42/52

8. The linear test equation Defintion The linear test equation is y = λy; y(0) = 1, t 0, where λ C Numerical Methods for Differential Equations p. 43/52

8. The linear test equation Defintion The linear test equation is y = λy; y(0) = 1, t 0, where λ C As y(t) = e λt, we have y(t) K Re(λ) 0 Mathematical stability Bounded solutions if Re(λ) 0 Numerical Methods for Differential Equations p. 43/52

8. The linear test equation Defintion The linear test equation is y = λy; y(0) = 1, t 0, where λ C As y(t) = e λt, we have y(t) K Re(λ) 0 Mathematical stability Bounded solutions if Re(λ) 0 When does a numerical method have the same property? Does Re(λ) 0 imply numerical stability? Numerical Methods for Differential Equations p. 43/52

Numerical stability: The stability region Definition The stability region D of a method is the set of all hλ C such that y n K when the method is applied to the test equation Numerical Methods for Differential Equations p. 44/52

Numerical stability: The stability region Definition The stability region D of a method is the set of all hλ C such that y n K when the method is applied to the test equation Example For Euler s method, y n+1 = (1 + hλ)y n, so y n remains bounded if and only if 1 + hλ 1 Numerical Methods for Differential Equations p. 44/52

Numerical stability: The stability region Definition The stability region D of a method is the set of all hλ C such that y n K when the method is applied to the test equation Example For Euler s method, y n+1 = (1 + hλ)y n, so y n remains bounded if and only if 1 + hλ 1 4 3 D Euler = {z C : 1 + z 1} 2 1 0 1 2 3 4 4 3 2 1 0 1 2 3 4 Numerical Methods for Differential Equations p. 44/52

A-stability Definition A method is called A-stable if its stability region contains C, i.e. C {z C : Re(z) 0} D. Numerical Methods for Differential Equations p. 45/52

A-stability Definition A method is called A-stable if its stability region contains C, i.e. C {z C : Re(z) 0} D. For the trapezoidal rule { D TR = z C : 1 + 1 2 z } 1 1 2 z 1 C so if Re(z) 0 then y n remains bounded for any h > 0 Numerical Methods for Differential Equations p. 45/52

A-stability Definition A method is called A-stable if its stability region contains C, i.e. C {z C : Re(z) 0} D. For the trapezoidal rule { D TR = z C : 1 + 1 2 z } 1 1 2 z 1 C so if Re(z) 0 then y n remains bounded for any h > 0 The explicit Euler method is not A-stable but the implicit Euler method and the trapezoidal rule are A-stable Usually: If the original problem is stable, then an A-stable method will replicate that behavior numerically Numerical Methods for Differential Equations p. 45/52

Relevance of the linear test equation When we have a system of equations y = Ay and A has a full set of eigenvectors with eigenvalues Λ = diag(λ 1,λ 2,...,λ N ), then A can be diagonalized, T 1 AT = Λ Putting y = Tx transforms the system to x = Λx (scalar systems)... equal to x k = λ kx k for k = 1,...,d The exact solution y(t) = e ta y 0 is stable if and only if Re(λ k ) 0 for all k = 1,...,d Numerical Methods for Differential Equations p. 46/52

Relevance of the linear test equation... Applying a method to y = Ay gives (say) y n+1 = (I + ha)y n Numerical Methods for Differential Equations p. 47/52

Relevance of the linear test equation... Applying a method to y = Ay gives (say) y n+1 = (I + ha)y n Noting that T 1 (I + ha)t = I + hλ, putting y n = Tx n produces x n+1 = (I + hλ)x n Numerical Methods for Differential Equations p. 47/52

Relevance of the linear test equation... Applying a method to y = Ay gives (say) y n+1 = (I + ha)y n Noting that T 1 (I + ha)t = I + hλ, putting y n = Tx n produces x n+1 = (I + hλ)x n The same as applying the method to the diagonalized system x = Λx Numerical Methods for Differential Equations p. 47/52

Relevance of the linear test equation... Applying a method to y = Ay gives (say) y n+1 = (I + ha)y n Noting that T 1 (I + ha)t = I + hλ, putting y n = Tx n produces x n+1 = (I + hλ)x n The same as applying the method to the diagonalized system x = Λx Diagonalization and discretization commute! Stability condition from linear test equation applies to all diagonalizable systems! Numerical Methods for Differential Equations p. 47/52

9. Stiff ODEs Example Solve ẏ = λ(y sin t) + cost with λ = 50 Solution Particular: Homogeneous: General: y P (t) = sin t y H (t) = e λt y(t) = e λ(t t 0) (y(t 0 ) sint 0 ) + sin t Study the flow of this equation and numerical solutions Numerical Methods for Differential Equations p. 48/52

Flow (solution trajectories) 1.2 1.1 1 0.9 y 0.8 0.7 0.6 0.5 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 t Numerical Methods for Differential Equations p. 49/52

Stiffness... With Explicit Euler the solution approaches sin t as t, as the exact solution does, only if 0 < h < 1 25 Numerical Methods for Differential Equations p. 50/52

Stiffness... With Explicit Euler the solution approaches sin t as t, as the exact solution does, only if 0 < h < 1 25 This means h must be kept small, not to keep errors small, but to have a stable numerical solution Numerical Methods for Differential Equations p. 50/52

Stiffness... With Explicit Euler the solution approaches sin t as t, as the exact solution does, only if 0 < h < 1 25 This means h must be kept small, not to keep errors small, but to have a stable numerical solution The trapezoidal rule solution tends to sin t as t, as the exact solution does, for every h > 0 This means h must be kept small only to keep errors small Numerical Methods for Differential Equations p. 50/52

Stiffness... With Explicit Euler the solution approaches sin t as t, as the exact solution does, only if 0 < h < 1 25 This means h must be kept small, not to keep errors small, but to have a stable numerical solution The trapezoidal rule solution tends to sin t as t, as the exact solution does, for every h > 0 This means h must be kept small only to keep errors small For A-stable methods, there is no stability restriction on h The step size is only restricted by accuracy Numerical Methods for Differential Equations p. 50/52

Stiffness... Stiff differential equations are characterized by homogeneous solutions being strongly damped Example ẏ = λ(y sint) + cos t with λ 1 Stability regions of explicit methods are bounded, and hλ D puts a strong stability restriction on h Example Explicit Euler applied to the problem above requires h 2/λ 1 Numerical Methods for Differential Equations p. 51/52

Stiffness... Implicit methods with unbounded stability regions put no restrictions on h, and the stepsize is only restricted by accuracy requirement! Example The implicit Euler applied to the problem above requires only that l n h2 y 2 h2 2 sin t n is sufficiently small, independently of λ Numerical Methods for Differential Equations p. 52/52