Numerical Linear Algebra Chap. 4: Perturbation and Regularisation

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Numerical Linear Algebra Chap. 4: Perturbation and Regularisation"

Transcription

1 Numerical Linear Algebra Chap. 4: Perturbation and Regularisation Heinrich Voss Hamburg University of Technology Institute of Numerical Simulation TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

2 Linear systems Sensitivity of linear systems Consider the linear system of equation Ax = b (1) where A R (n,n) is a nonsingular matrix, and a perturbed system (A + A)(x + x) = b + b. (2) TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

3 Linear systems Sensitivity of linear systems Consider the linear system of equation Ax = b (1) where A R (n,n) is a nonsingular matrix, and a perturbed system (A + A)(x + x) = b + b. (2) Our aim is to examine how perturbations of A and of b affect the solution of the system. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

4 Remarks Linear systems Small perturbations always have to be kept in mind when solving practical problems since TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

5 Remarks Linear systems Small perturbations always have to be kept in mind when solving practical problems since the data A and/or b may be obtained from measurements, and therefore they are erroneous TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

6 Remarks Linear systems Small perturbations always have to be kept in mind when solving practical problems since the data A and/or b may be obtained from measurements, and therefore they are erroneous using computers the representation of data as floating point numbers always produces errors. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

7 Remarks Linear systems Small perturbations always have to be kept in mind when solving practical problems since the data A and/or b may be obtained from measurements, and therefore they are erroneous using computers the representation of data as floating point numbers always produces errors. Hence, one always has to emanate from the fact that one solves a perturbed linear system instead of the given one. However, usually the pertubations are quite small. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

8 Perturbation lemma Linear systems Lemma Let B R (n,n), and assume that for some vector norm and the associate matrix norm the following inequality is satisfied B < 1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

9 Perturbation lemma Linear systems Lemma Let B R (n,n), and assume that for some vector norm and the associate matrix norm the following inequality is satisfied B < 1. Then the matrix I B is nonsingular, and it holds that (I B) B. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

10 Proof Linear systems For every x R n, x 0, (I B)x x Bx x B x = (1 B ) x > 0. Therefore, the linear system (I B)x = 0 has the unique solution x = 0, and I B is nonsingular. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

11 Proof Linear systems For every x R n, x 0, (I B)x x Bx x B x = (1 B ) x > 0. Therefore, the linear system (I B)x = 0 has the unique solution x = 0, and I B is nonsingular. The estimate of the norm of the inverse of I B follows from TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

12 Proof Linear systems For every x R n, x 0, (I B)x x Bx x B x = (1 B ) x > 0. Therefore, the linear system (I B)x = 0 has the unique solution x = 0, and I B is nonsingular. The estimate of the norm of the inverse of I B follows from 1 = (I B) 1 (I B) = (I B) 1 (I B) 1 B (I B) 1 (I B) 1 B (I B) 1 (I B) 1 B = (1 B ) (I B) 1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

13 Corollary Linear systems Let A R n be a nonsingular matrix, and A R n. Assume that A 1 A 1 for a matrix norm which is subordinate to some vector norm. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

14 Corollary Linear systems Let A R n be a nonsingular matrix, and A R n. Assume that A 1 A 1 for a matrix norm which is subordinate to some vector norm. Then A + A is nonsingular, and it holds that (A + A) 1 A 1 1 A 1 A A 1 1 A 1 A TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

15 Proof Linear systems The existence of (A + A) 1 follows from the perturbation lemma since A < 1 A 1 1 > A 1 A A 1 A and A + A = A(I + A 1 A). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

16 Proof Linear systems The existence of (A + A) 1 follows from the perturbation lemma since A < 1 A 1 1 > A 1 A A 1 A and A + A = A(I + A 1 A). (A + A) 1 = (I + A 1 A) 1 A 1 A 1 (I + A 1 A) 1 A 1 1 A 1 A A 1 1 A 1 A TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

17 Remark Linear systems The Corollary demonstrates that for a nonsingular matrix A the perturbed matrix A + A is also nonsingular if the perturbation A is sufficiently small. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

18 Linear systems Perturbed linear system We consider the perturbed linear system (A + A)(x + x) = b + b, and we assume that the perturbation A is so small that the condition of the Corollary is satisfied. Then A + A is nonsingular. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

19 Linear systems Perturbed linear system We consider the perturbed linear system (A + A)(x + x) = b + b, and we assume that the perturbation A is so small that the condition of the Corollary is satisfied. Then A + A is nonsingular. Solving for x one obtains the absolute error which is caused by the perturbations of A and b: x = (A + A) 1 ( b Ax) = (I + A 1 A) 1 A 1 ( b Ax). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

20 Linear systems Perturbed linear system We consider the perturbed linear system (A + A)(x + x) = b + b, and we assume that the perturbation A is so small that the condition of the Corollary is satisfied. Then A + A is nonsingular. Solving for x one obtains the absolute error which is caused by the perturbations of A and b: x = (A + A) 1 ( b Ax) = (I + A 1 A) 1 A 1 ( b Ax). Hence, with an arbitrary vector norm and the subordinate matrix norm we obtain x (I + A 1 A) 1 A 1 ( b + A x ). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

21 Linear systems Perturbed linear system ct. For b 0 and as a consequence x 0 it holds for the relative error x / x that ( ) x b x (I + A 1 A) 1 A 1 x + A. (3) TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

22 Linear systems Perturbed linear system ct. For b 0 and as a consequence x 0 it holds for the relative error x / x that ( ) x b x (I + A 1 A) 1 A 1 x + A. (3) and the Corollary yields x x A 1 ( 1 A 1 A b A A 1 A 1 A 1 A A A b + A ) ( A A + b ). (4) b TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

23 Linear systems Perturbed linear system ct. For b 0 and as a consequence x 0 it holds for the relative error x / x that ( ) x b x (I + A 1 A) 1 A 1 x + A. (3) and the Corollary yields x x A 1 ( 1 A 1 A b A A 1 A 1 A 1 A A A b + A ) ( A A + b ). (4) b Hence, for small perturbations (such that the denominator does not deviate very much from 1) the relative error b of the right hand side and the relative error A of the system matrix are amplified by the factor A 1 A. This amplification factor is called condition of the matrix A. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

24 Definition Linear systems Let A C (n,n) be a nonsingular matrix, and let be a matrix norm on C (n,n) which is subordinate to some vector norm. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

25 Definition Linear systems Let A C (n,n) be a nonsingular matrix, and let be a matrix norm on C (n,n) which is subordinate to some vector norm. Then κ(a) := A 1 A is called condition of the matrix A (or of the linear system of equations (1)) corresponding to the norm. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

26 Definition Linear systems Let A C (n,n) be a nonsingular matrix, and let be a matrix norm on C (n,n) which is subordinate to some vector norm. Then κ(a) := A 1 A is called condition of the matrix A (or of the linear system of equations (1)) corresponding to the norm. Remark For every nonsingular matrix A and every norm it holds that κ(a) 1, because 1 = I = AA 1 A A 1 = κ(a). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

27 Theorem Linear systems Let A, A R (n,n) and b, b R n, b 0, such that A is nonsingular, and assume that A 1 A < 1 for some matrix norm which is subordinate to some vector norm. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

28 Theorem Linear systems Let A, A R (n,n) and b, b R n, b 0, such that A is nonsingular, and assume that A 1 A < 1 for some matrix norm which is subordinate to some vector norm. Let x and x + x be the solution of the linear system (1) and the perturbed system (2), respectively, and the following estimation of the relative error holds x x κ(a) ( A 1 κ(a) A A A where κ(a) := A A 1 denotes the condition of A. + b ). b TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

29 Remark Linear systems Assume that the length of the mantissa (i.e. the number of leading digits in floating point representation) of our computer is l. Then that the relative input data error of A and b is 5 10 l. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

30 Remark Linear systems Assume that the length of the mantissa (i.e. the number of leading digits in floating point representation) of our computer is l. Then that the relative input data error of A and b is 5 10 l. If κ(a) = 10 γ, then (not considering the round of errors which occur in the numerical method for solving the linear system) we have to expect a relative error of approximately 5 10 γ l for a numerical solution the linear system Ax = b. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

31 Remark Linear systems Assume that the length of the mantissa (i.e. the number of leading digits in floating point representation) of our computer is l. Then that the relative input data error of A and b is 5 10 l. If κ(a) = 10 γ, then (not considering the round of errors which occur in the numerical method for solving the linear system) we have to expect a relative error of approximately 5 10 γ l for a numerical solution the linear system Ax = b. Roughly speaking solving a linear system numerically we are loosing γ digits Stellen if the order of magnitude of the condition of the system matrix A is 10 γ. This loss of accuracy has nothing to do with the algorithm of choice. It is problem immanent. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

32 Example Linear systems Consider the linear system of equations ( ) 1 1 x = which obviously has the solution x = (1, 1) T. ( ) 2, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

33 Example Linear systems Consider the linear system of equations ( ) 1 1 x = which obviously has the solution x = (1, 1) T. ( ) 2, For x + x := (5, 3.002) T it holds that ( ) A(x + x) = =: b + b TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

34 Example Linear systems Consider the linear system of equations ( ) 1 1 x = which obviously has the solution x = (1, 1) T. ( ) 2, For x + x := (5, 3.002) T it holds that ( ) A(x + x) = =: b + b Hence, b b = and x x = 4.002, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

35 Example ct. Linear systems and it follows for the condition κ (A) = TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

36 Example ct. Linear systems and it follows for the condition κ (A) = Indeed and therefore A 1 = ( 999 ) κ (A) = TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

37 Example ct. Linear systems and it follows for the condition κ (A) = Indeed and therefore A 1 = ( 999 ) κ (A) = This example demonstrates that the estimation of the relative error of the solution of a perturbed system is sharp. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

38 Linear systems Geometric condition The following Theorem contains a geometric characterization of the condition number. It says that the relative distance of a nonsingular matrix to the closest singular matrix with respect to Euclidean norm is the reciprokal of the condition number. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

39 Linear systems Geometric condition The following Theorem contains a geometric characterization of the condition number. It says that the relative distance of a nonsingular matrix to the closest singular matrix with respect to Euclidean norm is the reciprokal of the condition number. Theorem Let A R (n,n) be nonsingular. Then it holds that { A 2 min A 2 } : A + A singular} = 1 κ 2 (A). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

40 Proof Linear systems It suffices to prove that min { A 2 : A + A singular} = 1/ A 1 2. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

41 Proof Linear systems It suffices to prove that min { A 2 : A + A singular} = 1/ A 1 2. That the minimum is at least 1/ A 1 2 follows from the perturbation lemma: for A 2 < 1/ A 1 2 it holds that 1 > A 2 A 1 2 A 1 A 2. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

42 Proof Linear systems It suffices to prove that min { A 2 : A + A singular} = 1/ A 1 2. That the minimum is at least 1/ A 1 2 follows from the perturbation lemma: for A 2 < 1/ A 1 2 it holds that 1 > A 2 A 1 2 A 1 A 2. Hence, and is invertible. I + A 1 A = A 1 (A + A), A + A TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

43 Proof ct. Linear systems We now construct a matrix A, such that A + Ais singular and A 2 = 1/ A 1 2 which demonstrates that the minimum is greater or equal to 1/ A 1 2. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

44 Proof ct. Linear systems We now construct a matrix A, such that A + Ais singular and A 2 = 1/ A 1 2 which demonstrates that the minimum is greater or equal to 1/ A 1 2. From A 1 2 = max x 0 A 1 x 2 x 2 it follows that there exists x satisfying x 2 = 1 and A 1 2 = A 1 x 2 > 0. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

45 Proof ct. Linear systems We now construct a matrix A, such that A + Ais singular and A 2 = 1/ A 1 2 which demonstrates that the minimum is greater or equal to 1/ A 1 2. From A 1 2 = max x 0 A 1 x 2 x 2 it follows that there exists x satisfying x 2 = 1 and A 1 2 = A 1 x 2 > 0. With this x we define y := A 1 x A 1 = A 1 x x 2 A 1 and A := xy T 2 A 1. 2 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

46 Proof ct. Linear systems Then it holds that y 2 = 1 and A 2 = max z 0 xy T z 2 A 1 = max 2 z 2 z 0 where the maximum is attained for z = y, e.g. y T z z 2 x 2 A 1 2 = 1 A 1 2, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

47 Proof ct. Linear systems Then it holds that y 2 = 1 and A 2 = max z 0 xy T z 2 A 1 = max 2 z 2 z 0 where the maximum is attained for z = y, e.g. y T z z 2 x 2 A 1 2 = 1 A 1 2, From (A + A)y = Ay xy T y A 1 2 = we obtain the singularity of A + A. x A 1 2 x A 1 2 = 0 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

48 Theorem Least squares problems Let A = UΣV H be the singular value decomposition of A R m n where σ 1 σ 2 σ r > σ r+1 = = σ min(m,n) = 0. Then it holds that TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

49 Theorem Least squares problems Let A = UΣV H be the singular value decomposition of A R m n where σ 1 σ 2 σ r > σ r+1 = = σ min(m,n) = 0. Then it holds that (i) rank(a) = r, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

50 Theorem Least squares problems Let A = UΣV H be the singular value decomposition of A R m n where σ 1 σ 2 σ r > σ r+1 = = σ min(m,n) = 0. Then it holds that (i) rank(a) = r, (ii) null(a) := {x R n : Ax = 0} = span{v r+1,..., v n }, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

51 Theorem Least squares problems Let A = UΣV H be the singular value decomposition of A R m n where σ 1 σ 2 σ r > σ r+1 = = σ min(m,n) = 0. Then it holds that (i) rank(a) = r, (ii) null(a) := {x R n : Ax = 0} = span{v r+1,..., v n }, (iii) range(a) := {Ax : x R n } = span{u 1,..., u r }, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

52 Theorem Least squares problems Let A = UΣV H be the singular value decomposition of A R m n where σ 1 σ 2 σ r > σ r+1 = = σ min(m,n) = 0. Then it holds that (i) rank(a) = r, (ii) null(a) := {x R n : Ax = 0} = span{v r+1,..., v n }, (iii) range(a) := {Ax : x R n } = span{u 1,..., u r }, (iv) A = r i=1 σ i u i (v i ) T = U r Σ r V T r with U r = (u 1,..., u r ), V r = (v 1,..., v r ), Σ r = diag(σ 1,..., σ r ), TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

53 Theorem Least squares problems Let A = UΣV H be the singular value decomposition of A R m n where σ 1 σ 2 σ r > σ r+1 = = σ min(m,n) = 0. Then it holds that (i) rank(a) = r, (ii) null(a) := {x R n : Ax = 0} = span{v r+1,..., v n }, (iii) range(a) := {Ax : x R n } = span{u 1,..., u r }, (iv) A = r i=1 (v) A 2 S := m σ i u i (v i ) T = U r Σ r V T r with U r = (u 1,..., u r ), V r = (v 1,..., v r ), Σ r = diag(σ 1,..., σ r ), i=1 j=1 n aij 2 = r σi 2, i=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

54 Theorem Least squares problems Let A = UΣV H be the singular value decomposition of A R m n where σ 1 σ 2 σ r > σ r+1 = = σ min(m,n) = 0. Then it holds that (i) rank(a) = r, (ii) null(a) := {x R n : Ax = 0} = span{v r+1,..., v n }, (iii) range(a) := {Ax : x R n } = span{u 1,..., u r }, (iv) A = r i=1 (v) A 2 S := m (vi) A 2 := σ 1. σ i u i (v i ) T = U r Σ r V T r with U r = (u 1,..., u r ), V r = (v 1,..., v r ), Σ r = diag(σ 1,..., σ r ), i=1 j=1 n aij 2 = r σi 2, i=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

55 Proof Least squares problems (i): Multiplication by nonsingular matrices U T und V does not change the rank of A. Therefore, rank(a) = rank(σ) = r. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

56 Proof Least squares problems (i): Multiplication by nonsingular matrices U T und V does not change the rank of A. Therefore, rank(a) = rank(σ) = r. (ii): From V T v i = e i it follows that Av i = UΣV T v i = UΣe i = 0 for i = r + 1,..., n. Hence, v r+1,..., v n null(a). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

57 Proof Least squares problems (i): Multiplication by nonsingular matrices U T und V does not change the rank of A. Therefore, rank(a) = rank(σ) = r. (ii): From V T v i = e i it follows that Av i = UΣV T v i = UΣe i = 0 for i = r + 1,..., n. Hence, v r+1,..., v n null(a). dim null(a) = n r implies that these vectors form a basis of null(a). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

58 Proof ct. Least squares problems (iii): From A = UΣV T we obtain Range(A) = U Range(Σ) = U span(e 1,..., e r ) = span(u 1,..., u r ). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

59 Proof ct. Least squares problems (iii): From A = UΣV T we obtain Range(A) = U Range(Σ) = U span(e 1,..., e r ) = span(u 1,..., u r ). (iv): Blockmatrix multiplication yields A = UΣV T = ( (v 1 ) T u 1... u m) Σ. = (v n ) T r σ i u i (v i ) T. i=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

60 Proof ct. Least squares problems (v): Let A = (a 1,..., a n ). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

61 Proof ct. Least squares problems (v): Let A = (a 1,..., a n ). Multiplication by the orthogonal matrix U T does not change the Euclidean length of a vector. Jence, A 2 S = n a i 2 2 = i=1 n U T a i 2 2 = U T A 2 S. i=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

62 Proof ct. Least squares problems (v): Let A = (a 1,..., a n ). Multiplication by the orthogonal matrix U T does not change the Euclidean length of a vector. Jence, A 2 S = n a i 2 2 = i=1 n U T a i 2 2 = U T A 2 S. i=1 Similarly, multiplying the rows of U T A by the orthogonal matrix V from the right does not change their length, from which we get A 2 S = UT ΣV 2 S = Σ 2 S = r σi 2. i=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

63 Proof ct. Least squares problems (v): Let A = (a 1,..., a n ). Multiplication by the orthogonal matrix U T does not change the Euclidean length of a vector. Jence, A 2 S = n a i 2 2 = i=1 n U T a i 2 2 = U T A 2 S. i=1 Similarly, multiplying the rows of U T A by the orthogonal matrix V from the right does not change their length, from which we get A 2 S = UT ΣV 2 S = Σ 2 S = r σi 2. i=1 (vi): A 2 is a singular value of A, i.e. A 2 σ 1 (cf. proof of the existence theorem of the SVD). Thus A 2 = max{ Ax 2 : x 2 = 1} Av 1 2 = σ 1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

64 Least squares problems Condition of a matrix Let A = UΣV T be the SVD of a nonsingular matrix A. Then A 1 = V Σ 1 U T is the SVD of A 1, from which we get A 2 = σ 1 and A 1 2 = 1 σ n. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

65 Least squares problems Condition of a matrix Let A = UΣV T be the SVD of a nonsingular matrix A. Then A 1 = V Σ 1 U T is the SVD of A 1, from which we get A 2 = σ 1 and A 1 2 = 1 σ n. Hence, the condition of A with respect to the Euclidean norm is κ 2 (A) := σ 1 σ n. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

66 Remark Least squares problems Let A R (n,n) have eigenvalues µ 1,..., µ n. Then it follows from Ax i = µ i x i µ i 2 = (Ax i ) H Ax i (x i ) H x i = (x i ) H A T Ax i (x i ) H x i. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

67 Remark Least squares problems Let A R (n,n) have eigenvalues µ 1,..., µ n. Then it follows from Ax i = µ i x i µ i 2 = (Ax i ) H Ax i (x i ) H x i = (x i ) H A T Ax i (x i ) H x i. Rayleigh s principle yields λ min x H A T Ax x H x λ max für alle x C n, x 0, where λ min and λ max is the minimal und maximal eigenvalue of A T A, respectively. Hence, σ 1 µ i σ n for every i. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

68 Remark Least squares problems Let A R (n,n) have eigenvalues µ 1,..., µ n. Then it follows from Ax i = µ i x i µ i 2 = (Ax i ) H Ax i (x i ) H x i = (x i ) H A T Ax i (x i ) H x i. Rayleigh s principle yields λ min x H A T Ax x H x λ max für alle x C n, x 0, where λ min and λ max is the minimal und maximal eigenvalue of A T A, respectively. Hence, σ 1 µ i σ n for every i. For symmetric A it folds that σ 1 = µ 1 and σ r = µ r. For non symmetric matrices this is in general not the case. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

69 Least squares problems Numerical computation The singular values of A are the square roots of the eigenvalues of A T A. Hence, in principle the SVD of A can be determined with any eigensolver. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

70 Least squares problems Numerical computation The singular values of A are the square roots of the eigenvalues of A T A. Hence, in principle the SVD of A can be determined with any eigensolver. To this end one has to evaluate A T A and AA T which is costly and which deteriorates the condition number considerably. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

71 Least squares problems Numerical computation The singular values of A are the square roots of the eigenvalues of A T A. Hence, in principle the SVD of A can be determined with any eigensolver. To this end one has to evaluate A T A and AA T which is costly and which deteriorates the condition number considerably. Actually, one uses an algorithm of Golub and Reinsch (1971), which takes advantage of the QR algorithm for computing the eigenvalues of A T A, but which avoids the explicit computation of A T A and AA T. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

72 Least squares problems Data compression The singular value decomposition can be used for data compression. This is based upon the following theorem: TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

73 Least squares problems Data compression The singular value decomposition can be used for data compression. This is based upon the following theorem: Theorem Let A = UΣV T be the singular value decomposition of A R m n, and let U = (u 1,..., u m ) and V = (v 1,..., v n ). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

74 Least squares problems Data compression The singular value decomposition can be used for data compression. This is based upon the following theorem: Theorem Let A = UΣV T be the singular value decomposition of A R m n, and let U = (u 1,..., u m ) and V = (v 1,..., v n ). Then for k < n A k := k σ j u j (v j ) T j=1 is the best approximation of A with rank(a k ) = k with respect to the spectral norm, and it holds that A A k 2 = σ k+1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

75 Proof Least squares problems It holds that A A k 2 = n j=k+1 σ j u j (v j ) T 2 = Udiag{0,..., 0, σ k+1,..., σ n }V T 2 = σ k+1, and it remains to show, that there does not exist a matrix of rank k, the distance to A of which is less than σ k+1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

76 Proof Least squares problems It holds that A A k 2 = n j=k+1 σ j u j (v j ) T 2 = Udiag{0,..., 0, σ k+1,..., σ n }V T 2 = σ k+1, and it remains to show, that there does not exist a matrix of rank k, the distance to A of which is less than σ k+1. Let B be any matrix with rank(b) = k. Then the dimension of the null space of B is n k. The dimension of span{v 1,..., v k+1 } is k + 1, and therefore the intersection of these two spaces contains a nontrivial vector w with w 2 = 1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

77 Proof Least squares problems It holds that A A k 2 = n j=k+1 σ j u j (v j ) T 2 = Udiag{0,..., 0, σ k+1,..., σ n }V T 2 = σ k+1, and it remains to show, that there does not exist a matrix of rank k, the distance to A of which is less than σ k+1. Let B be any matrix with rank(b) = k. Then the dimension of the null space of B is n k. The dimension of span{v 1,..., v k+1 } is k + 1, and therefore the intersection of these two spaces contains a nontrivial vector w with w 2 = 1. Hence, A B 2 2 (A B)w 2 2 = Aw 2 2 = UΣV T w 2 2 = Σ(V T w) 2 2 σk+1 V 2 T w 2 2 = σk+1. 2 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

78 Least squares problems Data compression Let A R (m,n) be a matrix the elements a ij of which are color values of pixels of a picture. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

79 Least squares problems Data compression Let A R (m,n) be a matrix the elements a ij of which are color values of pixels of a picture. If A = UΣV T is the singular value decomposition of A, then A k = k σ j u j (v j ) T, k = 1,..., min(n, m) j=1 is an approximation to A. The storage of A k requires only k (n + m + 1)/(n m) memory cells whereas A requires mn. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

80 Least squares problems Data compression Let A R (m,n) be a matrix the elements a ij of which are color values of pixels of a picture. If A = UΣV T is the singular value decomposition of A, then A k = k σ j u j (v j ) T, k = 1,..., min(n, m) j=1 is an approximation to A. The storage of A k requires only k (n + m + 1)/(n m) memory cells whereas A requires mn. Notice that that using the SVD in this manner is a very simple way of data compression. There are algorithm in image processing which are much less costly. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

81 Example Least squares problems Original Figure: Original TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

82 Example ct. Least squares problems k=5; 2.6% Figure: Compression: k = 5 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

83 Example ct. Least squares problems k=10; 5.3% Figure: Compression: k = 10 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

84 Example ct. Least squares problems k=20; 10.5% Figure: Compression: k = 20 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

85 Pseudoinverse Least squares problems Consider the linear least squares problem Let A R (m,n) and b R m with m n. Find x R n such that Ax b 2 = min! (1) We examine this problem taking advantage of the singular value decomposition. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

86 Pseudoinverse Least squares problems Consider the linear least squares problem Let A R (m,n) and b R m with m n. Find x R n such that Ax b 2 = min! (1) We examine this problem taking advantage of the singular value decomposition. In the following we denote with σ 1 σ 2 σ r > 0 = σ r+1 = = σ n = 0 the singular values of A. A = UΣV T is the singular value decomposition of A, and u j and v k are the left and right singular vectors, respectively, i.e. the columns of U and V. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

87 Least squares problems Pseudoinverse ct. Theorem Let c := U T b R m. The set of solutions of the linear least squares problem is L = x + null(a), (2) where x is the following particular solution (1): x := r i=1 c i σ i v i. (3) TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

88 Pseudoinverse Least squares problems Multiplying a vector by an orthogonal matrix does not change its length. Hence, with z := V T x it holds that Ax b 2 2 = U T (Ax b) 2 2 = ΣV T x U T b 2 2 = Σz c 2 2 = (σ 1 z 1 c 1,..., σ r z r c r, c r+1,..., c m ) T 2 2. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

89 Pseudoinverse Least squares problems Multiplying a vector by an orthogonal matrix does not change its length. Hence, with z := V T x it holds that Ax b 2 2 = U T (Ax b) 2 2 = ΣV T x U T b 2 2 = Σz c 2 2 = (σ 1 z 1 c 1,..., σ r z r c r, c r+1,..., c m ) T 2 2. Therefore, the solution of problem (1) reads: z i := c i σ i, i = 1,..., r, und z i R beliebig für i = r + 1,..., n, i.e. x = r i=1 c i σ i v i + n i=r+1 z i v i, z i R, i = r + 1,..., n. (4) TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

90 Pseudoinverse Least squares problems Multiplying a vector by an orthogonal matrix does not change its length. Hence, with z := V T x it holds that Ax b 2 2 = U T (Ax b) 2 2 = ΣV T x U T b 2 2 = Σz c 2 2 = (σ 1 z 1 c 1,..., σ r z r c r, c r+1,..., c m ) T 2 2. Therefore, the solution of problem (1) reads: z i := c i σ i, i = 1,..., r, und z i R beliebig für i = r + 1,..., n, i.e. x = r i=1 c i σ i v i + n i=r+1 z i v i, z i R, i = r + 1,..., n. (4) Since the tailing n r columns of V span the null space of A, the set L of solutions of problem (1) has the form (2), (3). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

91 Least squares problems Pseudonormal solution This theorem demonstrates again that the linear least squares problem (1) has a unique solution if and only if r = rank(a) = n. We enforce the uniqueness also in the case r < n requiring additionally that the Euclidean norm of the solution is minimal. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

92 Least squares problems Pseudonormal solution This theorem demonstrates again that the linear least squares problem (1) has a unique solution if and only if r = rank(a) = n. We enforce the uniqueness also in the case r < n requiring additionally that the Euclidean norm of the solution is minimal. Definition Let L be the solution set of the linear least squares problem (1). x L is called pseudonormal solution of (1), if x 2 x 2 for every x L. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

93 Least squares problems Pseudonormal solution ct. The representation (4) of the general solution of (1) yields that x in (3) is the pseudonormal solution of (1): n n x + z i v i 2 2 = x z i 2 v i 2 2 x 2 2. i=r+1 i=r+1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

94 Least squares problems Pseudonormal solution ct. The representation (4) of the general solution of (1) yields that x in (3) is the pseudonormal solution of (1): n n x + z i v i 2 2 = x z i 2 v i 2 2 x 2 2. i=r+1 i=r+1 The pseudonormal solution is unique, and x obviously is the only solution of (1) with x null(a) L. Hence, we obtained TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

95 Least squares problems Pseudonormal solution ct. The representation (4) of the general solution of (1) yields that x in (3) is the pseudonormal solution of (1): n n x + z i v i 2 2 = x z i 2 v i 2 2 x 2 2. i=r+1 i=r+1 The pseudonormal solution is unique, and x obviously is the only solution of (1) with x null(a) L. Hence, we obtained Satz There exists a unique pseudonormal solution x of problem (1) which is characterized by x null(a) L. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

96 Pseudoinverse Least squares problems For every A R (m,n) R m b x R n : A x b 2 Ax b 2 x R n, x 2 minimal defines a mapping which obviously is linear (cf. the representation of x in (3)). Therefore, it can be representated as a matrix A R (n,m). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

97 Pseudoinverse Least squares problems For every A R (m,n) R m b x R n : A x b 2 Ax b 2 x R n, x 2 minimal defines a mapping which obviously is linear (cf. the representation of x in (3)). Therefore, it can be representated as a matrix A R (n,m). Definition For A R (m,n) the matrix A R (n,m), such that x := A b for every b R m is the pseudonormal solution of the linear least squares problem (1) is called pseudo inverse ( or Moore-Penrose inverse) of A. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

98 Pseudo inverse Least squares problems If rank(a) = n and m n, then the least squares problem (1) is uniquely solvable, and it follows from the normal equations that the solution is x = (A T A) 1 A T b. Hence, in this case A = (A T A) 1 A T. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

99 Pseudo inverse Least squares problems If rank(a) = n and m n, then the least squares problem (1) is uniquely solvable, and it follows from the normal equations that the solution is x = (A T A) 1 A T b. Hence, in this case A = (A T A) 1 A T. If n = m and A is nonsingular, then it holds that A = A 1. Hence, the pseudo inverse is the usual inverse, if this one exists, and the pseudo inverse is consistent extension of the inverse. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

100 Least squares problems Pseudo inverse ct. Theorem let A R (m,n) and A = UΣV T, its singular value decomposition Σ = (σ i δ ij ) i,j TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

101 Least squares problems Pseudo inverse ct. Theorem let A R (m,n) and A = UΣV T, its singular value decomposition Σ = (σ i δ ij ) i,j Then it holds that (i) Σ = (τ i δ ij ) j,i, τ i = { σ 1 i, falls σ i 0 0, falls σ i = 0, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

102 Least squares problems Pseudo inverse ct. Theorem let A R (m,n) and A = UΣV T, its singular value decomposition Σ = (σ i δ ij ) i,j Then it holds that (i) Σ = (τ i δ ij ) j,i, τ i = (ii) A = V Σ U T. { σ 1 i, falls σ i 0 0, falls σ i = 0, TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

103 Least squares problems Pseudo inverse ct. Remark The explicit representation of the pseudo inverse is needed only for theoretical considerations and is never computed explicitly (similarly to the inverse of a nonsingular matrix). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

104 Least squares problems Pseudo inverse ct. Remark The explicit representation of the pseudo inverse is needed only for theoretical considerations and is never computed explicitly (similarly to the inverse of a nonsingular matrix). Corollary For every matrix A R (m,n) it holds THAT A = A and (A ) T = (A T ). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

105 Least squares problems Pseudo inverse ct. Remark The explicit representation of the pseudo inverse is needed only for theoretical considerations and is never computed explicitly (similarly to the inverse of a nonsingular matrix). Corollary For every matrix A R (m,n) it holds THAT A = A and (A ) T = (A T ). A has the well known properties of the inverse A 1 of a nonsingular matrix A with the only exception that in general (AB) B A. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

106 Example Least squares problems Let A = ( ) 1 1 = I 0 0 ( ) ( ) 1 1 2, 1 1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

107 Example Least squares problems Let A = ( ) 1 1 = I 0 0 ( ) ( ) 1 1 2, 1 1 Its pseudo inverse is A = 1 ( ) ( ) 1 1 1/ 2 0 I = ( ) TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

108 Example Least squares problems Let A = ( ) 1 1 = I 0 0 ( ) ( ) 1 1 2, 1 1 Its pseudo inverse is A = 1 ( ) ( ) 1 1 1/ 2 0 I = ( ) Then A 2 = A and (A ) 2 = 1 2 A, i.e. (A 2 ) (A ) 2. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

109 Least squares problems Perturbation of least squares problems Consider the linear least squares problem Ax b 2 = min! (1) with A R (m,n), rank(a) = r, and a perturbed problem A(x + x) (b + b) 2 = min!, (2) where we incorporate only perturbations of the right hand side b, but not of the system matrix A. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

110 Least squares problems Perturbation of least squares problems Consider the linear least squares problem Ax b 2 = min! (1) with A R (m,n), rank(a) = r, and a perturbed problem A(x + x) (b + b) 2 = min!, (2) where we incorporate only perturbations of the right hand side b, but not of the system matrix A. Let x = A b and x + x = A (b + b) the pseudo normal solution of (1) and (2), respectively. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

111 Least squares problems Perturbation of least squares problems Consider the linear least squares problem Ax b 2 = min! (1) with A R (m,n), rank(a) = r, and a perturbed problem A(x + x) (b + b) 2 = min!, (2) where we incorporate only perturbations of the right hand side b, but not of the system matrix A. Let x = A b and x + x = A (b + b) the pseudo normal solution of (1) and (2), respectively. Then x = A b, and from A 2 = 1 σ r it follows x 2 A 2 b 2 = 1 σ r b 2. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

112 Least squares problems Perturbation of least squares problems It holds that r x 2 2 = c 2 i σ 2 i=1 i 1 σ 2 1 r ci 2 = 1 r 2 σ1 2 c i u i. i=1 i=1 2 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

113 Least squares problems Perturbation of least squares problems It holds that Obviously, x 2 2 = r c 2 i σ 2 i=1 i 1 σ 2 1 r ci 2 = 1 r 2 σ1 2 c i u i. i=1 r c i u i is the projection of b to the range of A. Therefore it follows i=1 for the relative error i=1 x 2 σ 1 b 2. (3) x 2 σ r P range(a)b 2 2 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

114 Least squares problems Perturbation of least squares problems It holds that Obviously, x 2 2 = r c 2 i σ 2 i=1 i 1 σ 2 1 r ci 2 = 1 r 2 σ1 2 c i u i. i=1 r c i u i is the projection of b to the range of A. Therefore it follows i=1 for the relative error i=1 x 2 σ 1 b 2. (3) x 2 σ r P range(a)b 2 2 This inequality specifies, how the relative error of the right hand side of a linear least squares problem effects the solution of the problem TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

115 condition Least squares problems Definition For A R (m,n) let A = UΣV T be the singular value decomposition, and let rank(a) = r. Then κ 2 (A) := σ 1 σ r is called condition of A. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

116 condition Least squares problems Definition For A R (m,n) let A = UΣV T be the singular value decomposition, and let rank(a) = r. Then κ 2 (A) := σ 1 σ r is called condition of A. If A R (n,n) is nonsingular then this definition coincides with the one with respect to the Euclidean norm for quadratic matrices given before. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

117 condition Least squares problems Definition For A R (m,n) let A = UΣV T be the singular value decomposition, and let rank(a) = r. Then κ 2 (A) := σ 1 σ r is called condition of A. If A R (n,n) is nonsingular then this definition coincides with the one with respect to the Euclidean norm for quadratic matrices given before. κ 2 (A T A) = κ 2 (A) 2 Hence, the normal equation of a linear least squares problem are much worse conditioned than the system matrix of the problem. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

118 Least squares problems Perturbed least squares problems For perturbations of the system matrix the following theorem holds TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

119 Least squares problems Perturbed least squares problems For perturbations of the system matrix the following theorem holds Theorem Assume that A R (m,n), m n, is not deficient, i.e. rank(a) = n. Let x be the solution of the least squares problem (1) and x be the solution of the perturbed problem (A + A)x (b + b) 2 = min!, (4) where ( A 2 ε := max, b ) 2 < 1 A 2 b 2 κ 2 (A) = σ n(a) σ 1 (A). (5) TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

120 Least squares problems Perturbed least squares problems For perturbations of the system matrix the following theorem holds Theorem Assume that A R (m,n), m n, is not deficient, i.e. rank(a) = n. Let x be the solution of the least squares problem (1) and x be the solution of the perturbed problem (A + A)x (b + b) 2 = min!, (4) where ( A 2 ε := max, b ) 2 < 1 A 2 b 2 κ 2 (A) = σ n(a) σ 1 (A). (5) Then it holds that x x 2 x 2 ( ) 2κ2 (A) ε cos θ + tan θ κ2 2(A) + O(ε 2 ), (6) where θ is the angle between b and its projection to range(a). TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

121 Example Regularization Consider the orthogonal projection of a given function f : [0, 1] R to the space Π n 1 of polynomial of maximum degree n 1 with respect to the scalar product f, g := 1 0 f (x)g(x) dx. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

122 Example Regularization Consider the orthogonal projection of a given function f : [0, 1] R to the space Π n 1 of polynomial of maximum degree n 1 with respect to the scalar product f, g := 1 0 f (x)g(x) dx. Choosing the basis {1, x,..., x n 1 } one obtains the linear system with Ay = b (1) A = (a ij ) i,j=1,...,n, a ij := 1 i + j 1, (2) the so called Hilbert Matrix, and b R n, b i := f, x i 1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

123 Example ct. Regularization For dimensions n = 10, n = 20 and n = 40 we choose the right hand side of (1) such that y = (1,..., 1) T is the unique solution, and we solve the resulting system by the known methods. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

124 Example ct. Regularization For dimensions n = 10, n = 20 and n = 40 we choose the right hand side of (1) such that y = (1,..., 1) T is the unique solution, and we solve the resulting system by the known methods. The LU factorization with column pivoting (in MATLAB A\b), the Cholesky factorization, the QR decomposition of A and the singular value decomposition of A yield the following errors with respect to the Euclidean norm: TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

125 Example ct. Regularization For dimensions n = 10, n = 20 and n = 40 we choose the right hand side of (1) such that y = (1,..., 1) T is the unique solution, and we solve the resulting system by the known methods. The LU factorization with column pivoting (in MATLAB A\b), the Cholesky factorization, the QR decomposition of A and the singular value decomposition of A yield the following errors with respect to the Euclidean norm: n = 10 n = 20 n = 40 LU factorization 5.24 E E E+2 Cholesky 7.15 E-4 numer. nicht pos. def. QR decomposition 1.41 E E E+3 SVD 8.24 E E E+2 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

126 Example ct. Regularization A similar behavior is observed for the least square problem. For n = 10, n = 20 and n = 40 and m = n + 10 we consider the least squares problem Ax b 2 = min! with the Hilbert matrix A R (m,n), where b is chosen such that x = (1,..., 1) T solves the problem with residual Ax b = 0. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

127 Example ct. Regularization A similar behavior is observed for the least square problem. For n = 10, n = 20 and n = 40 and m = n + 10 we consider the least squares problem Ax b 2 = min! with the Hilbert matrix A R (m,n), where b is chosen such that x = (1,..., 1) T solves the problem with residual Ax b = 0. n = 10 n = 20 n = 40 Normal equations 2.91 E E E+2 QR factorization 1.93 E E E+1 SVD 4.67 E E E+2 TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

128 Regularization Regularization For badly conditioned least squares problem or linear systems the following approach can yield reliable solutions: TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

129 Regularization Regularization For badly conditioned least squares problem or linear systems the following approach can yield reliable solutions: Determine the singular value decomposition A = UΣVT of A, and define { Σ σ 1 τ = diag(η i δ ji ), η i := i falls σ i τ, 0 sonst, where τ > 0 is a given threshold, and A τ := V Σ τ U T, x τ := A τ b. TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

130 Regularization Regularization For badly conditioned least squares problem or linear systems the following approach can yield reliable solutions: Determine the singular value decomposition A = UΣVT of A, and define { Σ σ 1 τ = diag(η i δ ji ), η i := i falls σ i τ, 0 sonst, where τ > 0 is a given threshold, and A τ := V Σ τ U T, x τ := A τ b. A τ is called effektive pseudo inverse of A. This method of approximatively solving Ax = b is called regularization by truncation TUHH Heinrich Voss Numerical Linear Algebra Chap. 4: Perturbation and Regularisation / 55

Matrix Norms. Tom Lyche. September 28, Centre of Mathematics for Applications, Department of Informatics, University of Oslo

Matrix Norms. Tom Lyche. September 28, Centre of Mathematics for Applications, Department of Informatics, University of Oslo Matrix Norms Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 28, 2009 Matrix Norms We consider matrix norms on (C m,n, C). All results holds for

More information

The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression

The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression The SVD is the most generally applicable of the orthogonal-diagonal-orthogonal type matrix decompositions Every

More information

NUMERICALLY EFFICIENT METHODS FOR SOLVING LEAST SQUARES PROBLEMS

NUMERICALLY EFFICIENT METHODS FOR SOLVING LEAST SQUARES PROBLEMS NUMERICALLY EFFICIENT METHODS FOR SOLVING LEAST SQUARES PROBLEMS DO Q LEE Abstract. Computing the solution to Least Squares Problems is of great importance in a wide range of fields ranging from numerical

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 3 Linear Least Squares Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

More information

4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns

4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns L. Vandenberghe EE133A (Spring 2016) 4. Matrix inverses left and right inverse linear independence nonsingular matrices matrices with linearly independent columns matrices with linearly independent rows

More information

NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

More information

NOTES on LINEAR ALGEBRA 1

NOTES on LINEAR ALGEBRA 1 School of Economics, Management and Statistics University of Bologna Academic Year 205/6 NOTES on LINEAR ALGEBRA for the students of Stats and Maths This is a modified version of the notes by Prof Laura

More information

13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions.

13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions. 3 MATH FACTS 0 3 MATH FACTS 3. Vectors 3.. Definition We use the overhead arrow to denote a column vector, i.e., a linear segment with a direction. For example, in three-space, we write a vector in terms

More information

CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

More information

Inner Product Spaces and Orthogonality

Inner Product Spaces and Orthogonality Inner Product Spaces and Orthogonality week 3-4 Fall 2006 Dot product of R n The inner product or dot product of R n is a function, defined by u, v a b + a 2 b 2 + + a n b n for u a, a 2,, a n T, v b,

More information

Factorization Theorems

Factorization Theorems Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization

More information

Lecture 6. Inverse of Matrix

Lecture 6. Inverse of Matrix Lecture 6 Inverse of Matrix Recall that any linear system can be written as a matrix equation In one dimension case, ie, A is 1 1, then can be easily solved as A x b Ax b x b A 1 A b A 1 b provided that

More information

Chapter 6. Orthogonality

Chapter 6. Orthogonality 6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

More information

Sec 4.1 Vector Spaces and Subspaces

Sec 4.1 Vector Spaces and Subspaces Sec 4. Vector Spaces and Subspaces Motivation Let S be the set of all solutions to the differential equation y + y =. Let T be the set of all 2 3 matrices with real entries. These two sets share many common

More information

3 Orthogonal Vectors and Matrices

3 Orthogonal Vectors and Matrices 3 Orthogonal Vectors and Matrices The linear algebra portion of this course focuses on three matrix factorizations: QR factorization, singular valued decomposition (SVD), and LU factorization The first

More information

Similarity and Diagonalization. Similar Matrices

Similarity and Diagonalization. Similar Matrices MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

More information

Vector and Matrix Norms

Vector and Matrix Norms Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

More information

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

More information

Linear Algebra Review. Vectors

Linear Algebra Review. Vectors Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

More information

Lecture 5: Singular Value Decomposition SVD (1)

Lecture 5: Singular Value Decomposition SVD (1) EEM3L1: Numerical and Analytical Techniques Lecture 5: Singular Value Decomposition SVD (1) EE3L1, slide 1, Version 4: 25-Sep-02 Motivation for SVD (1) SVD = Singular Value Decomposition Consider the system

More information

α = u v. In other words, Orthogonal Projection

α = u v. In other words, Orthogonal Projection Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v

More information

T ( a i x i ) = a i T (x i ).

T ( a i x i ) = a i T (x i ). Chapter 2 Defn 1. (p. 65) Let V and W be vector spaces (over F ). We call a function T : V W a linear transformation form V to W if, for all x, y V and c F, we have (a) T (x + y) = T (x) + T (y) and (b)

More information

CSE 494 CSE/CBS 598 (Fall 2007): Numerical Linear Algebra for Data Exploration Clustering Instructor: Jieping Ye

CSE 494 CSE/CBS 598 (Fall 2007): Numerical Linear Algebra for Data Exploration Clustering Instructor: Jieping Ye CSE 494 CSE/CBS 598 Fall 2007: Numerical Linear Algebra for Data Exploration Clustering Instructor: Jieping Ye 1 Introduction One important method for data compression and classification is to organize

More information

1 Introduction to Matrices

1 Introduction to Matrices 1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

More information

Notes on Symmetric Matrices

Notes on Symmetric Matrices CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.

More information

Applied Linear Algebra I Review page 1

Applied Linear Algebra I Review page 1 Applied Linear Algebra Review 1 I. Determinants A. Definition of a determinant 1. Using sum a. Permutations i. Sign of a permutation ii. Cycle 2. Uniqueness of the determinant function in terms of properties

More information

MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix.

MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix. MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix. Nullspace Let A = (a ij ) be an m n matrix. Definition. The nullspace of the matrix A, denoted N(A), is the set of all n-dimensional column

More information

1. True/False: Circle the correct answer. No justifications are needed in this exercise. (1 point each)

1. True/False: Circle the correct answer. No justifications are needed in this exercise. (1 point each) Math 33 AH : Solution to the Final Exam Honors Linear Algebra and Applications 1. True/False: Circle the correct answer. No justifications are needed in this exercise. (1 point each) (1) If A is an invertible

More information

The Moore-Penrose Inverse and Least Squares

The Moore-Penrose Inverse and Least Squares University of Puget Sound MATH 420: Advanced Topics in Linear Algebra The Moore-Penrose Inverse and Least Squares April 16, 2014 Creative Commons License c 2014 Permission is granted to others to copy,

More information

Au = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively.

Au = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively. Chapter 7 Eigenvalues and Eigenvectors In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and eigenvectors of matrices, concepts that were already introduced in Geometry

More information

9. Numerical linear algebra background

9. Numerical linear algebra background Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization

More information

Numerical Methods I Eigenvalue Problems

Numerical Methods I Eigenvalue Problems Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001, Fall 2010 September 30th, 2010 A. Donev (Courant Institute)

More information

7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization 7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

More information

Orthogonal Diagonalization of Symmetric Matrices

Orthogonal Diagonalization of Symmetric Matrices MATH10212 Linear Algebra Brief lecture notes 57 Gram Schmidt Process enables us to find an orthogonal basis of a subspace. Let u 1,..., u k be a basis of a subspace V of R n. We begin the process of finding

More information

Theta Functions. Lukas Lewark. Seminar on Modular Forms, 31. Januar 2007

Theta Functions. Lukas Lewark. Seminar on Modular Forms, 31. Januar 2007 Theta Functions Lukas Lewark Seminar on Modular Forms, 31. Januar 007 Abstract Theta functions are introduced, associated to lattices or quadratic forms. Their transformation property is proven and the

More information

Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Numerical Analysis Lecture Notes Peter J. Olver 6. Eigenvalues and Singular Values In this section, we collect together the basic facts about eigenvalues and eigenvectors. From a geometrical viewpoint,

More information

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

More information

Section 6.1 - Inner Products and Norms

Section 6.1 - Inner Products and Norms Section 6.1 - Inner Products and Norms Definition. Let V be a vector space over F {R, C}. An inner product on V is a function that assigns, to every ordered pair of vectors x and y in V, a scalar in F,

More information

by the matrix A results in a vector which is a reflection of the given

by the matrix A results in a vector which is a reflection of the given Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that

More information

CONTROLLABILITY. Chapter 2. 2.1 Reachable Set and Controllability. Suppose we have a linear system described by the state equation

CONTROLLABILITY. Chapter 2. 2.1 Reachable Set and Controllability. Suppose we have a linear system described by the state equation Chapter 2 CONTROLLABILITY 2 Reachable Set and Controllability Suppose we have a linear system described by the state equation ẋ Ax + Bu (2) x() x Consider the following problem For a given vector x in

More information

Solving Linear Systems, Continued and The Inverse of a Matrix

Solving Linear Systems, Continued and The Inverse of a Matrix , Continued and The of a Matrix Calculus III Summer 2013, Session II Monday, July 15, 2013 Agenda 1. The rank of a matrix 2. The inverse of a square matrix Gaussian Gaussian solves a linear system by reducing

More information

MAT 200, Midterm Exam Solution. a. (5 points) Compute the determinant of the matrix A =

MAT 200, Midterm Exam Solution. a. (5 points) Compute the determinant of the matrix A = MAT 200, Midterm Exam Solution. (0 points total) a. (5 points) Compute the determinant of the matrix 2 2 0 A = 0 3 0 3 0 Answer: det A = 3. The most efficient way is to develop the determinant along the

More information

Math 312 Homework 1 Solutions

Math 312 Homework 1 Solutions Math 31 Homework 1 Solutions Last modified: July 15, 01 This homework is due on Thursday, July 1th, 01 at 1:10pm Please turn it in during class, or in my mailbox in the main math office (next to 4W1) Please

More information

1 Eigenvalues and Eigenvectors

1 Eigenvalues and Eigenvectors Math 20 Chapter 5 Eigenvalues and Eigenvectors Eigenvalues and Eigenvectors. Definition: A scalar λ is called an eigenvalue of the n n matrix A is there is a nontrivial solution x of Ax = λx. Such an x

More information

Vector Spaces II: Finite Dimensional Linear Algebra 1

Vector Spaces II: Finite Dimensional Linear Algebra 1 John Nachbar September 2, 2014 Vector Spaces II: Finite Dimensional Linear Algebra 1 1 Definitions and Basic Theorems. For basic properties and notation for R N, see the notes Vector Spaces I. Definition

More information

Matrices, Determinants and Linear Systems

Matrices, Determinants and Linear Systems September 21, 2014 Matrices A matrix A m n is an array of numbers in rows and columns a 11 a 12 a 1n r 1 a 21 a 22 a 2n r 2....... a m1 a m2 a mn r m c 1 c 2 c n We say that the dimension of A is m n (we

More information

Matrix Representations of Linear Transformations and Changes of Coordinates

Matrix Representations of Linear Transformations and Changes of Coordinates Matrix Representations of Linear Transformations and Changes of Coordinates 01 Subspaces and Bases 011 Definitions A subspace V of R n is a subset of R n that contains the zero element and is closed under

More information

Systems of Linear Equations

Systems of Linear Equations Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

More information

Inner Product Spaces

Inner Product Spaces Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

More information

Lecture Note on Linear Algebra 15. Dimension and Rank

Lecture Note on Linear Algebra 15. Dimension and Rank Lecture Note on Linear Algebra 15. Dimension and Rank Wei-Shi Zheng, wszheng@ieee.org, 211 November 1, 211 1 What Do You Learn from This Note We still observe the unit vectors we have introduced in Chapter

More information

4.5 Linear Dependence and Linear Independence

4.5 Linear Dependence and Linear Independence 4.5 Linear Dependence and Linear Independence 267 32. {v 1, v 2 }, where v 1, v 2 are collinear vectors in R 3. 33. Prove that if S and S are subsets of a vector space V such that S is a subset of S, then

More information

Math 313 Lecture #10 2.2: The Inverse of a Matrix

Math 313 Lecture #10 2.2: The Inverse of a Matrix Math 1 Lecture #10 2.2: The Inverse of a Matrix Matrix algebra provides tools for creating many useful formulas just like real number algebra does. For example, a real number a is invertible if there is

More information

4.6 Null Space, Column Space, Row Space

4.6 Null Space, Column Space, Row Space NULL SPACE, COLUMN SPACE, ROW SPACE Null Space, Column Space, Row Space In applications of linear algebra, subspaces of R n typically arise in one of two situations: ) as the set of solutions of a linear

More information

17. Inner product spaces Definition 17.1. Let V be a real vector space. An inner product on V is a function

17. Inner product spaces Definition 17.1. Let V be a real vector space. An inner product on V is a function 17. Inner product spaces Definition 17.1. Let V be a real vector space. An inner product on V is a function, : V V R, which is symmetric, that is u, v = v, u. bilinear, that is linear (in both factors):

More information

3. Let A and B be two n n orthogonal matrices. Then prove that AB and BA are both orthogonal matrices. Prove a similar result for unitary matrices.

3. Let A and B be two n n orthogonal matrices. Then prove that AB and BA are both orthogonal matrices. Prove a similar result for unitary matrices. Exercise 1 1. Let A be an n n orthogonal matrix. Then prove that (a) the rows of A form an orthonormal basis of R n. (b) the columns of A form an orthonormal basis of R n. (c) for any two vectors x,y R

More information

5. Orthogonal matrices

5. Orthogonal matrices L Vandenberghe EE133A (Spring 2016) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal

More information

Matrix Differentiation

Matrix Differentiation 1 Introduction Matrix Differentiation ( and some other stuff ) Randal J. Barnes Department of Civil Engineering, University of Minnesota Minneapolis, Minnesota, USA Throughout this presentation I have

More information

2.1: MATRIX OPERATIONS

2.1: MATRIX OPERATIONS .: MATRIX OPERATIONS What are diagonal entries and the main diagonal of a matrix? What is a diagonal matrix? When are matrices equal? Scalar Multiplication 45 Matrix Addition Theorem (pg 0) Let A, B, and

More information

MATH 304 Linear Algebra Lecture 9: Subspaces of vector spaces (continued). Span. Spanning set.

MATH 304 Linear Algebra Lecture 9: Subspaces of vector spaces (continued). Span. Spanning set. MATH 304 Linear Algebra Lecture 9: Subspaces of vector spaces (continued). Span. Spanning set. Vector space A vector space is a set V equipped with two operations, addition V V (x,y) x + y V and scalar

More information

Solutions to Review Problems

Solutions to Review Problems Chapter 1 Solutions to Review Problems Chapter 1 Exercise 42 Which of the following equations are not linear and why: (a x 2 1 + 3x 2 2x 3 = 5. (b x 1 + x 1 x 2 + 2x 3 = 1. (c x 1 + 2 x 2 + x 3 = 5. (a

More information

MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets.

MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets. MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets. Norm The notion of norm generalizes the notion of length of a vector in R n. Definition. Let V be a vector space. A function α

More information

Recall that two vectors in are perpendicular or orthogonal provided that their dot

Recall that two vectors in are perpendicular or orthogonal provided that their dot Orthogonal Complements and Projections Recall that two vectors in are perpendicular or orthogonal provided that their dot product vanishes That is, if and only if Example 1 The vectors in are orthogonal

More information

Bindel, Spring 2012 Intro to Scientific Computing (CS 3220) Week 3: Wednesday, Feb 8

Bindel, Spring 2012 Intro to Scientific Computing (CS 3220) Week 3: Wednesday, Feb 8 Spaces and bases Week 3: Wednesday, Feb 8 I have two favorite vector spaces 1 : R n and the space P d of polynomials of degree at most d. For R n, we have a canonical basis: R n = span{e 1, e 2,..., e

More information

Lecture Notes: Matrix Inverse. 1 Inverse Definition

Lecture Notes: Matrix Inverse. 1 Inverse Definition Lecture Notes: Matrix Inverse Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk Inverse Definition We use I to represent identity matrices,

More information

Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain

Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain 1. Orthogonal matrices and orthonormal sets An n n real-valued matrix A is said to be an orthogonal

More information

1 VECTOR SPACES AND SUBSPACES

1 VECTOR SPACES AND SUBSPACES 1 VECTOR SPACES AND SUBSPACES What is a vector? Many are familiar with the concept of a vector as: Something which has magnitude and direction. an ordered pair or triple. a description for quantities such

More information

Diagonalisation. Chapter 3. Introduction. Eigenvalues and eigenvectors. Reading. Definitions

Diagonalisation. Chapter 3. Introduction. Eigenvalues and eigenvectors. Reading. Definitions Chapter 3 Diagonalisation Eigenvalues and eigenvectors, diagonalisation of a matrix, orthogonal diagonalisation fo symmetric matrices Reading As in the previous chapter, there is no specific essential

More information

INTRODUCTORY LINEAR ALGEBRA WITH APPLICATIONS B. KOLMAN, D. R. HILL

INTRODUCTORY LINEAR ALGEBRA WITH APPLICATIONS B. KOLMAN, D. R. HILL SOLUTIONS OF THEORETICAL EXERCISES selected from INTRODUCTORY LINEAR ALGEBRA WITH APPLICATIONS B. KOLMAN, D. R. HILL Eighth Edition, Prentice Hall, 2005. Dr. Grigore CĂLUGĂREANU Department of Mathematics

More information

LINEAR ALGEBRA & MATRICES

LINEAR ALGEBRA & MATRICES LINEAR ALGEBRA & MATRICES C.T. Abdallah, M. Ariola, F. Amato January 21, 2006 Contents 1 Review of Matrix Algebra 2 1.1 Determinants, Minors, and Cofactors................ 4 1.2 Rank, Trace, and Inverse......................

More information

Lecture Topic: Low-Rank Approximations

Lecture Topic: Low-Rank Approximations Lecture Topic: Low-Rank Approximations Low-Rank Approximations We have seen principal component analysis. The extraction of the first principle eigenvalue could be seen as an approximation of the original

More information

Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of

More information

Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Numerical Analysis Lecture Notes Peter J. Olver 5. Inner Products and Norms The norm of a vector is a measure of its size. Besides the familiar Euclidean norm based on the dot product, there are a number

More information

10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES

10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES 58 CHAPTER NUMERICAL METHODS. POWER METHOD FOR APPROXIMATING EIGENVALUES In Chapter 7 you saw that the eigenvalues of an n n matrix A are obtained by solving its characteristic equation n c nn c nn...

More information

DETERMINANTS. b 2. x 2

DETERMINANTS. b 2. x 2 DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

More information

x1 x 2 x 3 y 1 y 2 y 3 x 1 y 2 x 2 y 1 0.

x1 x 2 x 3 y 1 y 2 y 3 x 1 y 2 x 2 y 1 0. Cross product 1 Chapter 7 Cross product We are getting ready to study integration in several variables. Until now we have been doing only differential calculus. One outcome of this study will be our ability

More information

LINEAR ALGEBRA. September 23, 2010

LINEAR ALGEBRA. September 23, 2010 LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................

More information

6. Cholesky factorization

6. Cholesky factorization 6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix

More information

1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form

1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form 1. LINEAR EQUATIONS A linear equation in n unknowns x 1, x 2,, x n is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b, where a 1, a 2,..., a n, b are given real numbers. For example, with x and

More information

The determinant of a skew-symmetric matrix is a square. This can be seen in small cases by direct calculation: 0 a. 12 a. a 13 a 24 a 14 a 23 a 14

The determinant of a skew-symmetric matrix is a square. This can be seen in small cases by direct calculation: 0 a. 12 a. a 13 a 24 a 14 a 23 a 14 4 Symplectic groups In this and the next two sections, we begin the study of the groups preserving reflexive sesquilinear forms or quadratic forms. We begin with the symplectic groups, associated with

More information

LINEAR ALGEBRA W W L CHEN

LINEAR ALGEBRA W W L CHEN LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008 This chapter is available free to all individuals, on understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,

More information

The Inverse of a Square Matrix

The Inverse of a Square Matrix These notes closely follow the presentation of the material given in David C Lay s textbook Linear Algebra and its Applications (3rd edition) These notes are intended primarily for in-class presentation

More information

18.06 Problem Set 4 Solution Due Wednesday, 11 March 2009 at 4 pm in 2-106. Total: 175 points.

18.06 Problem Set 4 Solution Due Wednesday, 11 March 2009 at 4 pm in 2-106. Total: 175 points. 806 Problem Set 4 Solution Due Wednesday, March 2009 at 4 pm in 2-06 Total: 75 points Problem : A is an m n matrix of rank r Suppose there are right-hand-sides b for which A x = b has no solution (a) What

More information

MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued).

MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued). MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors Jordan canonical form (continued) Jordan canonical form A Jordan block is a square matrix of the form λ 1 0 0 0 0 λ 1 0 0 0 0 λ 0 0 J = 0

More information

MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1. MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

More information

MATH 551 - APPLIED MATRIX THEORY

MATH 551 - APPLIED MATRIX THEORY MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points

More information

1 Sets and Set Notation.

1 Sets and Set Notation. LINEAR ALGEBRA MATH 27.6 SPRING 23 (COHEN) LECTURE NOTES Sets and Set Notation. Definition (Naive Definition of a Set). A set is any collection of objects, called the elements of that set. We will most

More information

Linear Algebra: Determinants, Inverses, Rank

Linear Algebra: Determinants, Inverses, Rank D Linear Algebra: Determinants, Inverses, Rank D 1 Appendix D: LINEAR ALGEBRA: DETERMINANTS, INVERSES, RANK TABLE OF CONTENTS Page D.1. Introduction D 3 D.2. Determinants D 3 D.2.1. Some Properties of

More information

Methods for Finding Bases

Methods for Finding Bases Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,

More information

160 CHAPTER 4. VECTOR SPACES

160 CHAPTER 4. VECTOR SPACES 160 CHAPTER 4. VECTOR SPACES 4. Rank and Nullity In this section, we look at relationships between the row space, column space, null space of a matrix and its transpose. We will derive fundamental results

More information

We seek a factorization of a square matrix A into the product of two matrices which yields an

We seek a factorization of a square matrix A into the product of two matrices which yields an LU Decompositions We seek a factorization of a square matrix A into the product of two matrices which yields an efficient method for solving the system where A is the coefficient matrix, x is our variable

More information

Lecture 4: Partitioned Matrices and Determinants

Lecture 4: Partitioned Matrices and Determinants Lecture 4: Partitioned Matrices and Determinants 1 Elementary row operations Recall the elementary operations on the rows of a matrix, equivalent to premultiplying by an elementary matrix E: (1) multiplying

More information

Constrained Least Squares

Constrained Least Squares Constrained Least Squares Authors: G.H. Golub and C.F. Van Loan Chapter 12 in Matrix Computations, 3rd Edition, 1996, pp.580-587 CICN may05/1 Background The least squares problem: min Ax b 2 x Sometimes,

More information

Recall the basic property of the transpose (for any A): v A t Aw = v w, v, w R n.

Recall the basic property of the transpose (for any A): v A t Aw = v w, v, w R n. ORTHOGONAL MATRICES Informally, an orthogonal n n matrix is the n-dimensional analogue of the rotation matrices R θ in R 2. When does a linear transformation of R 3 (or R n ) deserve to be called a rotation?

More information

WHICH LINEAR-FRACTIONAL TRANSFORMATIONS INDUCE ROTATIONS OF THE SPHERE?

WHICH LINEAR-FRACTIONAL TRANSFORMATIONS INDUCE ROTATIONS OF THE SPHERE? WHICH LINEAR-FRACTIONAL TRANSFORMATIONS INDUCE ROTATIONS OF THE SPHERE? JOEL H. SHAPIRO Abstract. These notes supplement the discussion of linear fractional mappings presented in a beginning graduate course

More information

Linear Codes. In the V[n,q] setting, the terms word and vector are interchangeable.

Linear Codes. In the V[n,q] setting, the terms word and vector are interchangeable. Linear Codes Linear Codes In the V[n,q] setting, an important class of codes are the linear codes, these codes are the ones whose code words form a sub-vector space of V[n,q]. If the subspace of V[n,q]

More information

MATH1231 Algebra, 2015 Chapter 7: Linear maps

MATH1231 Algebra, 2015 Chapter 7: Linear maps MATH1231 Algebra, 2015 Chapter 7: Linear maps A/Prof. Daniel Chan School of Mathematics and Statistics University of New South Wales danielc@unsw.edu.au Daniel Chan (UNSW) MATH1231 Algebra 1 / 43 Chapter

More information

Linear Dependence Tests

Linear Dependence Tests Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

More information

A note on companion matrices

A note on companion matrices Linear Algebra and its Applications 372 (2003) 325 33 www.elsevier.com/locate/laa A note on companion matrices Miroslav Fiedler Academy of Sciences of the Czech Republic Institute of Computer Science Pod

More information

The Characteristic Polynomial

The Characteristic Polynomial Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem

More information

The Matrix Elements of a 3 3 Orthogonal Matrix Revisited

The Matrix Elements of a 3 3 Orthogonal Matrix Revisited Physics 116A Winter 2011 The Matrix Elements of a 3 3 Orthogonal Matrix Revisited 1. Introduction In a class handout entitled, Three-Dimensional Proper and Improper Rotation Matrices, I provided a derivation

More information