The Singular Value Decomposition and Least Squares Problems

Transcription

1 The Singular Value Decomposition and Least Squares Problems Tom Lyche University of Oslo Norway The Singular Value Decomposition and Least Squares Problems p. 1/2

2 Applications of SVD 1. solving over-determined equations 2. statistics, principal component analysis 3. numerical determination of the rank of a matrix 4. search engines (Google,...) 5. theory of matrices 6. and lots of other applications... The Singular Value Decomposition and Least Squares Problems p. 2/2

3 Singular Value Decomposition 1. Works for any matrix A C m,n 2. A = UΣV H with U, V unitary and Σ = [ Σ C m,n 3. Σ 1 = diag(σ 1,...,σ r ) with σ 1 σ 2 σ r > 0, 4. r is the rank of A. 5. We define σ r+1 = σ n = 0 if r < n and call σ 1,...,σ n the singular values of A. 6. The columns u 1,...,u m of U and v 1,...,v n of V are called left- and right singular vectors respectively. The Singular Value Decomposition and Least Squares Problems p. 3/2

4 Relation to eigenpairs for A T A and AA 1. A T Av i = σ 2 i v i for i = 1,...,n. 2. The columns of V are orthonormal eigenvectors of A T A 3. The columns of U are orthonormal eigenvectors of AA T The Singular Value Decomposition and Least Squares Problems p. 4/2

5 Three forms of SVD Suppose A R m,n, A = UΣV T is the SVD of A R m,n and let r := #Σ 1. We partition U and V as follows U = [U 1, U 2, U 1 R m,r, U 2 R m,m r V = [V 1, V 2, V 1 R n,r, V 2 R n,n r. A = [ U 1,U 2 [ [ Σ 1 0 V T = U 1 Σ 1 V T 1 The three forms V T 2 1. A = UΣV T full form 2. A = U 1 Σ 1 V T 1 compact form 3. A = r i=1 σ iu i v T i = min(m,n) i=1 σ i u i v T i outer product form The Singular Value Decomposition and Least Squares Problems p. 5/2

6 Subspaces of A column space and the null space of a matrix span(a) := {y R m : y = Ax, for some x R n }, ker(a) := {x R n : Ax = 0}. span(a) is a subspace of R m. ker(a) is a subspace of R n. We say that A is a basis for a subspace S of R m if 1. S = span(a), 2. A has linearly independent columns, i. e., ker(a) = {0}. Recall the four fundamental subspaces span(a), span(a T ), ker(a), ker(a T ). The Singular Value Decomposition and Least Squares Problems p. 6/2

7 The 4 fundamental Subspaces Let A = UΣV T be the SVD of A R m,n. Then A T = V Σ T U T and AV = UΣ, A T U = V Σ T or A [ V 1,V 2 = [ U 1,U 2 [ Σ , A T [ U 1,U 2 = [ V 1,V 2 [ Σ AV 1 = U 1 Σ 1, U 1 is an orthonormal basis for span(a) A T U 2 = 0, U 2 is an orthonormal basis for ker(a T ) A T U 1 = V 1 Σ 1, V 1 is an orthonormal basis for span(a T ) AV 2 = 0, V 2 is an orthonormal basis for ker(a). We obtain the fundamental relations 1. dim(span(a)) + dim(ker(a)) = #A :=number of columns of A, 2. dim(span(a T )) = dim(span(a)) =: rank(a) = #Σ 1. The Singular Value Decomposition and Least Squares Problems p. 7/2

8 Existence of SVD Theorem 1. Every matrix has an SVD. The Singular Value Decomposition and Least Squares Problems p. 8/2

9 Uniqueness If the SVD of A is A = UΣV T then A T A = V Σ T ΣV T. Thus σ 2 1,...σ2 n are uniquely given as the eigenvalues of A T A arranged in descending order. Taking the positive square root uniquely determines the singular values. From the proof of the existence theorem it follows that the orthogonal matrices U and V are in general not uniquely given. The Singular Value Decomposition and Least Squares Problems p. 9/2

10 Application I, rank Gauss-Jordan cannot be used to determine rank numerically Use singular value decomposition numerically will normally find σ n > 0. Determine minimal r so that σ r+1,...,σ n are "close" to round off unit. The Singular Value Decomposition and Least Squares Problems p. 10/2

11 Application II, overdetermined Equatio Given A m,n and b R m. The system Ax = b is over-determined if m > n. This system has a solution if b span(a), the column space of A, but normally this is not the case and we can only find an approximate solution. A general approach is to choose a vector norm and find x which minimizes Ax b. We will only consider the Euclidian norm here. The Singular Value Decomposition and Least Squares Problems p. 11/2

12 The Least Squares Problem Given A m,n and b R m with m n 1. The problem to find x R n that minimizes Ax b 2 is called the least squares problem. A minimizing vector x is called a least squares solution of Ax = b. Several ways to analyze: Quadratic minimization Orthogonal Projections SVD The Singular Value Decomposition and Least Squares Problems p. 12/2

13 Quadratic minimization Define function E : R n R by E(x) = Ax b 2 2 E(x) = (Ax b) T (Ax b) = x T Bx 2c T x + α, where B := A T A, c := A T b and α := b T b. B is positive semidefinite and positive definite if A has rank n. Since the Hessian HE(x) := ( 2 E(x) x i x j ) = 2B we can find minimum by setting partial derivatives equal zero. E(x) := ( E(x) x i ) = 2(Bx c) = 0 Normal equations A T Ax = A T b. The Singular Value Decomposition and Least Squares Problems p. 13/2

14 A simple example x 1 = 1 x 1 = 1, A = x 1 = 2 1 1, x = [x 1, b = 1 1 1, 2 Quadratic minimization problem: Ax b 2 2 = (x 1 1) 2 + (x 1 1) 2 + (x 1 2) 2. Setting the first derivative with respect to x 1 equal to zero we obtain 2(x 1 1) + 2(x 1 1) + 2(x 1 2) = 0 or 6x 1 8 = 0 or x 1 = 4/3 The second derivative is positive (it is equal to 6) and x = 4/3 is a global minimum. The Singular Value Decomposition and Least Squares Problems p. 14/2

15 Theory; Direct sum and Orthogonal Su Suppose S and T are subspaces of a vector space (V, F). We define 1. Sum: X := S + T := {s + t : s S and t T }; 2. Direct Sum: If S T = {0}, then S T := S + T. 3. Orthogonal Sum: Suppose (V, F,, ) is an inner product space. Then S T is an orthogonal sum if s, t = 0 for all s S and all t T. 4. orthogonal complement: T = S := {x X : s, x = 0 for all s S}. The Singular Value Decomposition and Least Squares Problems p. 15/2

16 Basic facts Lemma 1. Suppose S and T are subspaces of a vector space (V, F). 1. S + T = T + S and S + T is a subspace of V. 2. dim(s + T ) = dim S + dim T dim(s T ) 3. dim(s T ) = dim S +dim T. Every v S T can be decomposed uniquely as v = s + t, where s S and t T. s is called the projection of v into S. 4. Pythagoras: If s, t = 0 then s + t 2 = s 2 + t Here v := v, v. v t S s The Singular Value Decomposition and Least Squares Problems p. 16/2

17 Column space of A and null space of A R m = span(a) ker(a T ) and this is an orthogonal sum. Thus ker(a T ) = span(a) the orthogonal complement of span(a). Example A = [ , span(a) = span(e 1, e 2 ), ker(a T ) = e The Singular Value Decomposition and Least Squares Problems p. 17/2

18 Proof that R m = span(a) ker(a T ) using SVD s T t = 0 for all s span(a) and t ker(a T ). For if s span(a) and t ker(a T ) then s = Ax for some x R n and A T t = 0. But then s, t = (Ax) T t = x T (A T t) = 0 Suppose A = UΣV T = U 1 Σ 1 V T 1 is the SVD of A. Then I = UU T = [ U 1 U 2 [ U 1 U 2 = U1 U T 1 + U 2 U T 2. For any b R m we have b = (U 1 U T 1 + U 2 U T 2 )b = b 1 + b 2, where b 1 := U 1 U T 1 b = AA b with A = V 1 Σ 1 1 UT 1, and b 2 := U 2 U T 2 belongs to ker(a T ) since A T b 2 = (V 1 Σ 1 U T 1 )U 2 U T 2 b = V 1 Σ 1 (U T 1 U 2 )U T 2 b = 0. The Singular Value Decomposition and Least Squares Problems p. 18/2

19 Projections and pseudoinverse b 1 := AA b is the projection of b into span(a). The matrix A := V 1 Σ 1 1 U T 1 = V Σ U T R n,m is called the pseudoinverse of A = UΣV T R m,n. Σ := [ Σ R n,m is the pseudoinverse of Σ. b 2 := (I AA )b is the projection of b into ker(a T ). Example b = A = [ b1 b 2 b 3 [ = UΣV T = I 3 AI 2, A = I 2 [ I 3 = [ , b 1 = AA b = b 2 = (I 3 AA )b = [ b = [ b = [ 00 b 3 [ b1 b 20 The Singular Value Decomposition and Least Squares Problems p. 19/2

20 LSQ; Existence and Uniqueness Theorem 2. The least squares problem always has a solution. The solution is unique if and only if A has linearly independent columns. Proof. Let b = b 1 + b 2, where b 1 span(a) is the (orthogonal) projection of b into span(a) and b 2 ker(a T ). Since b 1 span(a) there is an x R n such that Ax = b 1. Thus b 2 = b Ax. By Pythagoras, for any s span(a) with s b 1 b s 2 = b 1 s 2 + b 2 2 = b 1 s 2 + b Ax 2 > b Ax 2. Since the projection b 1 is unique, the least squares solution x is unique if and only if A has linearly independent columns. The Singular Value Decomposition and Least Squares Problems p. 20/2

21 The Normal Equations Theorem 3. Any solution x of the least squares problem is a solution of the linear system A T Ax = A T b. The system is nonsingular if and only if A has linearly independent columns. Proof. Since b Ax ker(a T ), we have A T (b Ax) = 0 or A T Ax = A T b. A T A is nonsingular. Suppose A T Ax = 0 for some x R n. Then 0 = x T A T Ax = (Ax) T Ax = Ax 2 2. Hence Ax = 0 which implies that x = 0 if and only if A has linearly independent columns. The linear system A T Ax = A T b is called the normal equations. The Singular Value Decomposition and Least Squares Problems p. 21/2

22 Linear Regression A = 1 t 1 1 t t m b = [ y1 y 2. y m, min x 1 x 2 m (x 1 + t i x 2 y i ) 2. i=1 7 A = [ b = [ A T A = [ c = AT b = [ x 1 +15x 2 = x 1 +55x 2 = x 1 = x 2 =1.0451, x = A\b The Singular Value Decomposition and Least Squares Problems p. 22/2

23 Analysis of LSQ using A = UΣV T Define y := V T x = [ V T 1 x V T 2 x = [ y 1 y 2. Recall Uv 2 = v 2 for any U R n,n with U T U = I and any v R n. = [ U T 1 b Σ 1 y 1 U T 2 b [ U T 1 b U T 2 b b Ax 2 2 = UU T b UΣy 2 2 = U T b Σy 2 2 = 2 2 = U T 1 b Σ 1 y U T 2 b 2 2. We have b Ax 2 U T 2 b 2 for all x R n with equality if and only if [ x = V y = [ V 1 V 2 Σ 1 1 U T 1 b y 2 [ [ Σ 1 0 y1 0 0 y 2 = V 1 Σ 1 1 UT 1 b+v 2 y 2, for all y 2 R n r. (1) The Singular Value Decomposition and Least Squares Problems p. 23/2

24 The general solution of min Ax b 2 The columns of V 2 is a basis for ker(a) so that ker(a) = {z = V 2 y 2 : y 2 R n r }. Therefore the solution set is {x R n : Ax b 2 is minimized } = A b + ker(a). If r = n then A has linearly independent columns and A T A is nonsingular. Since A T Ax = A T b we obtain A = (A T A) 1 A T in this case. The Singular Value Decomposition and Least Squares Problems p. 24/2

25 The Minimal Norm Solution Suppose A is rank deficient (r < n). Let x = A b + V 2 y 2 be a solution of min Ax b 2. A b and V 2 y 2 are orthogonal By Pythagoras x 2 2 = A b V 2y A b 2 2. The solution x = A b is called the minimal norm solution to the LSQ problem. Orthogonal. Since V T 2 A = (V T 2 V 1)Σ 1 1 UT 1 = 0 we have (V 2 y 2 ) T A b = 0 for any y 2. The Singular Value Decomposition and Least Squares Problems p. 25/2

26 More on the pseudoinverse If A is square and nonsingular then A = A 1. A is always defined. Thus A is a generalization of usual inverse. If B R n,m satisfies 1. ABA = A 2. BAB = B 3. (BA) T = BA 4. (AB) T = AB then B = A. Thus A is uniquely defined by these axioms. The Singular Value Decomposition and Least Squares Problems p. 26/2

27 Example Show that the pseudoinverse of A = We have BA = 1 2 [ and AB = ABA = A 2. BAB = B 3. (BA) T = BA 4. (AB) T = AB and hence A = B. [ [ is B = 1 4 [ Thus The Singular Value Decomposition and Least Squares Problems p. 27/2