COM Math. Methods for SP I Lecture 4: Singular Value Decomposition &

Transcription

1 COM Math. Methods for SP I Lecture 4: Singular Value Decomposition & Orthogonal Projection Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 1 Singular Value Decomposition (SVD) Theorem 4.1 Every A C m n can be decomposed as A = UΣV H where U C m m and V C n n are unitary, and Σ = Diag(σ 1,..., σ p ) R m n, p = min(m, n), where σ 1 σ 2... σ p 0. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 2

2 The values σ i are called the singular values of A. The columns u i & v i of U & V are called the left and right singular vectors of A. Outer product representation of SVD: A = p σ i u i vi H i=1 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 3 Relationship with the 2-norm: Recall A 2 = λ max, where λ max is the max. eigenvalue of A H A. By SVD A = UΣV H, A H A = VΣ 2 V H. It follows that the eigenvalues of A H A are σi 2, and that the eigenvector matrix of A H A is V. Thus, A 2 = σ 1 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 4

3 Relationship with eigendecomposition: Consider a Hermitian A C n n. Eigendcomposition: A = QΛQ H AQ = QΛ Aq i = λ i q i, i = 1,..., n SVD: A = UΣV H AV = UΣ Av i = σ i u i, i = 1,..., n Hence, for Hermitian A we have U = V = Q & Λ = Σ. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 5 Partitioning the SVD Suppose that the number of nonzero singular values is r p; i.e., σ r+1 = σ r+2 =...σ p = 0. The SVD can be rewritten as ] 0 A = [U 1 U 2 Σ VH V2 H where Σ = Diag(σ 1,..., σ r ) R r r, U 1 C m r, U 2 C m m r, V 1 C n r, and V 2 C n m r. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 6

4 Property 4.1 rank(a) = r. Property 4.2 N(A) = R(V 2 ). Property 4.3 R(A) = R(U 1 ). Property 4.4 R(A H ) = R(V 1 ). Property 4.5 R (A) = R(U 2 ). Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 7 Inverse Consider a square, nonsingular A. A 1 = VΣ 1 U H An alternate form of the inverse: p A 1 1 = v i u H i σ i i=1 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 8

5 Linear System of Equations Given A C m n, b C n, the problem of the linear system of eqns. is find an x C m (or multiple x s) such that Ax = b We have learnt that for m = n, Ax = b is always satisfied if A is nonsingular. Can Ax = b be satisfied when m n, and/or when A is rank deficient? Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 9 Ax = b UΣV H x = b Σd = c where d = V H x = VH 1 x = d 1 V2 H x d 2 c = U H b = UH 1 x U H 2 b = c 1 c 2 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 10

6 Case A: m > n, and r = n. In this case V = V 1, d 1 = d, & Σd = c Σd = c 1 0 Ax = b can only be satisfied if b R (U 2 ) = R(U 1 ). c 2 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 11 Case B: m > n, and r = n. In this case U = U 1, c 1 = c, & Σd = c [ ] Σ 0 d 1 = d 2 [ ] c Σd 1 = c Ax = b can always be satisfied, but x is not unique. If x o is a solution to Ax = b, then x o + V 2 c 2, for any c 2 C n r is also a solution. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 12

7 Case C: r < min(m, n). Σd = c 0 Σ d 1 = c d 2 Ax = b can only be satisfied if b R(U 1 ). If x o is a solution to Ax = b, then x o + V 2 c 2, for any c 2 C n r is also a solution. c 2 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 13 Low Rank Approximation Theorem 4.2 Let UΣV H be the SVD of A. For k < r = rank(a), the solution to the problem is min A B 2 B C m n, rank(b)=k A k = k σ i u i vi H. i=1 Moreover, the minimal objective function value is A A k 2 = σ k+1 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 14

8 Theorem 4.3 Let UΣV H be the SVD of A. For k < r = rank(a), the solution to the problem min A B 2 B C m n F, rank(b)=k is A k = k σ i u i vi H. i=1 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 15 Recall the KL transform in Lecture 3. The vector ˆx n, formed from truncating N r KL coefficients, has the covariance matrix given by Rˆx = VDiag(λ 1,..., λ r, 0,..., 0)V H From Theorems 4.2 & 4.3 we know that Rˆx is the closest rank-r matrix to the true signal covariance matrix R x, in the 2-norm and Frobenius-norm senses. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 16

9 Orthogonal Projection The idea: An arbitrary vector y can be expressed as where y s S, & y c S. y = y s + y c We are interested in obtaining a matrix P, called the orthogonal projection, such that Py = y s Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 17 Application: noise reduction Consider a received signal that consists of a signal vector s S and noise w: y = s + w We don t know s, but we do know S. We can enhance the signal by performing a projection Py = s + w s where w s = Pw is a residual noise vector. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 18

10 A matrix P C n n is an orthogonal projection onto S if 1. R(P) = S, 2. P 2 = P, and 3. P H = P. Note that a matrix having the property P 2 = P is called an idempotent matrix. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 19 We have learnt that for a subspace S with a dimension m, there is a full rank matrix X C n m, such that S = R(X). An orthogonal projection onto S = R(X) is P = X(X H X) 1 X H ( ) Exercise: Verify that ( ) satisfies the 3 properties for an orthogonal projection matrix. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 20

11 Theorem 4.4 The orthogonal projection matrix in ( ) is unique (i.e., there does not exist P 1 such that P 1 is an orthogonal projection onto S and P 1 P). Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 21 The orthogonal complement projection: By observing that we obtain y = y s + y c = Py + y c, y c = (I P)y and that (I P) is the orthogonal projection onto the orthogonal complement subspace S. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 22

12 Property 4.6 V 1 V H 1 is the orthogonal projection onto R(AH ). V 2 V H 2 is the orthogonal projection onto N(A). U 1 U H 1 is the orthogonal projection onto R(A). U 2 U H 2 is the orthogonal projection onto R (A). Property 4.7 The eigenvalues of a projection matrix is either 1 or 0. The number of nonzero eigenvalues is the dimension of the associated subspace. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 23 Distance between subspaces: Let S 1 & S 2 be two subspaces with dim S 1 = dim S 2. Let P 1 & P 2 be the orthogonal projection matrices of S 1 & S 2, respectively. The distance between S 1 & S 2 is defined as dist(s 1, S 2 ) = P 1 P 2 2 = max x 2 =1 P 1x P 2 x 2 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 24

13 Theorem 4.5 Suppose W = [ W 1 W 2 ], Z = [ Z 1 Z 2 ] are unitary, where W 1,Z 1 C n k. If S 1 = R(W 1 ) & S 2 = R(Z 1 ), then dist(s 1, S 2 ) = W H 1 Z 2 2 = Z H 1 W 2 2 Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 25 Property dist(s 1, S 2 ) 1. Property 4.9 If S 1 = S 2, then dist(s 1, S 2 ) = 0. Property 4.10 If S 1 S 2 {0}, then dist(s 1, S 2 ) = 1. Institute Comm. Eng. & Dept. Elect. Eng., National Tsing Hua University 26