Chapter 4 Least-Mean-Square Algorithm ( LMS Algorithm )

Transcription

1 Chapter 4 Least-Mean-Square Algorthm ( LMS Algorthm ) 1. Search Methods he optmum tap-weghts of a transversal (FIR) Wener flter can be obtaned by solvng the Wener-Hopf equaton provded that the requred statstcs of the underlyng sgnals are avalable. An alternatve way of fndng the optmum tap-weghts s to use an teratve search algorthm that starts at some arbtrary ntal pont n the tap-weght vector space and progressvely moves towards the optmum pont n steps. here are many teratve search algorthms derved for mnmzng the underlyng cost functon wth the true statstcs replaced by ther estmate obtaned n some manner. Gradent-based teratve methods (1) Method of Steepest Descent (2) Newton s Method

2 2. Method of Steepest Descent 2.1 Consder a transversal Wener flter shown n Fg. 1. z -1 z -1 z -1 w w 1 w N-1 y( e( - + d( Fg. 1. Assumng that all the sgnals nvolved are real-valued sgnals. ap-weght vector sgnal nput w = ] [ w w1... w N 1 x ( = [ n 1) L n N + 1) ] flter output y( = w x ( error sgnal e( = d( y( performance functon ξ = E[ e 2 = E[( d ( ] 2 ( ] 2W p + W RW...(1) where R = E[ x ( ] autocorrelaton matrx of the flter nput and p = E[ d( ] cross-correlaton vector between x ( and d (

3 he performance functon ξ s a quadratc functon of the flter tap-weght vector W. ξ has a sngle global mnmum obtaned by solvng the Wener-Hopf equaton R W = p...(2) f R and p are avalable. Instead of tryng to solve equaton (2) drectly, we employ an teratve search method n whch startng wth an ntal guess for W, say () W, a recursve search method that may requre many teratons (or steps) to converge to W s used. he gradent of ξ s gven by ξ = 2RW 2 p...(3) Wth an ntal guess of W at n= the tap-weght vector at the k-th teraton s denoted as W (k). he followng recursve equaton may be used to update W (k) : µ k W ( k + 1) = W ( k) ξ...(4) where µ > s called the step-sze, kξ denotes the gradent vector ξ evaluated at the pont W = W (k) Substtutng (3) n (4), we get W ( k + 1) = W(k)-2µ(RW(k)-p(k))...(5)

4 he convergence of W (k) to the optmum soluton W and the convergence speed are dependent on the step-sze parameter µ. 2.2 Convergence of the steepest descent method Equaton(5) can be re-arranged as W ( k + 1) = (1 2µ R)( W ( k) W )...(6) Defnng the vector v (k) as v( k) = W ( k) W...(7) hen, we have, from eq.(6), v ( k + 1) = ( I 2µ R) v( k)...(8) he auto-correlaton matrx R may be dagonalzed by usng a untary smlarty decomposton : R = QΛQ...(9) where Λ s a dagonal matrx consstng of the egenvalues λ, λ1,..., 1 of R and the columns of Q contan the correspondng orthonormal egenvectors. Note that I = QQ...(1) Combnng eq.(8),(9)&(1), we get v( k + 1) = ( QQ 2µ QΛQ ) v( k) = Q( I 2µ Λ) Q v( k)...(11) λ N Denote that v' ( k) = Q v( k)...(12) Whch transforms (k) v to v ' ( k). Wth some mathematcal manpulatons, we obtan v' ( k + 1) = ( I 2µ Λ) v' ( k)...(13) hs vector recursve equaton may be separated nto scalar recursve equatons :

5 v ' ( k + 1) = (1 2µλ ) v ' ( k) for =,1,2,..., N 1...(14) where v' ( k) = [ v' ( k) v1' ( k) L v 1' ( k)] From eq.(14), we get v '( k) = (1 2 µλ for =,1,2,..., N 1 k ) v '()...(15) Eq.(15) mples that v ' ( k) can converges to zero f and only f the step-sze parameter µ s selected so that N 1 2µλ < 1 for =,1,2,..., N 1...(16) hat s, 1 < µ < λ for all or, equvalently, < µ <...(17) 1 λ max where λ max s the maxmum of the egenvalues λ1 λ N 1 λ L.

6 3. Newton s method he steepest descent algorthm may suffer from slow modes of convergence whch arse as a result of the spread n the egenvalues of R. he Newton s method can somehow get rd of the egenvalue spread. Startng from the steepest descent algorthm gven n eq.(5) : W ( k + 1) = W ( k) 2µ ( RW ( k) p( k)) Usng p = RW, eq.(18) becomes...(18) W ( k + 1) = W ( k) 2µ R( W ( k) W )...(19) he presence of R n eq.(19) cause the egenvalue-spread problem n the steepest descent algorthm. Newton s method overcomes ths problem by replacng the scalar step-sze µ wth a matrx step-sze gven by µr -1. he resultng algorthm s 1 W ( k + 1) = W ( k) µ R kξ...(2) Substtutng ξ = 2RW 2 p n eq.(2), we obtan W ( k + 1) = W ( k) 2µ R 1 = (1 2µ ) W ( k) + 2µ R ( RW ( k) p) = (1 2µ ) W ( K) + 2µ W -1 p...(21) And, by subtractng W from both sdes of eq.(21), we get W ( k + 1) W = (1 2µ )( W ( k) W )...(22) Startng wth an ntal value W () and teratng eq.(22), we obtan k W ( k) W = (1 2µ ) ( W () W )...(23) In actual mplementaton of adaptve flters, the exact values of k ξ and 1 R are not avalable and have to be estmated.

7 4. Leas-Mean-Square (LMS) Algorthm Fg. 2 shows an N-tap transversal adaptve flter. z -1 n-1) z -1 z -1 w ( w 1 ( w N-1 ( y( e( - + d( Fg. 2. N y( = w ( n ) e( = d( y( 1 =...(24)...(25) We assume that the sgnals nvolved are real-valued. he LMS algorthm changes (adapts) the flter tap weghts so that e( s mnmzed n the mean-square sense. When the processes & d( are jontly statonary, ths algorthm converges to a set of tap-weghts whch, on average, are equal to the Wener-Hopf soluton. he LMS algorthm s a practcal scheme for realzng Wener flters, wthout explctly solvng the Wener-Hopf equaton. he conventonal LMS algorthm s a stochastc mplementaton of the steepest descent algorthm. It smply replaces the cost functon ξ = E[ e 2 ( ] by ts nstantaneous coarse estmate ˆ ξ = e 2 (.

8 Substtutng ˆ ξ = e 2 ( for ξ n the steepest descent recurson, we obtan W ( n + 1) = W ( µ e 2 (...(26) where W = [ w ( w ( L w N ( ] ( 1 1 = [ L w w1 w N 1 ] Note that the -th element of the gradent vector e 2 ( s 2 e ( e( = 2e( w w y( = 2e( w = 2e( n )...(27) hen e 2 ( = 2e( where x ( = [ n 1) L n N + 1) ] Fnally, we obtan W ( n + 1) = W ( + 2µ e(...(28) Eq.(28) s referred to as the LMS recurson. Summary of the LMS algorthm Input : tap-weght vector, W ( Output : nput vector, x ( desred output, d( Flter output, y( ap-weght vector update, W ( n +1)

9 1. Flterng : y( = W ( 2. Error estmaton : e(=d(-y( 3. ap-weght vector adaptaton : W ( n + 1) = W ( + 2µ e( Advantages & dsadvantages of LMS algorthm : (1) Smplcty n mplementaton (2) Stable and robust performance aganst dfferent sgnal condtons (3) slow convergence ( due to egenvalue spread )

10 5. MSE Behavor of the LMS Algorthm We are to study