Lecture 11 Fast Fourier Transform (FFT)

Lecture 11 Fast Fourier Transform (FFT) Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn No.1 Science Building, 1575

Outline Examples Fast Fourier Transform Applications

Signal processing Filtering: a polluted signal 1.5 1 0.5 0 0.5 1 1.5 0 200 400 600 800 1000 1200 High pass and low pass filter (signal and noise) 1.5 1 0.5 0 0.5 1 1.5 0 200 400 600 800 1000 1200 How to obtain the high frequency and low frequency quickly?

Solving PDEs on rectangular mesh Solving the Poisson equations u = f in Ω u = 0 on Ω in the rectangular domain After discretization we will obtain the linear system with about N 2 unknowns ui+1,j + ui 1,j + ui,j+1 + ui,j 1 4ui,j 4h 2 = f ij The FFT would give a fast algorithm to solve the system above with computational efforts O(N 2 log 2 N).

Computing convolution ( ) Suppose h(x) = 2π 0 f(x y)g(y)dy is the convolution of f and g, where f(x), g(x) C 2π are period 2π functions. Take x j = jδ, j = 0, 1,..., N 1, δ = 2π N discretization h(x i) N 1 j=0 and apply simple rectangular f(x i x j)g(x j) δ i = 0, 1,..., N 1 Define f i = f(x i), g i = g(x i), and let f i is period N respect to the subscript i, define h i = N 1 j=0 The direct computation is O(N 2 ). f i jg j δ i = 0, 1,..., N 1

Fast Fourier Transform Fast Fourier Transform is one of the top 10 algorithms in 20th century. But its idea is quite simple, even for a high school student!

Fourier Transform Suppose f(x) is absolutely integrable in (, + ), then the Fourier transform of f(x) is ˆf(k) = + f(x)e ikx dx. Moreover if f(x) is square integrable, then the inverse Fourier transform of ˆf(k) is f(x) = 1 + ˆf(k)e ikx dk. 2π

Properties of Fourier transform Some important properties of Fourier transform: 1. Derivative to coefficient: 2. Translation property: 3. Convolution to multiplication: (f (x))(k) = ik ˆf(k); (f(x a))(k) = e ika ˆf(k); (f g)(k) = ˆf(k)ĝ(k); where (f g)(x) = + f(x y)g(y)dy. 4. Parseval s identity: + f(x) 2 dx = 1 2π + ˆf(k) 2 dk.

Discrete Fourier transform (DFT) Suppose we have a = (a 0, a 1,, a N 1) T, define DFT of a as c = (c 0, c 1,, c N 1) T = â, where c k = N 1 j=0 jk 2πi a je N, k = 0, 1,..., N 1. Here i is the imaginary unit, e 2πi N = ω is the N-th root of unity. a is the inverse discrete Fourier transform of c defined as a j = 1 N N 1 k=0 jk 2πi c k e N, j = 0, 1,..., N 1. DFT is closely related to the trigonometric interpolation for 2π-periodic function T (x) = N 2 k= N 2 +1 b k e ikx. such that at x j = 2jπ, T (xj) = aj, j = 0, 1,..., N 1. The readers N may find the relation between c k and b k.

Remark on DFT DFT can be considered as a linear transformation. Define Fourier matrix 1 1 1 F = 1 ω ω N 1 = (ωjk ) N 1 1 ω N 1 ω (N 1)2 j,k=0 where ω is the N-th root of unity. c is the Fourier transform of a can be represented as c = F a

Remark on DFT Inverse DFT can also be considered as a linear transformation. Define inverse Fourier matrix 1 1 1 F 1 = G = 1 1 ω 1 ω (N 1) N = (ω jk ) N 1 1 ω (N 1) ω (N 1)2 j,k=0 where ω is the N-th root of unity. a is the inverse Fourier transform of c can be represented as a = Gc

Properties of DFT Convolution to multiplication: where (f g) k = ˆf k ĝ k k = 0, 1,..., N 1 (f g) l = N 1 j=0 f l j g j l = 0, 1,..., N 1, and f l is period N with respect to index l, i.e. f 1 = f N 1, f 2 = f N 2,... Parseval s identity: N N 1 j=0 a j 2 = N 1 k=0 c k 2

FFT idea FFT is proposed by J.W. Cooley and J.W. Tukey in 1960s, but the idea may be traced back to Gauss. The basic motivation is if we compute DFT directly, i.e. c = F a we need N 2 multiplications and N(N 1) additions. Is it possible to reduce the computation effort? First consider the case N = 4 1 1 1 1 (a 0 + a 2) + (a 1 + a 3) F = 1 i 1 i 1 1 1 1, F a = (a 0 a 2) i(a 1 a 3) (a 0 + a 2) (a 1 + a 3) 1 i 1 i (a 0 a 2) + i(a 1 a 3)

FFT idea From the concrete form of DFT, we actually need 2 multiplications (timing ±i) and 8 additions (a 0 + a 2, a 1 + a 3, a 0 a 2, a 1 a 3 and the additions in the middle). This observation may reduce the computational effort from O(N 2 ) into O(N log 2 N) Because It is a typical fast algorithm. log lim 2 N N N = 0 Fast algorithms of this type of recursive halving are very typical in scientific computing.

Construction of FFT Consider N = 2 m and denote p(x) = a 0 + a 1x + + a N 1x N 1, divide p(x) into odd ( ) and even ( ) power parts p(x) = (a 0 + a 2x 2 + ) + x(a 1 + a 3x 2 + ) = p e(x 2 ) + xp o(x 2 ) where p e(t) = a 0 + a 2t +... + a N 2t N 2 1, p o(t) = a 1 + a 3t +... + a N 1t N 2 1 Define ω k = e 2πi k (k-th root of unity), then when j = 0, 1,..., N 2 1 c j c N 2 +j = p e(ω 2j N ) + ωj N po(ω2j N ) = p e(ω 2( N 2 +j) N ) + ω N 2 +j N p o(ω 2( N 2 +j) N )

Construction of FFT Pay attention that ω 2j N = ωj N 2, ω N 2 +j N = ω j N, ωn+2j N = ω j N 2 then c j = v j + ω j N uj, c j+ N 2 = v j ω j N uj j = 0, 1,..., N 2 1 where v j = p e(ω j N 2 ), u j = p o(ω j N 2 ) The formula above show that the DFT of N components vector a could be converted to compute the DFT of two N components vectors ae, ao 2 and some simple additions and multiplications. This is called Danielson-Lanczos algorithm. The recursive application of this idea gives FFT.

A simple example: N = 8 Suppose the array a = (a 0, a 1,, a 7) T Step A: Splitting (reordering) (odd parts and even parts): Step 1 Step 2 Step 3 a e = (a 0, a 2, a 4, a 6 ) T, a o = (a 1, a 3, a 5, a 7 ) T ; a ee = (a 0, a 4 ) T, a eo = (a 2, a 6 ) T, a oe = (a 1, a 5 ) T, a oo = (a 3, a 7 ) T ; a eee a eeo a eoe a eoo a oee a oeo a ooe a ooo a 0 a 4 a 2 a 6 a 1 a 5 a 3 a 7

A simple example: N = 8 Step B: Combination: Step 1 c ee = (a 0 + ω2a 0 4, a 0 ω2a 0 4) T, c eo = (a 2 + ω2a 0 6, a 2 ω2a 0 6) T, c oe = (a 1 + ω2a 0 5, a 1 ω2a 0 5) T, c oo = (a 3 + ω2a 0 7, a 3 ω2a 0 7) T, Define the notations w 4 (w 0 4, w 1 4) T, w 8 (w 0 8, w 1 8, w 2 8, w 3 8) T, and X Y (x jy j) j as the vector product through multiplication by components.

A simple example: N = 8 Step B: Combination: Step 2 Step 3 [ ] [ ] cee + w 4 c eo coe + w 4 c oo c e =, c o =, c ee w 4 c eo c oe w 4 c oo [ ] ce + w 8 c 0 c = c e w 8 c 0

A simple sketch of FFT (N = 8) a reordering a 0 a 4 a 2 a 6 a 1 a 5 a 3 a 7 combination 1 c ee c eo c eo c oo combination 2 [ ce c o ] combination 3 c

A remark on the reordering If we map e to 0, and o to 1 we can find the binary representation of the indices after reordering is just the bit reversal before reordering 0= 000 2 1= 001 2 2= 010 2 3 = 011 2 4 = 100 2 5 = 101 2 6 = 110 2 7 = 111 2 Bit reversal 000 2 = 0 100 2 = 4 010 2 = 2 110 2 = 6 001 2 = 1 101 2= 5 011 2= 3 111 2= 7

Compute the convolution From the discretization at the beginning, we have h i = N 1 j=0 f i jg j δ i = 0, 1,..., N 1 thus h = (ĥ) = (δ ˆf ĝ) After using FFT, N 2 + N multiplications and N(N 1) additions are reduced to 3 2 N log 2 N + 2N multiplications and 3N log 2 N additions.

Solving the linear system with loop matrix Let L = c 0 c N 1 c 1 c 1 c 0 c 2 Solving Lx = b. L is a loop matrix. We have c N 1 c N 2 c 0 (Lx) i = N 1 j=0 c i jx j where we assume c is period N with respect to the subscripts, and x = (x 0, x 1,..., x N 1) T.

Solving the linear system with loop matrix First consider the Jordan form of L. From the formula before Lx = c x = λx Take DFT we have then eigenvalues ĉ ˆx = λˆx The eigenvectors where δ kj is Kronecker s δ. Take inverse transform we obtain λ k = ĉ k ˆx (k) j = δ kj, (j, k = 0, 1,..., N 1) x (0) = (1, 1,..., 1) T, x (1) = (1, ω 1,..., ω (N 1) ) T,......... x (N 1) = (1, ω (N 1),..., ω (N 1)2 ) T

Solving the linear system with loop matrix Spectral decomposition of L L = (x (0) x (1) x (N 1) ) λ 0 λ 1... (x (0) x (1) x (N 1) ) 1 = (NF 1 )Λ (NF 1 ) 1 = F 1 ΛF λ N 1 Solving Lx = b is equivalent to F 1 ΛF x = b, i.e. Λ(F x) = F b. Then it is composed of three steps: Step 1: Compute F b i.e. apply FFT to b to obtain ˆb; Step 2: Compute Λ i.e. apply FFT to c to obtain ĉ; Step 3: Compute ˆx k = ˆb k /ĉ k, and then compute (ˆx) to obtainx.

Homework assignment Familiarize the FFT and IFFT command in MATLAB; Compute the convolution for h(x) = 2π 0 sin(x y)e cos y dy