18.310 lecture otes November 18, 2013 Fast Fourier Trasform Lecturer: Michel Goemas I these otes we defie the Discrete Fourier Trasform, ad give a method for computig it fast: the Fast Fourier Trasform. We the use this techology to get a algorithms for multiplyig big itegers fast. Before goig ito the core of the material we review some motivatio comig from the classical theory of Fourier series. 1 Motivatio: Fourier Series I this sectio we discuss the theory of Fourier Series for fuctios of a real variable. I the ext sectios we will study a aalogue which is the discrete Fourier Trasform. Early i the Nieteeth cetury, Fourier studied soud ad oscillatory motio ad coceived of the idea of represetig periodic fuctios by their coefficiets i a expasio as a sum of sies ad cosies rather tha their values. He oticed, for example, that you ca represet the shape of a vibratig strig of legth L, fixed at its eds, as y(x) = a k si(πkx/l). k=1 (Observe that ideed y(0) = y(l) = 0.) The coefficiets, a k, cotai importat ad useful iformatio about the quality of the soud that the strig produces, that is ot easily accessible from the ordiary y = f(x) descriptio of the shape of the strig. This kid of represetatio is called a Fourier Series, ad there is a tremedous amout of mathematical lore about properties of such series ad for what classes of fuctios they ca be show to exist. Oe particularly useful fact about them is the orthogoality property of sies: L L si(πkx/l)si(πjx/l)dx = δ j,k, x=0 2 for oegative itegers j ad k. Hereδ j,k is the Kroecker delta fuctio, which is 0 if j = k ad 1ifj = k. The itegral above, the, is 0 uless j = k, i which case it is L/2. To see this, you ca write 1 1 si(πkx/l)si(πjx/l) = cos(π(k j)x/l) cos(π(k + j)x/l), 2 2 ad realize that uless j = ±k, each of these cosies itegrates to 0 over this rage. By multiplyig the expressio for y(x) abovebysi(πjx/l), ad itegratig the result from 0 to L, by the orthogoality property everythig cacels except the si(πjx/l) term, ad we get the expressio 2 L a j = L f(x)si(πjx/l)dx. x=0 Now, the above sum of sies is a very useful way to represet a fuctio which is 0 at both edpoits. If we are tryig to represet a fuctio o the real lie which is periodic with period L, FFT-1
it is ot quite as useful. This is because the sum of sies above is ot periodic with period L but oly periodic with period 2L. For periodic fuctios, a better Fourier expasio is y(x) =a 0 + a j cos(2πjx/l)+ j=1 bk si(2πkx/l). It is fairly easy to rewrite this as a sum of expoetials (over the complex umbers), usig the idetity e ix = cos(x)+i si(x) which implies cos x = si x = k=1 e ix + e ix 2 e ix e ix. 2i This results i the expressio (with a differet set of coefficiets c j ) y(x) = 1 c e 2πijx/L L j= j, (1) where i is the stadard imagiary uit with i 2 = 1. The scalig factor 1 L simplicity; we will see why shortly. The orthogoality relatios are ow L e 2πijx/L e 2πikx/L dx = δ j,kl, x=0 is itroduced here for ad thus, after dividig by L, we get that the itegral is 0 or 1. This meas that we ow ca recover the c j coefficiet from y by calculatig the itegral L y(x)e 2πijx/L x=0 c j = dx. (2) (2) is referred to as the Fourier trasform ad (1) to as the iverse Fourier trasform. If we had t itroduced the factor 1/L i (1), we would have to iclude it i (2), but the covetio is to put it i (1). 2 The Discrete Fourier Trasform Suppose that we have a fuctio from some real-life applicatio which we wat to fid the Fourier series of. I practice, we re ot goig to kow the value of the fuctio o every poit betwee 0adL, but just o some fiite umber of poits. Let s assume that we have the fuctio at equally spaced poits, ad do the best that we ca. This gives us the fiite Fourier trasform, also kow as the Discrete Fourier Trasform (DFT). We have the fuctio y(x) o poits j, forj =0, 1,...,; let us deote these values by y j for j =0, 1,, 1. We defie the discrete Fourier trasform of y 0,...,y 1 to be the coefficiets c 0,...,c,where c k = y j e 2πijk/, (3) FFT-2
for k =0,, 1. Observe that it would ot make sese to defie (these complex Fourier coefficiets) c k for more values of k sice the above expressio is uchaged whe we add to k (sice e 2πi =1). This makes sese if we start with complex umbers y j s, we ed up with complex umbers c k s, so we keep the same umber of degrees of freedom. Ca we recover the y j s, give the c k s? Yes, this is kow as the iverse Fourier trasform, ad is stated below. Theorem 1. If c 0,c 1,...,c for j =0,, 1. 1 is the discrete Fourier trasform of y0,...,y.the 1 1 y j = c 2 ke πijk/, (4) k=0 The equatio (4) is kow as the iverse discrete Fourier trasform. Observe that this is similar to (1), except that the scalig factor 1 which replaces 1 L is ot at the same place. 1 The proof of Theorem 1 will be based o the followig lemma. Lemma 1. If z is a complex umber satisfyig z =1ad z,z 2,...,z =1the we have the followig orthogoality relatio : for all j, k {0, 1,..., 1}, z jl z kl = δ j,k, where δ j,k is the Kroecker delta fuctio, which is 0 if j = k ad1ifj = k. Observe that z = e 2πi/ satisfies the coditio of the Lemma 1 so the orthogoality relatios tur ito the sum e 2πijl/ e 2πikl/ = δ j,k. Proof of Lemma 1. z jl z kl = (z j k ) l = w l where w = z j k. If k = l the w = 1 ad the above sum equals. O the other had, if k = l the our assumptio o z i =1fori {1, 2,, 1} meas that w =1. Thus,wehave wl =(w 1)/(w 1) = 0, sice w =1. We are ow ready to prove the Theorem. 1 This is just a matter of covetio. Actually, to avoid the cofusio that this 1 factor may create, sometimes this factor of 1/ is distributed equally, with a 1/ o both the forward ad the iverse Fourier trasforms; we will ot use this. FFT-3
Proof of Theorem 1. The defiitio of c k ca be writte as where z = e 2πi/. Now we ca compute c = y z kj k j, 1 1 cle = cl z 2πijl/ jl k=0 1 = ( yk z kl )z jl k=0 = ( ) 1 y k z kl z jl k=0 = y j. where the last equality comes from applyig Lemma 1 to z (which shows that all but oe of the ier sums are 0). 3 Computig the discrete Fourier trasform It s easy to compute the fiite Fourier trasform or its iverse if you do t mid usig O( 2 ) computatioal steps. The formulas (4) ad (3) above both ivolve a sum of terms for each of coefficiets. However, there is a beautiful way of computig the fiite Fourier trasform (ad its iverse) i oly O( log ) steps. Oe way to uderstad this algorithm is to realize that computig a fiite Fourier trasform is equivalet to pluggig ito a degree 1 polyomial at all the -th roots of uity, e 2πik/,for 0 k 1. (Recall that a -th root of uity is ay (complex) umber such that z =1;for example, the 4th root of uity are 1, e iπ/2 = i, e iπ = 1 ade i3π/2 = i.) The Fourier trasform ad its iverse are essetially the same for this part, the oly differece beig which -th root of uity you use, ad that oe of them has to get divided by. So, let s do the forward discrete Fourier trasform (3). Suppose we kow the values of y j ad we wat to compute the c k usig the Fourier trasform, (3). Let the polyomial p(x) be p(x) = y j x j. Now, let z = e 2πi/. The, it is easy to check that we have c k = p(z k ). This shows we ca express the problem of computig the Fourier trasform as evaluatig the polyomial p (of degree 1) at the -th roots of uity. (If we were computig the iverse oe FFT-4
(i.e. exchage the role of yj ad c k), we would use the root z = e 2πi/ ad divide the overall result by 1/.) What we will show is that if is eve, say =2s, it will be possible to fid two degree s 1 polyomials (thus of degrees roughly half the degree of p(x)), p eve ad p odd, such that we get all of the values c k for 0 k 1 by pluggig i the s-th roots of uity (rather tha the -th roots of uity) ito p eve ad p odd. The evaluatio of p at eve powers of z will appear whe evaluatig p eve, ad the odd powers of z will appear i p odd.if is a multiple of 4, we ca the repeat this step for each of p eve ad p odd, so we ow have our values of c k appearig as the values of four polyomials of degree /4 1, whe we plug the 4 -th uits of uity, i.e., the powers of z 4,itoall of them. If is a power of 2, we ca cotiue i the same way, ad evetually reduce the problem to evaluatig polyomials of degree 0. But it s really easy to evaluate a polyomial of degree 0: the evaluatio is the polyomial itself, which oly has a costat term. So at this poit we will be doe. The ext questio we address is: how do we fid these two polyomials p eve ad p odd? We will dothecaseofp eve first. Let us cosider a eve power of z, sayz 2k, at which we wat to evaluate p( ). We look at the j-th term ad the (j + s)-th term. These are But sice z 2s = z =1,wehave y j z 2kj ad y j+s z 2kj+2ks. z 2kj+2ks = z 2kj. Thus, we ca combie these terms ito a ew term i the polyomial p eve, with coefficiets If we let we fid that b j = y j + y j+s. s 1 p eve (x) = b j x j p(z 2k )=p eve (z 2k ). Observe furthermore that sice z k is a -throotofuity,z 2k is a s-throotofuity(sice =2s). Now, let us do the case of the odd powers. Suppose we are evaluatig p at a odd power of z, say z 2k+1. Agai, let s cosider the cotributio from the j-th ad the (j + s)-th terms together. This cotributio is y j z (2k+1)j (2k+1)(j+ s) + yj+ sz. Here we fid that z (2k+1)s = e (2πi)(2k+1)s/ = e (πi)(2k+1) = 1. We ow have yj z (2k+1)j + yj+sz (2k+1)(j+s) = (y z j )z 2kj +( j 2 j y j+s z )z kj ( 1) = j 2kj (y j yj+s)z z. Settig the j-th coefficiet of p odd to bj =(y j y j+s )z j FFT-5
ad lettig we see that s 1 p odd (x) = bj x j p(z 2k+1 )=p odd (z 2k ). What we just did was reduce the problem of evaluatig oe degree 1 polyomial, p, atthe -th roots of uity to that of evaluatig two degree 2 1 polyomials, p odd ad p eve at the 2 -th roots of uity. That is, we have take a problem of size ad reduced it to solvig two problems of size 2. We ve see this type of recursio before i sortig, ad you should recogize that it will give you a O( log ) algorithm for fidig the fiite Fourier trasform. So ow, we ca show how the Fast Fourier trasform is doe. Let s take =2 t. Now, cosider a t table, as we might make i a spreadsheet. Let s put i our top row the umbers y 0 through y 1. I the ext row, we ca, i the first 2 places, put i the coefficiets of peve, adthei the ext 2 places, put i the coefficiets of p odd. I the ext row, we repeat the process, to get four polyomials, each of degree 4 1. After we have evaluated the secod row, we treat each of peve ad p odd separately, so that othig i the first 2 colums subsequetly affects aythig i the last colums. I the third row, we will have i the first 2 4 places the coefficiets of p eve,eve, which give us the value of p(z 4k ) whe we evaluate p eve,eve(z 4k ). The i the ext 4 places, we put i the coefficiets of p eve,odd. This polyomial will give the value of p(z 4k+2 ) whe we evaluate p eve,odd (z 4k ). The third 4 places will cotai the coefficiets of p odd,eve,whichgivesusthevalues of p(z 4k+1 ). The last 4 places will be occupied by the coefficiets of podd,odd, which gives the values of p(z 4k+3 ). From ow o, we treat each of these four blocks of 4 colums separately. Ad so o. There are two remaiig steps we must remember to carry out. The first step arises from the fact that is that the values of p(z k ) come out i the last row i a fuy order. We have to reshuffle them so that they are i the right order. I will do the example of = 8. Recall that i the secod row, the polyomial p o, givig odd powers of z, followed p e, givig eve powers of z. Ithethird row, first we get the polyomial givig z 4k,thez 4k+2,thez 4k+1,thez 4k+3. So i the fourth row (which is the last row for = 8), we get the values of p(z k ) i the order idicated below. 0 1 2 3 4 5 6 7 coefficiets of p p e(z 2k )=p(z 2k ) p o (z 2k )=p(z 2k+1 ) p e,e(z 4k )=p(z 4k ) p e,o (z 4k )=p(z 4k+2 ) p o,e (z 4k )=p(z 4k+1 ) p o,o (z 4k )=p(z 4k+3 ) p(z 0 ) p(z 4 ) p(z 2 ) p(z 6 ) p(z 1 ) p(z 5 ) p(z 3 ) p(z 7 ) You ca figure out where each etry is supposed to go is by lookig at the umbers i biary, ad turig the bits aroud. For example, the etry i colum 6 (the 7th colum as we start labelig with 0) is p(z 3 ). You ca figure this out by expressig 6 i biary: 110. You the read this biary umber from right to left, to get 011, which is 3. Thus, the etry i the 6 colum is p(z 3 ). The reaso this works is that i the procedure we used, puttig i the eve powers of z first, ad the the odd powers of z, we were essetially sortig the powers of z by the 1 s bit. The ext row eds up sortig them by the 2 s bit, ad the ext row the 4 s bit, ad so forth. If we had sorted startig with the leftmost bit rather tha the rightmost, this would have put the powers i umerical order. So, by umberig the colums i biary, ad reversig the bits of these biary umbers, we get the right order of the trasformed sequece. FFT-6
The other thig we have to do is to remember to divide by if it is ecessary. We oly eed do this for the iverse Fourier trasform, ad ot the forward Fourier trasform. 4 Computig covolutios of sequeces usig Fast Fourier Trasform Suppose you have two sequeces f 0,f 1,...,f 1 ad g 0,g 1,...,g 1 ad wat sequece h 0,h 1,...,h 1 defied by h k = f j g k j to compute the where the idex k j is take modulo. Clearly it is possible to compute the umbers h 0,...,h i 2 arithmetic operatios. We will ow explai how to do it faster. Let a k ad b k be the discrete Fourier trasform of f k ad g k ad their fiite Fourier, that is, a k = f j e 2πijk/ j b k = g j e 2πijk/. j Now let s compute the iverse Fourier trasform of the sequece a k b k. For all l =0,, 1we get : 1 1 e 2πilk/ akb k = e f j e gj e k = 2πilk/ 2πijk/ 2πij k/ k j j 1 f g j j k = f j g l j = h l e 2πik(l j j j j )/ where the secod last equality holds because the sum over k is 0 uless l j + j (mod ). We have just foud a way of computig the sequece h 0,h 1,...,h by first applyig Fourrier trasform to the sequeces f 0,f 1,...,f 1 ad g 0,g 1,...,g 1 ad the takig the iverse Fourier trasform of the sequece a k b k. Sice the Fourier trasforms ad iverse Fourrier trasform ca be computed i O( log()) operatios, the sequece h 0,h 1,...,h ca be computed i O( log()) istead of O( 2 )operatios. We ca ow use this method i order to multiply polyomials efficietly. Suppose we have two degree d polyomials, ad we wat to multiply them. This correspods to covolutio of the two series that make up the coefficiets of the polyomials. If we do this the obvious way, it takes O(d 2 ) steps. However, if we use the Fourier trasform, multiply them poitwise, ad trasform back, we use O(d log d) steps for the Fourier trasforms ad O(d) steps for the multiplicatio. This gives O(d log d) total, a great savigs. We must choose the for the Fourier series carefully. If we multiply two degree d polyomials, the resultig polyomial has degree 2d, or2d +1terms. We FFT-7
must choose 2d + 1, because we eed to have room i our sequece f 0, f 1,...f for all the coefficiets of the polyomial; if we choose too small, the covolutio will wrap aroud ad we ll ed up addig the last terms of our polyomial to earlier terms. 5 Fourier trasforms modulo p ad fast iteger multiplicatio So far, we ve bee doig fiite Fourier trasforms over the complex umbers. We ca actually work over ay field with a primitive -th root of uity, thatis,aumberz such that z = 1 ad z,z 2,...,z = 1. Ideed if such a z exists, we ca defie the Fourier trasform of some umber y 0,...,y as c k = y j z jk. I this case we ca prove similarly as i Sectio 2 that the iverse Fourier trasform is y j = 1 c z jk k. k=0 The factor 1 is the multiplicative iverse of over this field, ad comes from the fact that 1 0 k=0 z =. If we take a prime p, the the field of itegers mod p has a primitive -throotofuityif p = m + 1 for some iteger m. I this case, we ca take the Fourier trasform over the itegers mod p. Thus, 17 has a primitive 16-th root of uity, oe of which ca be see to be 3. (By Fermat s little theorem, ay a = 0 satisfies a 16 1 (mod 17), but for may a s, a smaller power tha 16 will give 1. For example, modulo 17, 1 is a primitive 1st root of uity, 16 is a primitive 2d root of uity, 4 ad 13 are primitive 4-th root of uity, 2, 8, 9 ad 15 are primitive 8th roots of uity ad 3, 5, 6, 7, 10, 11, 12 ad 14 are primitive 16-th root of uity.) So if we use z = 3 i our fast Fourier trasform algorithm, ad take all arithmetic modulo 17, we get a fiite Fourier trasform. Ad we have see how to compute 1 modulo a prime p by the Euclidea gcd. We ca use this for multiplyig polyomials. Suppose we have two degree d polyomials, each of which has iteger coefficiets of size less tha B. The largest possible coefficiet i the product is (B 1) 2 (d + 1). If we wat to distiguish betwee positive ad egative coefficiets of this size, we eed to make sure that p>2(b 1) 2 (d+1). We also eed to choose 2d+1,soastohaveatleast as may terms as there are coefficiets i the product. We ca the use the Fast Fourier trasform (mod p) to multiply these polyomials, with oly O(d log d) operatios (additios, multiplicatios, takig remaiders modulus p), where we would have eeded d 2 origially. Now, suppose you wat to multiply two very large itegers. Our regular represetatio of these itegers is as k d k10 k,whered k k are the digits. We ca replace this by k d kx to tur it ito a polyomial, the multiply the two polyomials usig the fast Fourier trasform. How may steps does this take? To make thigs easier, let s assume that our large itegers are give i biary, ad that we use a base B which is a power of 2. Let s assume the large itegers have N bits each ad that we use a base B (e.g., 10 i the decimal system, 2 i biary) that has b bits. We the have our umber broke up ito N/b digits of b bits each. How large does our prime have to be? It has to be larger tha the largest possible coefficiet i the product of our two FFT-8
polyomials. This coefficiet comes from the sum of at most N/b terms, each of which has size at most (2 b 1) 2 < 2 2b. Thismeasthatwearesafeifwetakep at least N ( )2 2b b or takig logs, p must have aroud 2b +log2 N b bits. Rather tha optimizig this perfectly, let s just set the two terms i this formula to be approximately equal by lettig b =log 2 N; this is much simpler ad will give us the right asymptotic growth rate. We thus get that p has aroud 3 log 2 N bits. We the set to be a power of 2 larger tha 2 N b, so that our fiite Fourier trasform ivolves O( log ) =O(N) operatios, each of which may be a operatio o a (3 log 2 N)-bit umber. If we use loghad multiplicatio ad divisio (takig O(b 2 ) time) to do these operatios, we get a O(N log 2 N)-time algorithm. There s o reaso that we eed to stop there. We could always use recursio ad perform these operatios o the 3b-bit umbers usig fast iteger multiplicatio as well. If we use two levels of recursio, we get a O(N log N(log log N) 2 ) time algorithm. If we use three levels of recursio, we get a O(N log N(log log N)(log log log N) 2 time algorithm, ad so forth. It turs out, although we wo t go ito the details, that you ca get a O(N log N log log N) time algorithm. The mai differece from what we ve doe is that you choose the umber you use k to do the FFT ot of size aroud log N, but of a umber of the form 2 2 + 1 of size aroud N (it actually does t have to be prime). You the carefully compute the time take by applyig k this algorithm recursively, makig sure that you use the fact that mod 2 2 + 1, multiplicatio by small powers 2 ca be accomplished fairly easily by just shiftig bits. Details ca be foud i Aho, Hopcroft ad Ullma s book Desig ad Aalysis of Computer Algorithms. I fact, very recetly, still usig the fiite Fourier trasform, Furer foud a way to speed up multiplicatio eve further so that the ruig time is oly a tiy bit more tha O(N log N). FFT-9
MIT OpeCourseWare http://ocw.mit.edu 18.310 Priciples of Discrete Applied Mathematics Fall 2013 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.