Lecture 7: Hypothesis Testing and KL Divergence

Size: px

Start display at page:

Download "Lecture 7: Hypothesis Testing and KL Divergence"

Edward Marsh
7 years ago
Views:

1 C 83 Sprng 25 Statstcal Sgnal Processng nstructor: R. Nowak Lecture 7: Hypothess Testng and KL Dvergence Introducng the Kullback-Lebler Dvergence Suppose X, X 2,..., X n d qx and we have two models for qx, p x and p x. In past lectures we have seen that the lkelhood rato test LRT s optmal, assumng that q s p or p. The error probabltes can be computed numercally n many cases. The error probabltes converge to as the number of samples n grows, but numercal calculatons do not always yeld nsght nto rate of convergence. In ths lecture we wll see that the rate s exponental n n and parameterzed the Kullback-Lebler KL dvergence, whch quantfes the dfferences between the dstrbutons p and p. Our analyss wll also gve nsght nto the performance of the LRT when q s nether p nor p. Ths s mportant snce n practce p and p may be mperfect models for realty, q n ths context. The LRT acts as one would expect n such cases, t pcks the model that s closest n the sense of KL dvergence to q. To begn our dscuson, recall the lkelhood rato s Λ = n = p x p x The log lkelhood rato, normalzed by dvdng by n, s then ˆΛ n = n n = log p x p x Note that ˆΛ n s tself a random varable, and s n fact a sum of d random varables L = log px p x whch are ndependent because the x are. In addton, we know from the strong law of large numbers that for large n, ˆΛ n ˆΛ a.s. = n ˆΛn n L = = L = log p x p x qxdx p x qx = log qxdx p x qx = log qx qx log qxdx p x p x = log qx p x qxdx log qx p x qxdx

2 Lecture 7: Hypothess Testng and KL Dvergence 2 The quantty log qx px qxdx s known as the Kullback-Lebler Dvergence of p from q, or the KL dvergence for short. We use the notaton for contnuous random varables, and Dq p = qx log qx px dx Dq p = q log q p for dscrete random varables. The above expresson for ˆΛn can then be wrtten as ˆΛn = Dq p Dq p Therefore, for large n, the log lkelhood rato test ˆΛ n H λ s approxmately performng the comparson Dq p Dq p H λ snce ˆΛ n wll be close to ts mean when n s large. Recall that the mnmum probablty of error test assumng equal pror probabltes for the two hypotheses s obtaned by settng λ =. In ths case, we have the test Dq p H Dq p For ths case, usng the LRT s selectng the model that s closer to q n the sense of KL dvergence. xample Suppose we have the hypotheses H : X,..., X n d N µ, σ 2 : X,..., X n d N µ, σ 2

3 Lecture 7: Hypothess Testng and KL Dvergence 3 Then we can calculate the KL dvergence: log p x p x = log exp 2πσ 2 2σ x µ 2 2 exp 2πσ 2 2σ x µ 2 2 = x µ 2σ 2 2 x µ 2 = 2xµ 2σ 2 + µ 2 + 2xµ µ 2 Dp p = log p x p x p x dx = p log p p = p 2xµ 2σ 2 + µ 2 + 2xµ µ 2 = 2µ 2σ 2 µ p x + µ 2 µ 2 = 2mu 2 2σ 2 + µ 2 + 2µ µ µ 2 = µ 2 2σ 2 2µ µ + µ 2 = µ µ 2 2σ 2 So the KL dvergence between two Gaussan dstrbutons wth dfferent means and the same varance s just proportonal to the squared dstance between the two means. In ths case, we can see by symmetry that Dp p = Dp p, but n general ths s not true. 2 A Key Property The key property n queston s that Dq p, wth equalty f and only f q = p. To prove ths, we wll need a result n probablty known as Jensen s Inequalty: Jensen s Inequalty: If a functon fx s convex, then A functon s convex f λ, fx f x f λx + λy λfx + λfy The left hand sde of ths nequalty s the functon value at some pont between x and y, and the rght hand sde s the value of a straght lne connectng the ponts x, fx and y, fy. In other words, for a convex functon the functon value between two ponts s always lower than the straght lne between those ponts. Now f we rearrange the KL dvergence formula,

4 Lecture 7: Hypothess Testng and KL Dvergence 4 Dq p = qx log qx px dx = q log qx px = q log px qx we can use Jensen s nequalty, snce log z s a convex functon. Therefore Dq p. px log q qx = log qx px qx dx = log pxdx = log = 3 Boundng the rror Probabltes The KL dvergence also provdes a means to bound the error probabltes for a hypothess test. For ths we wll need the followng tal boud for averages of ndependent subgaussan random varables. SubGaussan Tal Bound: If Z,..., Z n are ndependent and P Z Z t ae bt2 /2,, then P n Z Z > ɛ e cnɛ2 and wth c = b 6a. P Z Z > ɛ e cnɛ2 n Proof: Follows mmedately from Theorems 2 and 3 n Now suppose that p and p have the same support and that the log lkelhood rato statstc L := log px p x has a subgaussan dstrbuton;.e., P L L t ae bt2 /2. For example, f p and p are Gaussan dstrbutons wth a common varance, then Z s a lnear functon of x and thus s Gaussan and hence subgaussan. Note that ˆΛ n = n L s an average of d subgaussan random varables. Ths allows us to use the tal bound above.

5 Lecture 7: Hypothess Testng and KL Dvergence 5 Consder the hypothess test ˆΛ H d n. We wll now assume that the data X,..., X n q, wth q ether p or p. We can wrte the probablty of false postve error as P F P ˆΛn > ˆΛn ˆΛn > ˆΛn The quantty ˆΛn wll be the ɛ n tal bound. We can re-express t as p ˆΛn = p x log p x p x dx = p x log p x p x dx = Dp p Applyng the tal bound, we get P F P ˆΛn Dp p > Dp p e cnd2 p p. Thus the probablty of false postve error s bounded n terms of the KL dvergence Dp p. As n or Dp p ncrease, the error decreases exponentally. The bound for the probablty of a false negatve error can be found n a smlar fashon: P F N ˆΛn < H ˆΛn Dp p < Dp p H Dp p ˆΛ n > Dp p H e cnd2 p p.

THE METHOD OF LEAST SQUARES THE METHOD OF LEAST SQUARES

THE METHOD OF LEAST SQUARES THE METHOD OF LEAST SQUARES The goal: to measure (determne) an unknown quantty x (the value of a RV X) Realsaton: n results: y 1, y 2,..., y j,..., y n, (the measured values of Y 1, Y 2,..., Y j,..., Y n ) every result s encumbered