Supplementary Materials for Tensor Analyzers

Transcription

1 Supplementary Materals for Tensor Analyzers Ychuan Tang Ruslan Salakhudnov Geoffrey Hnton Department of Computer Scence Unversty of Toronto Toronto, Ontaro, Canada. 1 Dervatons of the M-step n EM learnng of TAs The objectve functon Q of the EM algorthm s the expected complete log-lkelhood, taken over the posteror dstrbuton of the latent factors, and summed over N tranng cases: [ Q = E log N 2π D/2 Ψ 1/2 exp{ 1 ] 2 x e T Ψ 1 x e } = const N 2 log Ψ N [ 1 E 2 xt Ψ 1 x x T Ψ 1 e + 1 ] 2 et Ψ 1 e = const N N 2 log Ψ 1 E[ 2 xt Ψ 1 x x T Ψ 1 Wy x T Ψ 1 T 1 u + y T W T Ψ 1 T 1 u yt W T Ψ 1 Wy ut T T 1 Ψ 1 T 1 u ] = const N N 2 log Ψ 1 2 xt Ψ 1 x x T Ψ 1 WE[y ] x T Ψ 1 T 1 E[u ] + E[u T T T 1 Ψ 1 Wy ] E[yT W T Ψ 1 Wy ] E[uT T T 1 Ψ 1 T 1 u ] The closed-form M-step update equatons are: Q N W = Ψ 1 x E[y ] T + Ψ 1 T 1 E[u y T ] + Ψ 1 WE[y y T ] = 0 5 N W = x E[y T ] T 1 Q T 1 = N N N 1 ] E[u y T ] E[y y T 6 Ψ 1 x E[u T ] + Ψ 1 WE[y u T ] Ψ 1 T 1 2E[u u T ] = 0 7 N N N 1 T 1 = x E[u T ] W E[y u T ] E[u u T ] 8 1

2 Q Ψ 1 = N 2 Ψ N 1 2 x x T WE[y ]x T T 1 E[u ]x T + T 1 E[u y T ]W T WE[y y T ]W T T 1E[uu T ]T T 1 = 0 9 Ψ = 1 N dag { N x x T 2WE[y ]x T 2T 1 E[u ]x T + 2T 1 E[u y T ]W T + WE[y y T ]W T + T 1 E[uu T ]T T 1 } 10 = 1 N dag { N x x T 2T 1 E[u ]x T E[u y T ]W T 1 2 E[uuT ]T T 1 2W E[y ]x T 1 2 E[y y T ]W T} 11 2 Convergence of Gbbs Samplng Posteror nference n TA usng alternatng Gbbs samplng s effcent. We present trace plots of the 6 random latent factors of two TAs n the fgure below. Left panel s from a TA learned on 2D synthetc datasets of Sec. 4, whle the rght panel s from a TA modelng hgh dmensonal face mages under llumnaton varatons. The plots demonstrate that samples mxes very quckly after around 20 Gbbs teratons. Fgure 1: left: MCMC trace plot for TA learnng on synthetc data. rght:mcmc trace plot for TA learnng on hgh dmensonal face mages. We also looked at how posteror nference converges n a 200 component MTA traned on 8 x 8 natural mage patches. 2

3 Fgure 2: Gbbs samplng quckly converges around 20 teratons. 9 components were randomly pcked from a MTA wth 200 components traned on natural mage patches. For each component, we randomly selected 6 latent factors one for every color. 3 Annealed Importance Samplng for TA We can treat the problem of estmatng log px as calculatng the partton functon of unnormalzed posteror dstrbuton p z x px, z, where p denotes an unnormalzed dstrbuton. The basc Importance Samplng gves: px = dz px, z 12 px = Z p 1 1 M px = M z z dz p z x qz 13 qz w, w = p z x, z qz 14 qz Annealed Importance Samplng specfes a set of ntermedate dstrbutons, where β vares from 0.0 to 1.0. p β z qz 1 β p z x β 15 For TAs, the log of the tractable base dstrbuton qz s: log q{z 1, z 2,..., z J } = J j=1 whch s smply the pror dstrbuton over the latent factors. d j 2 log2π 1 2 zt j z j, 16 Snce we are usng the pror as the base dstrbuton and the unnormalzed posteror dstrbuton s the dstrbuton of nterest, we can wrte the ntermedate dstrbuton as: p β {z j } q{z j } 1 β px, {z j } β = p{z j }p β x {z j } 17 3

4 Algorthm 1 AIS for TA let k = 1, 2,..., K, β k=1 = 0, β k=k = 1. for = 1 to M do Draw Z 2 from qz w = p β 2 Z 2 p β1 Z 2 for k = 2 to K 1 do Sample Z k+1 from T βk Z Z k w = w p β k+1 Z k+1 p βk Z k+1 end for end for px 1 M M w Fgure 3: left: AIS algorthm for TA. rght: Expermental valdaton of AIS on a small model, where 100,000 Monte Carlo samples estmate the true log-lkelhood. Therefore, log p β {z j} = J j dj 2 log2π 1 2 zt j z j To ensure that p β {z j} sums to 1.0, we fnd that: Therefore, log p β {z j} = J j βd 2 log2π β 2 log Ψ β 2 x et Ψ 1 x e + Cβ 18 Cβ = β 2 log Ψ + βd 2 log2π 1 2 log βψ D log2π 19 2 dj 2 log2π 1 2 zt j z j D 2 log2π 1 2 log βψ 1 2 x et βψ 1 x e For AIS, we also need a MCMC operator whch leaves p β {z 1, z 2,..., z J } nvarant. We use a Gbbs sampler, whch smply performs alternatng Gbbs samplng of the TA s posteror where the orgnal dagonal nose Ψ s modfed to be βψ. We denote ths operator as T β z z. In Fg. 3 left, We present the AIS algorthm. For clarty, we use Z k to denote the set of all latent factors Z = {z j } of the k-th ntermedate dstrbuton, k = 1, 2,..., K. M s the number of ndependent AIS chans. On the rght panel of Fg. 3, we expermented wth the varance of the AIS estmator. Usng a small model of TA{2,2,2} traned on a 2D dataset from Sec. 4.1 of the man paper, we ran AIS algorthm wth varyng number of ntermedate dstrbutons to estmate the average data log-lkelhood. M s set to 10 n all experments. In the plot, we can see that the varance of the estmator quckly shrnks to less than 0.1 nats as we use 500 or more ntermedate dstrbutons. The dashed lne represent the estmated log-lkelhood by samplng from the pror Sec. 3.3 of man paper, usng 100,000 samples. Snce we are samplng from only 2 dmensons, ths Monte Carlo estmator has very low varance and ts value s taken to be the true data log-lkelhood. 4 Experments - Synthetc Data We present both the tranng and test average log-lkelhood on 4 2D synthetc datasets n Table 1. The frst two s presented n the man paper. MTAs are better models for complcated densty and just as good as MFAs on the smpler ones. 20 4

5 a Data b Data c Samples from TA d Samples from FA e Samples from TA f Samples from FA Fgure 4: TA vs. FA on 2D synthetc data. a Data b Data v c Samples from MTA d Samples from MFA e Samples from MTA f Samples from MFA Fgure 5: MTA vs. MFA on 2D synthetc data. A mxture of two components are used n both MTA and MFA. Data and are generated by a randomly ntalzed TA model, whle Data and v are generated by a randomly ntalzed MFA model. 5

6 Dataset v MTA MFA Table 1: Average test and tranng log-lkelhood n nats of Mxture of Tensor Analyzers and Mxture of Factor Analyzers on the 2D synthetc datasets. 6