Georey E. Hinton. University oftoronto. Technical Report CRG-TR May 21, 1996 (revised Feb 27, 1997) Abstract

Transcription

1 The EM Algorthm for Mxtures of Factor Analyzers Zoubn Ghahraman Georey E. Hnton Department of Computer Scence Unversty oftoronto 6 Kng's College Road Toronto, Canada M5S A4 Emal: zoubn@cs.toronto.edu Techncal Report CRG-TR-96- May 2, 996 (revsed Feb 27, 997) Abstract Factor analyss, a statstcal method for modelng the covarance structure of hgh dmensonal data usng a small number of latent varables, can be extended by allowng derent local factor models n derent regons of the nput space. Ths results n a model whch concurrently performs clusterng and dmensonalty reducton, and can be thought of as a reduced dmenson mxture of Gaussans. We present an exact Expectaton{Maxmzaton algorthm for ttng the parameters of ths mxture of factor analyzers. Introducton Clusterng and dmensonalty reducton have long been consdered two of the fundamental problems n unsupervsed learnng (Duda & Hart, 973 Chapter 6). In clusterng, the goal s to group data ponts by smlartybetween ther features. Conversely, n dmensonalty reducton, the goal s to group (or compress) features that are hghly correlated. In ths paper we present an EM learnng algorthm for a method whch combnes one of the basc forms of dmensonalty reducton factor analyss wth a basc method for clusterng the Gaussan mxture model. What results s a statstcal method whch concurrently performs clusterng and, wthn each cluster, local dmensonalty reducton. Local dmensonalty reducton presents several benets over a scheme n whch clusterng and dmensonalty reducton are performed separately. Frst, derent features may be correlated wthn derent clusters and thus the metrc for dmensonalty reducton may need to vary between derent clusters. Conversely, the metrc nduced n dmensonalty reducton may gude the process of cluster formaton.e. derent clusters may appear more separated dependng on the local metrc. Recently, there has been a great deal of research on the topc of local dmensonalty reducton, resultng n several varants on the basc concept wth successful applcatons to character and face recognton (Bregler and Omohundro, 994 Kambhatla and Leen, 994 Sung and Poggo, 994 Schwenk and Mlgram, 995 Hnton et al., 995). The algorthm used by these authors for dmensonalty reducton s prncpal components analyss (PCA).

2 - z? x Fgure : The factor analyss generatve model (n vector form). PCA, unlke maxmum lkelhood factor analyss (FA), does not dene a proper densty model for the data, as the cost of codng a data pont s equal anywhere along the prncpal component subspace (.e. the densty s un-normalzed along these drectons). Furthermore, PCA s not robust to ndependent nose n the features of the data (see Hnton et al., 996, for a comparson of PCA and FA models). Hnton, Dayan, and Revow (996), also explorng an applcaton to dgt recognton, were the rst to extend mxtures of prncpal components analyzers to a mxture of factor analyzers. Ther learnng algorthm conssted of an outer loop of approxmate EM to t the mxture components, combned wth an nner loop of gradent descent to t each ndvdual factor model. In ths note we present an exact EM algorthm for mxtures of factor analyzers whch obvates the need for an outer and nner loop. Ths smples the mplementaton, reduces the number of heurstc parameters (.e. learnng rates or steps of conugate gradent descent), and can potentally result n speed-ups. In the next secton we present background materal on factor analyss and the EM algorthm. Ths s followed by the dervaton of the learnng algorthm for mxture of factor analyzers n secton 3. We close wth a dscusson n secton 4. 2 Factor Analyss In maxmum lkelhood factor analyss (FA), a p-dmensonal real-valued data vector x s modeled usng a k-dmensonal vector of real-valued factors, z, where k s generally much smaller than p (Evertt, 984). The generatve model s gven by: x =z + u () where s known as the factor loadng matrx (see Fgure ). The factors z are assumed to be N (0 I) dstrbuted (zero-mean ndependent normals, wth unt varance). The p- dmensonal random varable u s dstrbuted N (0 ), where s a dagonal matrx. The dagonalty of soneofthekey assumptons of factor analyss: The observed varables are ndependent gven the factors. Accordng to ths model, x s therefore dstrbuted wth zero mean and covarance 0 + and the goal of factor analyss s to nd the and that best model the covarance structure of x. The factor varables z model correlatons between the elements of x, whle the u varables account for ndependent nose n each elementofx. The k factors play the same role as the prncpal components n PCA: They are nformatve proectons of the data. Gven and, the expected value of the factors can be 2

3 computed through the lnear proecton: E(zx) =x (2) where 0 ( + 0 ), a fact that results from the ont normalty of data and factors: " #! " # " #! x P = N : (3) z 0 0 I Note that snce s dagonal, the p p matrx ( + 0 ), can be ecently nverted usng the matrx nverson lemma: ( + 0 ) = ; (I + 0 ) 0 where I s the k k dentty matrx. Furthermore, t s possble (and n fact necessary for EM) to compute the second moment of the factors, E(zz 0 x) = Var(zx)+E(zx)E(zx) 0 = I ; +xx 0 0 (4) whch provdes a measure of uncertanty n the factors, a quantty that has no analogue n PCA. The expectatons (2) and (4) form the bass of the EM algorthm for maxmum lkelhood factor analyss (see Appendx A and Rubn & Thayer, 982): E-step: Compute E(zx )ande(zz 0 x ) for each data pont x, gven and. M-step: = nx nx x E(zx ) 0! = l= = ( nx n dag x x 0 ; E[zx ]x 0 = E(zz 0 x l )! (5) ) (6) where the dag operator sets all the o-dagonal elements of a matrx to zero. 3 Mxture of Factor Analyzers Assume wehave a mxture of m factor analyzers ndexed by!, = ::: m. The generatve model now obeys the followng mxture dstrbuton (see Fgure 2): P (x) = mx Z = P (xz! )P (z! )P (! )dz: (7) As n regular factor analyss, the factors are all assumed to be N (0 I) dstrbuted, therefore, P (z! )=P (z) =N (0 I): (8) 3

4 ! S S - Sw / x z Fgure 2: The mxture of factor analyss generatve model. Whereas n factor analyss the data mean was rrelevant and was subtracted before ttng the model, here we have the freedom to gve each factor analyzer a derent mean,, thereby allowng each to model the data covarance structure n a derent part of nput space, P (xz! )=N ( + z ): (9) The parameters of ths model are f( ) m = g the vector parametrzes the adaptable mxng proportons, = P (! ). The latent varables n ths model are the factors z and the mxture ndcator varable!, where w = when the data pont was generated by!. For the E-step of the EM algorthm, one needs to compute expectatons of all the nteractons of the hdden varables that appear n the log lkelhood. Fortunately, the followng statements can be easly vered, Denng E[w zx ] = E[w x ] E[z! x ] (0) E[w zz 0 x ] = E[w x ] E[zz 0! x ]: () h = E[w x ] / P (x! )= N (x ; 0 + ) (2) and usng equatons (2) and (0) we obtan E[w zx ]=h (x ; ) (3) where 0 ( + 0 ). Smlarly, usng equatons (4) and () we obtan E[w zz 0 x ]=h I ; + (x ; )(x ; ) 0 0 : (4) The EM algorthm for mxtures of factor analyzers therefore becomes: E-step: Compute h, E[zx! ]ande[zz 0 x! ] for all data ponts and mxture components. M-step: Solve a set of lnear equatons for,, and (see Appendx B). The mxture of factor analyzers s, n essence, a reduced dmensonalty mxture of Gaussans. Each factor analyzer ts a Gaussan to a porton of the data, weghted by the posteror probabltes, h. Snce the covarance matrx for each Gaussan s speced through the lower dmensonal factor loadng matrces, the model has mkp + p, rather than mp(p + )=2, parameters dedcated to modelng covarance structure. Note that each model can also be allowed to have a separate matrx. Ths, however, changes ts nterpretaton as sensor nose. 4

5 4 Dscusson We have descrbed an EM algorthm for ttng a mxture of factor analyzers. Matlab source code for the algorthm can be obtaned from ftp://ftp.cs.toronto.edu/pub/zoubn/ mfa.tar.gz. An extenson of ths archtecture to tme seres data, n whch both the factors z and the dscrete varables! depend on ther value at a prevous tme step, s currently beng developed. One of the mportant ssues not addressed n ths note s model selecton. In ttng a mxture of factor analyzers the modeler has two free parameters to decde: The number of factor analyzers to use (m), and the number of factor n each analyzer (k). One method by whch these can be selected s cross-valdaton: several values of m and k are t to the data and the log lkelhood on a valdaton set s used to select the nal values. Greedy methods based on prunng or growng the mxture may be more ecent at the cost of some performance loss. Alternatvely, a full-edged Bayesan analyss, n whch these model parameters are ntegrated over, may also be possble. Acknowledgements We thank C. Bshop for comments on the manuscrpt. The research was funded by grants from the Canadan Natural Scence and Engneerng Research Councl and the Ontaro Informaton Technology Research Center. GEH s the Nesbtt-Burns fellow of the Canadan Insttute for Advanced Research. A EM for Factor Analyss The expected log lkelhood for factor analyss s Q = E " log Y = c ; n 2 log ;X = c ; n 2 log ;X (2) p=2 =2 expf; 2 [x ; z] 0 [x ; z]g E 2 x0 x ; x 0 z + 2 z0 0 2 x0 x ; x 0 # z E[zx ]+ 2 tr h 0 E[zz 0 x ] where c s a constant, ndependent of the parameters, and tr s the trace operator. To re-estmate the factor loadng matrx = ; X x E[zx ] 0 + X l E[zz 0 x l ]=0 obtanng E[zz 0 x l ] 0! = X x E[zx ] 0 X l 5

6 from whch we get equaton (5). We re-estmate the matrx through ts nverse, = n X 2 2 x x 0 ; E[zx ] x E[zz 0 x ] 0 Substtutng equaton (5), =0: n 2 = X 2 x x 0 ; 2 E[zx ] x 0 and usng the dagonal constrant, = n dag ( X x x 0 ; E[zx ]x 0 ) : B EM for Mxture of Factor Analyzers The expected log lkelhood for mxture of factor analyss s Q = E 2 4 Y Y log (2) p=2 =2 expf; 2 [x ; ; z] 0 3 w [x ; ; z]g 5 To ontly estmate the mean and the factor loadngs t s useful to dene an augmented column vector of factors " # z ~z = and an augmented factor loadng matrx ~ =[ ]. The expected log lkelhood s then Q = E 2 Y Y 4 log = c ; n 2 log ;X (2) p=2 =2 expf; 2 [x ; ~ ~z] 0 where c s a constant. To estmate ~ weset 3 [x ; ~ w ~z]g 5 2 h x 0 x ; h x 0 ~ E[~zx! ]+ h 2 h tr ~ 0 ~ E[~z~z 0 ~ = ; X h x E[~zx! ] 0 + h ~ E[~z~z 0 x! ]=0: Ths results n a lnear equaton for re-estmatng the means and factor loadngs, h = ~ X = h x E[~zx! ] 0! X l h l E[~z~z 0 x l! ]! (5) 6

7 where and We re-estmate the matrx E[~z~z 0 x l! ]= " E[zx! E[~zx! ]= ] " E[zz0 x l! ] E[zx l! ] E[zx l! ] 0 through ts nverse, settng # = n 2 ; X 2 h x x 0 ; h ~ E[~zx! ]x h ~ E[~z~z 0 x! ] ~ 0 =0: Substtutng equaton (5) for ~ and usng the dagonal constrant on = n dag 8 < :X h x ; ~ Fnally, to re-estmate the mxng proportons we use the denton, = P (! )= Z we obtan, 9 = E[~zx! ] x 0 : (6) P (! x)p (x) dx: Snce h = P (! x ), usng the emprcal dstrbuton of the data as an estmate of P (x) we get References = n Bregler, C. and Omohundro, S. M. (994). Surface learnng wth applcatons to lp-readng. In Cowan, J. D., Tesauro, G., and Alspector, J., edtors, Advances n Neural Informaton Processng Systems 6, pages 43{50. Morgan Kaufman Publshers, San Francsco, CA. Duda, R. O. and Hart, P. E. (973). Pattern Classcaton and Scene Analyss. Wley, New York. Evertt, B. S. (984). An Introducton to Latent Varable Models. Chapman and Hall, London. Hnton, G., Revow, M., and Dayan, P. (995). Recognzng handwrtten dgts usng mxtures of Lnear models. In Tesauro, G., Touretzky, D., and Leen, T., edtors, Advances n Neural Informaton Processng Systems 7, pages 05{022. MIT Press, Cambrdge, MA. nx = h : Hnton, G. E., Dayan, P., and Revow, M. (996). handwrtten dgts. Submtted for Publcaton. Modelng the manfolds of Images of 7

8 Kambhatla, N. and Leen, T. K. (994). Fast non-lnear dmenson reducton. In Cowan, J. D., Tesauro, G., and Alspector, J., edtors, Advances n Neural Informaton Processng Systems 6, pages 52{59. Morgan Kaufman Publshers, San Francsco, CA. Rubn, D. and Thayer, D. (982). EM algorthms for ML factor analyss. Psychometrka, 47():69{76. Schwenk, H. and Mlgram, M. (995). Transformaton nvarant autoassocaton wth applcaton to handwrtten character recognton. In Tesauro, G., Touretzky, D., and Leen, T., edtors, Advances n Neural Informaton Processng Systems 7, pages 99{998. MIT Press, Cambrdge, MA. Sung, K.-K. and Poggo, T. (994). Example-based learnng for vew-based human face detecton. MIT AI Memo 52, CBCL Paper 2. 8