Multi-Conditional Learning for Joint Probability Models with Latent Variables

Transcription

1 Mult-Condtonal Learnng for Jont Probablty Models wth Latent Varables Chrs Pal, Xueru Wang, Mchael Kelm and Andrew McCallum Department of Computer Scence Unversty of Massachusetts Amherst Amherst, MA USA Abstract We ntroduce Mult-Condtonal Learnng, a framework for optmzng graphcal models based not on jont lkelhood, or on condtonal lkelhood, but based on a product of several margnal condtonal lkelhoods each relyng on common sets of parameters from an underlyng jont model and predctng dfferent subsets of varables condtoned on other subsets. When appled to undrected models wth latent varables, such as the Harmonum, ths approach can result n powerful, structured latent varable representatons that combne some of the advantages of condtonal random felds wth the unsupervsed clusterng ablty of popular topc models, such as latent Drchlet allocaton and ts successors. We present new algorthms for parameter estmaton usng expected gradent based optmzaton and develop fast approxmate nference algorthms nspred by the contrastve dvergence approach. Our ntal expermental results show mproved cluster qualty on synthetc data, promsng results on a vowel recognton problem and sgnfcant mprovement nferrng hdden document categores from multple attrbutes of documents. 1 Introducton Recently, there has been substantal nterest n Condtonal Random Felds (CRFs) [6] for sequence processng. CRFs are random felds for a jont dstrbuton globally condtoned on feature observatons. The CRF constructon can be contrasted wth MRFs whch have been used n the past to defne and model the jont dstrbuton of both labels and features. CRFs are also optmzed usng a Maxmum Condtonal Lkelhood objectve as opposed to a Maxmum (Jont) Lkelhood objectve whch s a tradtonal objectve functon used for MRFs. In the Machne Learnng communty the Boltzmann machne [1] s another well known example of an MRF and recent attenton has been gven to a restrcted type of Boltzman machne [3] known as a Harmonum [8]. We are nterested n more deeply explorng the relatonshps between these models, the dstrbutons they defne and the ways n whch they can be optmzed. In the approach we propose and develop here, one begns by specfyng a jont probablty model for all quanttes one wshes to consder as random quanttes, for our dscusson here we can thnk of ths as a jont model for: labels, features, hdden varables and parameters f desred. However, to optmze the jont model we propose fndng pont estmates for parameters usng an objectve functon consstng of the product of select

2 (margnal) condtonal dstrbutons. The goal of ths objectve s thus to obtan a Random Feld that has been optmzed to be good at modelng a number of condtonal dstrbutons that the modeler s partcularly nterested n focusng on capturng well wthn the context of a sngle jont model. Our experments show that latent varable models obtaned by optmzaton wth respect to ths type of objectve can produce both qualtatvely better clusters as well as quanttatvely better structured latent topc spaces. 2 Mult-Condtonal Learnng n Jont Models 2.1 A Smple Illustratve Example n a Locally Normalzed Model Here we present an example of how one can optmze a jont probablty model under a number of dfferent objectves. Consder a Gaussan mxture model (GMM) for real valued random observed varables x (e.g., observed 2D values) wth an unobserved sub-class, s assocated wth each observed class label c. We wll use the notaton x and c to denote observatons or nstantatons of contnuous and dscrete random varables. We can wrte a model for the jont dstrbuton of these random varables as P (x, c, s) = p(x s)p (s c)p (c), where P (s c) s a sparse matrx assocatng a number of sub-classes. We shall use Θ to denote all the parameters of the model. Now consder that t s possble to optmze the GMM n a number of dfferent ways. Frst, consder the log margnal jont lkelhood L x,c of such a model, whch can be expressed as: L x,c (Θ; { x }, { c }) = log P ( x, c Θ) = log s P ( x, s, c Θ) = L x,c (Θ) (1) Second, n contrast to the log margnal jont lkelhood, the log margnal condtonal lkelhood L c x can be expressed as: L c x (Θ; { x }, { c }) = log s P ( c, s x, Θ) = L x,c (Θ) L x (Θ) (2) Thrd, consder the followng mult-condtonal objectve functon, L c x,x c whch we express as: L c x,x c (Θ; { x }, { c }) = log P ( c x, Θ) + log P ( x c, Θ) = 2L x,c (Θ) L x (Θ) L c (Θ) (3) Consder now the followng smple example data set whch s smlar to the example presented n Jebara s work [4] to llustrate hs Condtonal Expectaton Maxmzaton (CEM) approach. Smlarly, we generate data from two classes, each wth four sub-classes drawn from 2D sotropc Gaussans. The data are llustrated by red s and blue s n Fgures 1. In contrast to [4], here we ft models wth dagonal covarance matrces and we use the condtonal expected gradent [7] optmzaton approach to update parameters. To llustrate the effects of the dfferent optmzaton crtera we have ft models wth two subclasses for each class. We run each algorthm wth 30 random ntalzatons usng gradent based optmzaton for the three objectve functons and choose the best model under the jont, condtonal and mult-condtonal objectves, (1), (2) and (3), respectvely. We llustrate the model parameters usng ellpses of constant probablty under the model. From ths llustratve example, we see that the jont lkelhood based objectve encodes no element explctly enforcng a good model of the condtonal dstrbuton of the class label and can thus place probablty mass n poor locatons wth respect to classfcaton. The condtonal objectve focuses completely on the decson boundary and can produce parameters wth very lttle nterpretablty. Whereas our mult-condtonal objectve explctly optmzes for a good class condtonal dstrbuton and a good settng of parameters for makng classfcatons.

3 Fgure 1: (Left) Jont lkelhood optmzaton. (Mddle) One of the many near optmal solutons found by condtonal lkelhood optmzaton. (Rght) An optmal soluton found by our multcondtonal objectve. Quanttatvely, we have found that a smlar mult-condtonal optmzaton and model selecton procedure for the 11 class, solated vowel recognton problem n [2] leads to a test set error rate of.36 compared to.40 usng the ML and CL objectves. In contrast, the best publshed result s.39 usng multvarate adaptve regresson splnes (MARS) [2]. 2.2 Our Proposed Mult-Condtonal Objectve More generally, our objectve functon can be expressed as follows. Consder a data set consstng of = 1... N observaton nstances, hdden dscrete and contnuous varables {z} and z respectvely. We defne j = 1... M pars of dsjont subsets of observed varables where { x},j represents the th nstance of the varables n subset j and { x}, j s the other half of the par (whch we wll condton upon). Usng these defntons, the optmal parameter settngs under our mult-condtonal crteron are gven by argmax P ({ x},j, {z},j, z,j { x}, j; θ)dz,j, (4) θ j {z},j where we derve these margnal condtonal lkelhoods from a sngle underlyng jont model whch tself may be normalzed locally, globally or usng some combnaton of the two. 2.3 A Structured, Globally Normalzed, Latent Varable Model for Documents A Harmonum model [8] s a Markov Random Feld (MRF) consstng of observed varables and hdden varables. Lke all MRFs the model we present here wll be defned n terms of a globally normalzed product of (un-normalzed) potental functons defned upon subsets of varables. A Harmonum can also be descrbed as a type of restrcted Boltzmann machne [3] whch can be wrtten as an exponental famly model. In partcular, the exponental famly Harmonum structured model we develop here can be wrtten as { P (x, y Θ) = exp θ T f (x ) + θ T j f j (y j ) + } θ T jf j (x, y j ) A(Θ), j j (5) where y s a vector of hdden varables, x s a vector of observatons, θ represents parameter vectors (or weghts), θ j represents a parameter vector on a cross product of states, f denotes potental functons, Θ = {θ j, θ, θ j } s the set of all parameters and A s the log-partton functon or normalzaton constant. A Harmonum model factorzes the thrd term of (5) nto θ T jf j (x, y j ) = f (x ) T Wj T f j(y j ), where Wj T s a parameter matrx wth dmensons a b,.e., wth rows equal to the number of states of f (x ) and columns equal to the number of states of f j (y j ). Fgure 2 (rght) llustrates a Harmonum model as a factor graph [5]. Importantly, a Harmonum descrbes the factorzaton of a

4 jont dstrbuton for observed and hdden varables nto a globally normalzed product of local functons. In our experments here we shall use the Harmonum s factorzaton structure to defne a MRF and we wll then defne sets of margnal condtonals dstrbutons of some observed varables gven others that are of partcular nterest so as to form our mult-condtonal objectve. Importantly, usng a globally normalzed jont dstrbuton wth ths constructon t s also possble to derve two consstent condtonal models, one for hdden varables gven observed varables and one for observed varables gven hdden varables [9]. The condtonal dstrbutons defned by these models can also be used to mplement samplng schemes for varous probabltes n the underlyng jont model. However, s mportant to remember that the orgnal model parameterzaton s not defned n terms of these condtonal dstrbutons. In our specfc experments below we use a jont model wth a form defned by (5) wth W T = [Wb T WT d ] such that the (exponental famly) condtonal dstrbutons consstent wth the jont model are gven by P (y n x) = N (y n ; ˆµ, I), ˆµ = µ + W T x and (6) P (x b ỹ) = B(x b ; ˆθ b ), ˆθb = θ b + W b ỹ and (7) P (x d ỹ) = D(x d ; ˆθ), ˆθd = θ d + W d ỹ (8) Where N (), B() and D() represent Normal, Bernoull and Dscrete dstrbutons respectvely. The followng equaton can be used to represent the margnal P (x θ, Λ) = exp{θ T x + x T Λx A(θ, Λ)} (9) where Λ = 1 2 WWT. In an exponental famly model wth exponental functon F(x; θ), t s easy to verfy that the gradent of the log jont lkelhood can be expressed as: [ ] L(θ; x) F(x; θ) F(x; θ) = N E P (x) E P (x;θ) (10) where E P (x) denotes the expectaton under the emprcal dstrbuton, E P (x) s an expectaton under the models margnal dstrbuton and N s the number of data elements. We can thus compute the gradent of the log-lkelhood under our constructon usng L(W T ; X) W T = 1 N d ( ) N d W T x x T 1 N s W T x,(j) x T,(j) N s =1 where N d are the number of vectors of observed data, x,(j) are samples ndexed by j and N s are the number of MCMC samples used per data vector and computed Gbbs samplng and condtonals (6), (7) and (8). In our experments here we have found t possble to use ether one or a small number of MCMC steps ntalzed from the data vector (the contrastve dvergence approach) but a more standard MCMC approxmaton s also possble. Fnally, for condtonal lkelhood and mult-condtonal lkelhood based learnng, gradent values can be obtaned from L = N [ j [ E P (xj,x j) F(xj, x j; θ) 3 Experments, Results and Analyss j=1 E P (x j) E P (xj x j;θ) (11) ]] F(xj, x j; θ) We are nterested n examnng the qualty of the latent representatons obtaned when optmzng mult-attrbute Harmonum structured models under ML, CL and MCL objectves. We use a smlar testng strategy to [9] but focus on comparng the dfferent latent spaces obtaned wth the dfferent optmzaton objectves. For our experments, we use the reduced 20newsgroups dataset prepared n MATLAB by Sam Rowes. In ths data (12)

5 Observed x Unobserved y Precson Recall ML 1 ML 2 CL MCL Fgure 2: (Left) A factor graph for a Harmonum model. (Rght) Precson-recall curves for the 20newsgroups data usng ML, CL and MCL wth 20 latent varables. Random guessng s a horzontal lne at.25. set, documents are represented by 100 word vocabulary bnary occurrences and are labeled as one of four domans. To evaluate the qualty of our latent space, we retreve documents that have the same doman label as a test document based on ther cosne coeffcent n the latent space when observng only bnary occurrences. We randomly splt data nto a tranng set of 12,000 documents and a test set of 4242 documents. We use jont model wth a correspondng full rank multvarate Bernoull condtonal for bnary word occurrences and a dscrete condtonal for domans. Fgure 2 shows precson-recall results. ML-1 s our model wth no doman label nformaton. ML-2 s optmzed wth doman label nformaton. CL s optmzed to predct domans from words and MCL s optmzed to predct both words from domans and domans from words. From Fgure 2 we see that the latent space captured by the model s more relevant for doman classfcaton when the model s optmzed under the CL and MCL objectves. Further, at low recall both the CL and MCL derved latent spaces produced smlar precsons. However, as recall ncreases the precson for comparsons made n the MCL derved latent space s consstently better. In concluson, these results lead us to beleve that further nvestgaton s warranted nto the the use of Mult-Condtonal Learnng methods for dervng both more meanngful and more useful hdden varable models. References [1] D. H. Ackley, G. E. Hnton, and T. J. Sejnowsk. A learnng algorthm for Boltzmann machnes. Cogntve Scence, 9: , [2] T. Haste, R. Tbshran, and J. Fredman. The Elements of Statstcal Learnng: Data Mnng, Inference, and Predcton. Sprnger-Verlag, [3] G. Hnton. Tranng products of experts by mnmzng contrastve dvergence. Neural Computaton, 14: , [4] T. Jebara and A. Pentland. On reversng jensen s nequalty. NIPS 13, [5] F. R. Kschschang, B. Frey, and H.-A. Loelger. Factor graphs and the sum-product algorthm. IEEE Transactons on Informaton Theory, 47(2): , [6] John Lafferty, Andrew McCallum, and Fernando Perera. Condtonal random felds: Probablstc models for segmentng and labelng sequence data. In ICML, pages , [7] Ruslan Salakhutdnov, Sam Rowes, and Zoubn Ghahraman. Optmzaton wth EM and expectaton-conjugate-gradent. Proceedngs of (ICML), [8] P. Smolensky. Informaton processng n dynamcal systems: foundatons of harmony theory, chapter 2, pages McGraw-Hll, New York, [9] Max Wellng, Mchal Rosen-Zv, and Geoffrey Hnton. Exponental famly harmonums wth an applcaton to nformaton retreval. In NIPS17, pages

6 4 Acknowledgements Ths work was supported n part by the Center for Intellgent Informaton Retreval, n part by the Defense Advanced Research Projects Agency (DARPA) under contract number HR C-0023, and n part by the Defense Advanced Research Projects Agency (DARPA), through the Department of the Interor, NBC, Acquston Servces Dvson, under contract number NBCHD Ths work was also supported n part by Mcrosoft Research under the Memex and escence programs. Any opnons, fndngs and conclusons or recommendatons expressed n ths materal are those of the authors and do not necessarly reflect those of the sponsor.