Model-based clustering of longitudinal data. McNicholas, Paul D.; Murphy, Thomas Brendan. Canadian Journal of Statistics, 38 (1):

Transcription

1 Povided by the autho(s) and Univesity College Dublin Libay in accodance with publishe policies Please cite the published vesion when available Title Model-based clusteing of longitudinal data Autho(s) McNicholas, Paul D; Muphy, Thomas Bendan Publication Date Publication infomation Canadian Jounal of Statistics, 8 (1): Publishe Wiley Link to publishe's vesion This item's ecod/moe infomation Rights This is the autho's vesion of the following aticle: "Modelbased clusteing of longitudinal data" published in The Canadian Jounal of Statistics Vol 4, No 4, 2006, available at DOI Downloaded T15:59:06Z Some ights eseved Fo moe infomation, please see the item ecod link above

2 The Canadian Jounal of Statistics Vol 4, No 4, 2006, Pages???-??? La evue canadienne de statistique Model-based clusteing of longitudinal data Paul D McNICHOLAS and Thomas Bendan MURPHY Key wods and phases: Cholesky decomposition; longitudinal data; mixtue models; model-based clusteing; time couse data; yeast spoulation MSC 2000: Pimay 62H0; seconday 62P10 Abstact: A new family of mixtue models fo the model-based clusteing of longitudinal data is intoduced The covaiance stuctues of eight membes of this new family of models ae given and the associated maximum likelihood estimates fo the paametes ae deived via expectation-maximization (EM) algoithms The Bayesian infomation citeion is used fo model selection and a convegence citeion based on Aitken s acceleation is used to detemine convegence of these EM algoithms This new family of models is applied to yeast spoulation time couse data, whee the models give good clusteing pefomance Futhe constaints ae then imposed on the decomposition to allow a deepe investigation of coelation stuctue of the yeast data These constaints geatly extend this new family of models, with the addition of many pasimonious models Title in Fench: we can supply this Résumé : A new family of mixtue models fo the model-based clusteing of longitudinal data is intoduced The covaiance stuctues of eight membes of this new family of models ae given and the associated maximum likelihood estimates fo the paametes ae deived via expectation-maximization (EM) algoithms The Bayesian infomation citeion is used fo model selection and a convegence citeion based on Aitken s acceleation is used to detemine convegence of these EM algoithms This new family of models is applied to yeast spoulation time couse data, whee the models give good clusteing pefomance Futhe constaints ae then imposed on the decomposition to allow a deepe investigation of coelation stuctue of the yeast data These constaints geatly extend this new family of models, with the addition of many pasimonious models 1 INTRODUCTION Longitudinal data aise when measuements ae taken on each subject at a numbe of points in time The esulting insight into behaviou ove time sepaates longitudinal data fom othe types of data Howeve, modelling longitudinal data equies special consideations; in paticula, the coelation between measuements on each subject must be taken into account Subjects in longitudinal studies, o panel studies, ae often consideed to be independent, but this is not always the case Conside, fo example, data on the weights of calves on one of two diffeent methods of contolling intestinal paasites (Kenwad, 1987) Diggle et al (1994) and Eveitt (1995) pesent a vaiety of methods that can be used to analyze these data and, in geneal, to analyze longitudinal data with known goups Due to the typically pospective natue of longitudinal data studies, goup membeships will usually be known a pioi Howeve, situations do aise whee goup membe- 1

3 ships ae not known and even whee the pupose of the analysis is to find goups, o clustes, in the data One such situation aises when the pupose of the study is to find goups of genes with simila activation pattens ove time An example of this is the study that was conducted by Chu et al (1998) to investigate the behaviou of yeast spoulation data ove time The esulting data ae analyzed in Section using the model-based clusteing technique that is intoduced in this wok Model-based clusteing is a technique fo clusteing data though the imposition of a mixtue modelling famewok A Gaussian mixtue model is most fequently used and its density is of the fom G f(x) = π g φ(x µ g, Σ g ), whee π g is the pobability of membeship of goup g and φ(x µ g, Σ g ) is the density of a multivaiate Gaussian distibution with mean µ g and covaiance Σ g Banfield and Raftey (199), Celeux and Govaet (1995) and Faley and Raftey (1998, 2002) exploited an eigenvalue decomposition of the goup covaiance matices to give a wide ange of pasimonious covaiance stuctues This wok culminated in the MCLUST family of models, which consists of ten mixtue models that aise fom the imposition of constaints on the goup covaiance matix Σ g = λ g H g A g H g, whee λ g is a constant, H g is a matix of eigenvectos of Σ g and A g is a diagonal matix with enties popotional to the eigenvalues of Σ g Details of the constaints that can be imposed ae summaized in Faley and Raftey (2006, Table 1) MCLUST is the most well-established model-based clusteing technique within the liteatue, which is patly due to the mclust package (Faley and Raftey, 200) that is available within the R softwae (R Development Coe Team, 2009) Bouveyon et al (2007) intoduced a family of mixtue models specifically fo the analysis of high-dimensional data and McNicholas and Muphy (2008) developed a family of pasimonious Gaussian mixtue models that is closely elated to the mixtue of facto analyzes model (Ghahamani and Hinton, 1997; McLachlan et al, 200) In all of these cases, the classical appoach to model-based clusteing is taken, whee each altenative covaiance stuctue coesponds to a membe of the family of mixtue models Howeve, some non-classical appoaches have been taken to the model-based clusteing of longitudinal data De la Cuz-Mesía et al (2008) use a mixtue of non-linea hieachical models The modelling paadigm that they popose, which is essentially an extension of Paule and Laid (2000), makes each component density subject-specific and the only modelling of the component covaiance matix that they engage in is the imposition of the isotopic constaint Although classical model-based clusteing continues to extend into new application aeas, none of the models that ae cuently available have a covaiance stuctue specifically designed fo the analysis of longitudinal data The aim of this pape is to intoduce a family of mixtue models with a covaiance stuctue specifically designed fo the model-based clusteing of longitudinal data Since the outcome vaiable x is ecoded in a time odeed manne, a covaiance stuctue that explicitly accounts fo the elationship between measuements at diffeent time points is necessay Pouahmadi (1999, 2000) exploited the fact that covaiance matix Σ of a andom vaiable can be decomposed using the elation T ΣT = D, whee T is a unique lowe tiangula matix with diagonal elements 1 and D is a unique diagonal matix with stictly positive enties This elation is known as the modified Cholesky decomposition and it was used by Kzanowski et al (1995) in a disciminant analysis application The modified Cholesky decomposition may equivalently be expessed in the fom Σ 1 = T D 1 T, which is convenient when modelling the covaiance of a multivaiate Gaussian distibution The values of T and D have intepetations as genealized autoegessive paametes and innovation vaiances, espectively (Pouahmadi, 1999) so that the 2

4 linea least-squaes pedicto of Y t, based on Y t 1,,Y 1, can be witten t 1 Ŷ t = µ t + ( φ ts )(Y s µ s )+ d t ɛ t, (1) s=1 whee ɛ t N (0, 1), the φ ts ae the (sub-diagonal) elements of T and the d t ae the diagonal elements of D Pan and MacKenzie (200) exploited the modified Cholesky decomposition to jointly model the mean and covaiance in longitudinal studies Pouahmadi et al (2007) developed a method of simultaneously modelling seveal covaiance matices using this decomposition; this wok gives an altenative to common pincipal components analysis (Fluy, 1988) fo longitudinal data In Section 2, we develop a model-based clusteing famewok fo longitudinal data by using Gaussian mixtue models whee the modified Cholesky decomposition of the goup covaiance matices ae constained in ode to give pasimonious models The mixtue models ae fitted using an EM algoithm (Dempste et al, 1977), as outlined in Section 22 The models ae applied to time couse gene expession data in Section, whee they exhibit good clusteing pefomance In Section 4, the stuctue of the lowe tiangula matix is exploited to extend this family of models to allow fo situations whee only autocoelations up to lag d ae equied This extension of the family of models gives ise to moe pasimonious models The extended family of models is then applied to a data set on the weight of ats on one of thee diffeent dietay supplements, whee one of the extended models is chosen The esults of this wok ae summaized in Section 5 2 GAUSSIAN MIXTURE MODELS WITH CHOLESKY-DECOMPOSED COVARIANCE STRUCTURE 21 The model We assume a Gaussian mixtue model, with a modified Cholesky-decomposed covaiance stuctue, fo each mixtue component Theefoe, the density of an obsevation x i in goup g is given by 1 f(x i µ g,t g,d g )= { (2π)p D g exp 1 } 2 (x i µ g ) T g D 1 g T g(x i µ g ), whee T g is the p p lowe tiangula matix and D g is the p p diagonal matix that follow fom the modified Cholesky decomposition of Σ g Now, thee is the option to constain the T g o the D g to be equal acoss goups and thee is also the option to impose the isotopic constaint D g = δ g I p (cf Tipping and Bishop, 1999), which leads to a family of eight Gaussian mixtue models Each membe of this family, along with thei espective nomenclatue and numbe of covaiance paametes, is given in Table 1 The nomenclatue is quite intuitive; fo example, the VEA model has vaiable autoegessive stuctue and equal, anisotopic noise acoss goups Constaining the T g to be equal acoss goups suggests that the coelation stuctue of the longitudinally ecoded data values is the same fo all of the goups In this context, the coelation stuctue eflects the autoegessive elationship between time points as outlined in Equation 1 Imposing the constaint that the D g ae equal acoss goups suggests that the vaiability at each time point is the same fo each goup and imposing the isotopic constaint D g = δ g I p suggests that the vaiability is the same at all time points Fo each given data set, any of the eight combinations of these constaints given in Table 1 might be most appopiate Two of the models given in Table 1, EEA and VVA, ae equivalent, fom a clusteing viewpoint, to models that aleady exist within the MCLUST famewok Howeve, the MCLUST covaiance stuctue does not explicitly account fo the longitudinal coelation stuctue and so the models

5 Table 1: The nomenclatue, covaiance stuctue and numbe of covaiance paametes fo each model Id Model T g D g D g Numbe of Covaiance Paametes 1 EEA Equal Equal Anisotopic p(p 1)/2+p 2 VVA Vaiable Vaiable Anisotopic G[p(p 1)/2] + Gp VEA Vaiable Equal Anisotopic G[p(p 1)/2] + p 4 EVA Equal Vaiable Anisotopic p(p 1)/2+Gp 5 VVI Vaiable Vaiable Isotopic G[p(p 1)/2] + G 6 VEI Vaiable Equal Isotopic G[p(p 1)/2] EVI Equal Vaiable Isotopic p(p 1)/2+G 8 EEI Equal Equal Isotopic p(p 1)/2 + 1 intoduced heein ae moe natual fo longitudinal data Futhe, these models will give infomation about the natue of the covaiance stuctue specifically, egading the autoegessive stuctue and the innovation vaiances that will not aise fom MCLUST 22 Model fitting The models ae fitted using an EM algoithm The missing data ae taken to be the goup membeship labels, which we denote z, whee z ig = 1 if obsevation i is in goup g and z ig =0 othewise Combining the missing data z with the known data x, gives the complete-data (x, z) The complete-data likelihood fo the mixtue model is given by L c (π g,µ g,t g,d g )= n i=1 G [π g f(x i µ g,t g,d g )] zig, and the expected value of the complete-data log-likelihood fo the mixtue model is Q(π g,µ g,t g,d g )= G n g log π g np 2 log 2π G whee the z ig have been eplaced by thei expected values ẑ ig = n g 2 log D g ˆπ g f(x i ˆµ g, ˆT g, ˆD g ) G h=1 ˆπ hf(x i ˆµ h, ˆT h, ˆD h ), G n g 2 t{ T g S g T gd g 1 }, (2) n g = n i=1 ẑig and S g = (1/n g ) n i=1 ẑig(x i µ g )(x i µ g ) Now, maximising Q with espect to π g and µ g gives ˆµ g = n i=1 ẑigx i / n i=1 ẑig and ˆπ g = n g /n, espectively The paamete estimates fo T g and D g ae also deived by maximizing Q and these depend on the constaints (Table 1) used in the model The paamete estimates fom the M-step of the VVI model ae deived in Section 2 Aside fom the EVA and EVI models, estimates fo the paametes of the othe models ae aived at in a simila fashion and ae available fom the authos upon equest The deivation of estimates fo the EVA model ae given in the Appendix; the EVI estimation pocedue is simila to the EVA pocedue 4

6 2 Paamete Estimates fo the VVI Model Imposing the constaint D g = δ g I p and diffeentiating Equation 2 with espect to T g and δg 1 espectively gives the following scoe functions S 1 (T g,δ g )= Q(T g,δ g ) T g S 2 (T g,δ g )= Q(T g,δ g ) δg 1 ) = n g δ g T g S g = n g ( T g Sg + S g 2δ g = n g ( pδg t { T g S g T }) g 2 Only the lowe tiangula pat of T g needs to be estimated, so we need to solve the system of equations given by the lowe tiangle of S 1 ( ˆT g,δ g ) = 0 Using the notation of Pouahmadi et al (2007), let φ (g) ij epesent those elements of T g that ae to be estimated, so that T g = φ (g) φ (g) 1 φ (g) φ (g) p 1,2 φ (g) p 1,p φ (g) p1 φ (g) p2 φ (g) p,p 2 φ (g) p,p 1 1 φ (g) p 1,1, () and wite S 1 (T g,δ g ) S 1 (Φ g,δ g ), whee Φ g = {φ (g) ij } fo i > j and i, j {1,, p} Also, let LT{ } denote the lowe tiangula pat of a matix Now, solving LT { S 1 (ˆΦ g,δ g ) } = 0 fo ˆΦ g leads to a total of p 1 systems of linea equations and the solution to each of these equations can be witten 1 2, 1 = , ,2 1, 1 2, 2 1, , 1 fo =2,, p Solving diag{s 2 (ˆΦ g, ˆδ g )} = 0 fo ˆδ g, gives ˆδ g = (1/p) t { ˆTg S g ˆT g } ANALYSES 1 Convegence citeion The Aitken acceleation was used to povide an asymptotic estimate of the log-likelihood at each iteation and this estimate was then used to detemine the convegence of each EM algoithm The Aitken acceleation at iteation m is given by a (m) = l(m+1) l (m) l (m) l (m 1), whee l (m+1), l (m) and l (m 1) ae the log-likelihood values fom iteations m + 1, m and m 1 espectively Then, the asymptotic estimate of the log-likelihood at iteation m + 1 is given by l (m+1) = l (m) a (m) (l(m+1) l (m) ), 5

7 (Böhning et al, 1994) and the EM algoithm can be said to have conveged when l (m+1) l (m+1) <ɛ (Lindsay, 1995) o when l (m+1) l (m) <ɛ (McNicholas et al, 2010) The latte citeion has the advantage that it is necessaily at least as stict as the lack of pogess l (m+1) l (m) <ɛ 2 Model selection The Bayesian infomation citeion (BIC; Schwatz, 1978) is used to select the best membe of this family of Gaussian mixtue models The BIC can be witten BIC =2l(x, ˆθ) ρ log N, whee l(x, ˆθ) is the maximized log-likelihood, ˆθ is the maximum likelihood estimate of θ, ρ is the numbe of fee paametes in the model and N is the numbe of obsevations The use of the BIC fo mixtue model selection is justified by Keibin (1998, 2000), who shows that it gives consistent estimates of the numbe of components in a mixtue model, unde cetain egulatoy conditions Futhemoe, Faley and Raftey (1998, 2002) and McNicholas and Muphy (2008) give pactical evidence that the BIC is effective as a model selection citeion fo Gaussian mixtue models Yeast Spoulation Time Couse Data Spoulation is a pocess by which diploid cells of budding yeast give ise to haploid cells Chu et al (1998) measue changes in gene expession duing spoulation using 97% of yeast genes; thei study used 6118 gene expessions that wee measued ove seven time points t {0, 05, 20, 50, 70, 90, 115} The ole of clusteing in such time couse analyses is impotant since the objective is to find goups of genes that expess similaly ove the couse of the expeiment Genes with simila expessions ae said to be co-expessed Cetain genes ae known to have specific functions and when othe genes ae found to co-expess with these genes, new insight can be gained into thei function Chu et al (1998) eliminated about 80% of the genes pio to thei analysis by focusing on the genes that showed obvious changes in expession and esticting themselves to genes that wee induced (showed inceased expession) Mitchell (1994) had peviously suggested that thee wee fou tempoal classes of these genes Chu et al (1998) contended that fou goups was not sufficient to epesent the divesity of the obseved expession pattens and they chose seven tempoal pattens that seemed appopiate These pattens wee chosen by eye A total of thity genes (Table 2) wee selected as epesentative of these seven pattens; these ae known as model expession pofiles Then the emaining genes wee clusteed into the seven goups based on thei coelation with these model pofiles Wakefield et al (200) used a fou-stage Bayesian hieachical model to analyze this time couse data Befoe applying the hieachical model, Wakefield et al (200) used Bayes factos to educe the numbe of genes fom 6118 to 1104, conceding that thei modelling paadigm may be computationally pohibitive fo a lage numbe of genes They found that the numbe of tempoal classes was pobably between 11 and 14, with G = 12 being the most pobable On compaison of thei G = 12 component model to the model pofiles (40 genes) of Chu et al (1998), they concluded that thee model offeed new insights into co-expession Note that, while Chu et al (1998) consideed seven time points, Wakefield et al (200) consideed six time points, dopping t = 0 and and taking the values at each othe time elative to time t = 0 fo each gene In ou analyses of these data, all seven time points wee consideed and no genes wee deleted As is common, the negative logaithm, base 2, of each obsevation was taken and then each time point was standadized to have mean zeo and vaiance one pio to the analysis Then the novel model-based clusteing technique intoduced heein was applied to the these data fo G =1,, 20 using five andom stating values fo ẑ fo each of the eight models and each value of G The BIC fo each model and each of the 20 values of G is depicted in Figue 1; aside fom a few of the ealie G values, fo which the VVA model was chosen, the EVA model was selected Fom Figue 1, it is appaent that this family of models suggest that the tue numbe of goups in the spoulation data is somewhee in the ealy to mid teens Moe pecisely, the best model 6

8 BIC G Figue 1: A plot of BIC values vesus numbe of goups fo all eight models was a EVA model with G = 1 and a BIC of Selection of the EVA model indicates that, while the autoegessive stuctue of the data, as suggested by the T matix, is the same acoss goups (tempoal pattens), the innovation vaiances diffe both between tempoal pattens and between times A coss tabulation (Table 2) of the cluste membeship and the model pofiles of Chu et al (1998) shows that thee is some coespondence fo these 40 genes; the Late goup coesponds pefectly and some of the late goups ae vey simila This is what one would expect since genes that wee induced late would be the easiest to spot by eye Howeve, thee ae notable diffeences in estimated co-expessions fo the ealie goups Note that thee ae only eight columns in Table 2, despite the fact that the best model in ou analysis had G = 1 goups This is because ou model is based on all 6,118 genes and the 40 model pofiles of Chu et al (1998) appeaed in just eight of ou 1 goups Table 2: Fequencies of the 40 genes selected as model pofiles by Chu et al (1988), coss-tabulated by the seven tempoal pattens suggested by Chu et al (1998) and by ou tempoal pattens (A H) Goup A Goup B Goup C Goup D Goup E Goup F Goup G Goup H Metabolic 4 2 Ealy 1 5 Ealy Ealy-Mid 4 1 Middle 4 Mid-Late Late 4 7

9 This analysis pesents new insight into these data by poviding 1 distinct tempoal pattens, based all 6118 genes This insight is quite diffeent fom the esults of Wakefield et al (200), who split 1104 of the 6118 genes into twelve tempoal pattens The lagest goup in ou analysis had 180 genes and so it is cetainly not the case that ou goups ae just the twelve of Wakefield et al (200) plus a noise goup containing the 5014 genes that Wakefield et al (200) deleted As mentioned in Section 1, MCLUST is the most well-established Gaussian model-based clusteing technique within the liteatue In ode to illustate the usefulness of the novel technique intoduced heein, elative to existing methods, the MCLUST family of models was also used to analyze these data The data was pepocessed in the same fashion, by taking negative logaithms and standadizing, and the mclust softwae fo R was used All ten MCLUST models wee un fo G =1,, 20 and the best model was a VVV model with six components This model, which is equivalent to the VVA model fom Table 1, had a BIC of , which is notably less than the BIC fo the G = 1 component EVA model ( ) Fom Table, it is appaent that thee is no coespondence between the model pofiles of Chu et al (1998) and the MCLUST tempoal pattens As mentioned ealie, one would expect to see a coespondence in some of the late model pofiles The coespondence between the MCLUST esults and the method intoduced heein is given in Table 4: it is appaent that ou EVA model is not simply splitting the MCLUST goups but is suggesting a substantially diffeent suctue Table : Fequencies of the 40 genes selected as model pofiles by Chu et al (1988), coss-tabulated by the tempoal pattens suggested by Chu et al (1998) and by the MCLUST pattens (I IV) Goup I Goup II Goup III Goup IV Metabolic 6 Ealy 1 5 Ealy Ealy-Mid 5 Middle 4 Mid-Late 1 2 Late 4 4 CONSTRAINING SUB-DIAGONALS 41 Intoduction Duing the analysis of the yeast spoulation data it was noted that many of the values below the fist sub-diagonal of the estimated T matix (Equation 4) wee small In fact, all but one of the elements below the fist sub-diagonal could be consideed small; T = (4) 8

10 Table 4: Fequencies of all 6,118 genes, coss-tabulated by ou tempoal pattens (A M) and by the MCLUST pattens (I VI) A B C D E F G H I J K L M Goup I Goup II Goup II Goup IV Goup V Goup VI While it is difficult to detemine if a value in T is small without taking the values of the D g in context, it led to the notion of setting vaious sub-diagonals of T g to zeo and hence the possibility of a moe pasimonious class of models This constained coelation stuctue emoves any autocoelation ove lage time lags; that is, T g constained to d sub-diagonals implies an ode d autoegessive stuctue within the famewok of Equation 1 In this section we deive paamete estimates when cetain sub-diagonals of T g ae set to zeo 42 Constaints & nomenclatue We constain the elements of T g to be zeo below a given numbe of sub-diagonals The notation V 1 VA is used to denote the VVA model whee the elements of T g ae zeo below the fist subdiagonal, V 2 VA denotes the VVA model whee the elements of T g ae zeo below the second sub-diagonal, and so foth Note that, although not used heein, models whee all sub-diagonal elements ae zeo, such as V 0 VA, ae equivalent to the diagonal MCLUST models Woking out paamete estimates when only the fist sub-diagonal of T g is non-zeo is tivial Fo example, fom the computations of Section 2 it is clea that the paamete estimates fo T g (g) in the M-step of the V 1 VI model will be ˆφ, 1 = s(g), 1 /s(g) 1, 1, fo =2,, p 4 Paamete estimates fo the V 2 VA and V d VA models Diffeentiating Equation 2 with espect to T g and Dg 1, espectively, gives the scoe functions S 1 (T g,d g )= n g Dg 1T gs g and S 2 (T g,d g )=n g /2 ( D g T g S g T g) Using the familia notation, we can wite T g as in Equation but with zeos below the second sub diagonal The notation SD { } is { used heetofoe to denote the fist sub-diagonals of a matix Now, solving SD 2 S1 (ˆΦ g,d g ) } =0 fo ˆΦ g leads to a total of p 1 systems of linea equations, all but one of which is 2 2 This one (g) exception is 1 1, which gives the familia solution ˆφ 21 = s(g) 21 /s(g) 11 The solutions in the 2 2 cases ae given by ( ˆφ(g), 2, 1 ) = ( 2, 2 2, 1 1, 2 1, 1 ) 1 (, 2, 1 fo =,, p and solving diag { S 2 (ˆΦ g, ˆD g ) } = 0 fo ˆD g, gives ˆD g = diag { ˆTg S g ˆT g } ), 9

11 The paamete estimates fo T g and D g can be genealized to the V d VA case Using the same notation, the ˆΦ g ae given by, d, (d 1), 1 = d, d d, (d 1) d, 1 (d 1), d 1, d (d 1), (d 1) 1, (d 1) (d 1), 1 1, 1 1, d, (d 1), 1 fo =2,, p and, once again, ˆDg = diag { ˆTg S g ˆT g } Paamete estimates in the othe cases fo this geneal constaint ae simila except in the EVA and EVI cases Estimates fo the EVA case ae given in the Appendix and those fo the EVI case ae vey simila 44 Application to yeast spoulation data The eight models intoduced heein (Table 1) wee applied to the yeast spoulation data with all elements of T g below the fist and second sub-diagonals, espectively, set to zeo (d =1,d = 2) The esult of this analysis was that the EVA model, with full T (d = 6), was still the best model The best of these two constained models fo G = 1 was the E 1 VA model, with BIC= The E d VA model was then fitted to these data fo d =, 4, 5 The best of these models was the E 5 VA with BIC Theefoe, the best model was still the full EVA model Inteestingly, the full model being bette than the E 5 VA model indicates that coelation pesists acoss all time points including time zeo; as peviously mentioned, this time point was dopped in the analysis of Wakefield et al (200) 45 Application to ats data In ode to show that a model with constaints imposed on the sub-diagonals of T g will sometimes be selected, the following analysis is pesented Data on the body weights of ats on one of thee diffeent dietay supplements wee souced fom the nlme package (Pinheio et al, 2008) fo the R softwae These data wee published in Cowde and Hand (1990) and have been analyzed many times: see Hand and Cowde (1996) and Haslett (1997) fo examples They ae used solely fo illustative puposes hee and what follows is not intended to be an in-depth analysis of these data Fo one thing, we make no attempt to model the component means which one may do by allowing fo a systematic tend, fo example A total of 16 ats wee put on one of thee diffeent diets; eight ats wee on Diet 1, fou wee put on Diet 2 and fou on Diet Weights wee fist ecoded afte a settling-in peiod and then weekly fo a peiod of nine weeks An exta measuement was taken at 44 days to help gauge the effect of a teatment that occued duing the sixth week These 11 measuements can be seen on the time seies plot in Figue 2 Fom this figue, it is clea that the ats ae gouped by weight, with two exceptions: a heavie at on Diet 2 and a lighte at on Diet Although the tue diets ae known, we teat this as a clusteing poblem and so assume no pio knowledge of component membeship The eight models in Table 1 wee fitted to these data fo G =1, 2,, 6 The best model was an EEA model with G = 5 This model put the two exceptions into goups on thei own and all othe ats wee coectly classified Selection of an EEA model suggests that the covaiance stuctue is the same fo each goup and so the classification is effectively based on the mean Futhe, the fact that the isotopic constaint was not imposed implies that the innovation vaiance is not the same at each time point To illustate that a model with constaints imposed on the sub-diagonals of T g will sometimes be selected, the E d EA models wee fitted to these data fo d =1, 2,, 10, whee the E 10 EA model is equivalent to the full EEA model The BIC fo each model is given in Table 5 and the, 10

12 Weight (g) Time (days) Figue 2: Time seies fo each at, by diet: Diet 1 (solid lines), Diet 2 (dashed) and Diet (dotted) best model was the E 8 EA model, which has a T matix with eight non-zeo sub-diagonals The estimated cluste membeships wee identical to those fo the EEA model Table 5: BIC values fo the E i EA models fitted to the ats data fo i =1, 2,, 10 Model BIC Model BIC E 1EA E 6EA 527 E 2EA E 7EA 5691 E EA E 8EA E 4EA 5047 E 9EA E 5EA E 10EA SUMMARY A new model-based clusteing technique, using Gaussian mixtue models, has been intoduced fo the analysis of longitudinal data This family of mixtue models follows the classical appoach whee each membe of the family has diffeent constaints imposed on the modified Cholesky-decomposed covaiance stuctue Initially, eight membes of this family wee given and the associated maximum likelihood estimates fo thei paametes wee deived using an EM algoithm These models povided new insight when applied to yeast spoulation time couse data Futhemoe, by constaining cetain sub-diagonals of T g to be zeo, the numbe of membes of this family of models was geatly inceased, including membes with moe pasimonious covaiance stuctues The family of models offes much scope fo futhe expansion by constaining subsets of the T matix, which captue the coelation stuctue of the data Initial extensions constaining sub- 11

13 diagonals of the T matix wee given in this wok, focusing on adjacent sub-diagonals The exta models that wee obtained as a esult of this extension whee shown to be useful when applied to a well known data set on the weight of ats Futue wok will focus on constaining diffeent combinations of the sub-diagonals of T thee ae 2 p 1 possible combinations and on the incopoation of missing data into the modelling famewok APPENDIX Paamete estimates fo the EVA model when T g is not constained Fo the EVA model, T g = T and so, fom Equation 2, the expected value of the complete-data log-likelihood Q can be witten Q(T, D g )=C G n g 2 log D g espectively gives the following scoe func- Diffeentiating Equation 5 with espect to T g and Dg 1 tions, S 1 (T, D g )= Q(T, D g) T = G G n g 2 t { TS g T Dg 1 } (5) n g Dg 1 TS g, and S 2 (T, D g )= Q(T, D g) Dg 1 = n g 2 (D g TS g T ) Now, solving LT { S 1 ( ˆT, D g ) } S 1 (ˆΦ,D g ) = 0 fo ˆΦ leads again to a total of p 1 systems of linea equations The solution fo the fist ow of the lowe tiangle of T g is, G n g [ 11 ˆφ 21 ˆd (g) 22 ] + s(g) 21 ˆd (g) 22 G n g =0, and so ˆφ21 = [ G n g Fo convenience, we intoduce the notation κ ij m = [ G π g ij [ 21 ˆd (g) ˆd (g) 22 ] [ G π g ] = [ G π g 21 ˆd (g) ˆd (g) 22 ] ] (6) ] (g) / ˆd mm, so that Equation 6 can be witten ˆφ 21 = κ 21 2 /κ 11 2 Now, solving the second ow means solving the linea system ( κ 11 κ 21 κ 12 κ 22 )( ) ( ˆφ1 κ 1 = ˆφ 2 κ 2, 1 ), and so, ( ˆφ1 ˆφ 2 ) ( κ 11 = κ 21 κ 12 κ 22 It follows that the solution to the ( 1)st system of equations is given by ˆφ 1 κ 11 κ 21 κ 1,1 1 ˆφ 2 = κ 12 κ 22 κ 1,2 κ 1, 1 κ 2, 2 κ 1, 1 κ 1 κ 2 κ, 1 ) 1 ( κ 1 κ 2 ), (7) fo =2,, p Note that κ ij m = κ ji m and so the ( 1) ( 1) matix in Equation 7 is symmetic Solving the second scoe function, diag { S 2 ( ˆT g, ˆD g ) } = 0, gives ˆD g = diag { ˆTSg ˆT } Paamete estimates when sub-diagonals of T g ae constained In most of the cases whee subdiagonals ae set to zeo, the solutions ae vey simila to two and d-sub-diagonal cases detailed in Section 4, and so they ae not given hee Howeve, in the EVA and EVI cases, the solutions ae a little moe involved than the othe cases and so the deivations of the paametes in the M-step fo the E 2 VA and E d VA cases ae povided in full The coesponding estimates fo the EVI model 12

14 ae simila Recall that diffeentiating Equation 5 with espect to T g and Dg 1 espectively gives the following scoe functions, S 1 (T, D g )= Q(T, D g) T = G n g Dg 1 TS g, and S 2 (T, D g )= Q(T, D g) Dg 1 = n g 2 (D g TS g T ) { Fo model E 2 VA, solving SD 2 S1 ( ˆT, D g ) } = 0 fo ˆΦ leads again to a total of p 1 systems of linea equations, all but one of which is 1 1 The solution fo the 1 1 system is ˆφ 21 = κ 21 2 /κ11 2 and solving the emaining systems gives the solution ( ) ( ˆφ, 2 κ 2, 2 = ˆφ, 1 κ 2, 1 κ 1, 2 κ 1, 1 ) 1 ( κ, 2 κ, 1 ), fo =,, p Now, these estimates can be extended to the E d VA case as follows;, d, (d 1), 1 = κ d, d κ d, (d 1) κ d, 1 κ (d 1), d κ 1, d κ (d 1), (d 1) κ 1, (d 1) κ (d 1), 1 κ 1, 1 1 κ, d κ, (d 1) κ, 1, fo =2,, p Fo any d, solving diag { S 2 ( ˆT g, ˆD g ) } = 0 fo ˆD g, gives ˆD g = diag { ˆTSg ˆT } ACKNOWLEDGEMENTS This eseach was suppoted by a Discovey Gant fom the Natual Sciences and Engineeing Reseach Council of Canada and a Basic Reseach Gant fom Science Foundation Ieland The authos ae gateful to an Associate Edito and two efeees fo thei helpful and insightful suggestions REFERENCES J D Banfield & A E Raftey (199) Model-based Gaussian and non-gaussian clusteing Biometics, 49, D Böhning, E Dietz, R Schaub, P Schlattmann & B Lindsay (1994) The distibution of the likelihood atio fo mixtues of densities fom the one-paamete exponential family Annals of the Institute of Statistical Mathematics, 46, 7 88 C Bouveyon, S Giad & C Schmid (2007) High-dimensional data clusteing Computational Statistics and Data Analysis, 52, G Celeux & G Govaet (1995) Gaussian pasimonious clusteing models Patten Recognition, 28, S Chu, J DeRisi, M Eisen, J Mulholland, D Botstein, P Bown & I Heskowitz (1998) The tansciptional pogam of spoulation in budding yeast Science, 282, M J Cowde and D J Hand (1990) Analysis of Repeated Measues Chapman and Hall, London R De la Cuz-Mesía, F A Quintana & G Mashall (2008) Model-based clusteing fo longitudinal data Computational Statistics and Data Analysis, 52,

15 A P Dempste, N M Laid & D B Rubin (1977) Maximum likelihood fom incomplete data via the EM algoithm Jounal of the Royal Statistical Society Seies B, 9, 1 8 P J Diggle, K-Y Liang & S L Zege (1994) Analysis of Longitudinal Data Oxfod Univesity Pess, Oxfod B Eveitt (1995) The analysis of epeated measues: A pactical eview with examples The Statistician, 44, B Fluy (1988) Common Pincipal Components and Related Multivaiate Models Wiley, New Yok C Faley & A E Raftey (1998) How many clustes? Which clusteing methods? Answes via modelbased cluste analysis The Compute Jounal, 41, C Faley & A E Raftey (2002) Model-based clusteing, disciminant analysis, and density estimation Jounal of the Ameican Statistical Association, 97, C Faley & A E Raftey (200) Enhanced softwae fo model-based clusteing, density estimation, and disciminant analysis: MCLUST Jounal of Classification, 20, C Faley & A E Raftey (2006) MCLUST vesion fo R: Nomal mixtue modeling and model-based clusteing Technical Repot 504, Depatment of Statistics, Univesity of Washington Z Ghahamani & G E Hinton (1997) The EM algoithm fo facto analyzes Technical Repot CRG- TR-96-1, Univesity Of Toonto D J Hand & M J Cowde (1996) Pactical Longitudinal Data Analysis Chapman and Hall, London J Haslett (1997) Conditional expectations and esidual analysis fo the linea models Applied Stochastic Models and Data Analysis, 1, M G Kenwad (1987) A method fo compaing pofiles of epeated measuements Jounal of the Royal Statistical Society Seies C, 6, C Keibin (1998) Estimation consistante de l ode de modèles de mélange Comptes Rendus de l Académie des Sciences Séie I Mathématique, 26, C Keibin (2000) Consistent estimation of the ode of mixtue models Sankhyā The Indian Jounal of Statistics Seies A, 62, W J Kzanowski, P Jonathan, W V McCathy & M R Thomas (1995) Disciminant analysis with singula covaiance matices: Methods and applications to spectoscopic data Jounal of the Royal Statistical Society Seies C, 44, B G Lindsay (1995) Mixtue Models: Theoy, Geomety and Applications, volume 5 of NSF-CBMS Regional Confeence Seies in Pobability and Statistics Institute of Mathematical Statistics, Haywad, Califonia G J McLachlan, D Peel & R W Bean (200) Modelling high-dimensional data by mixtues of facto analyzes Computational Statistics and Data Analysis, 41, P D McNicholas & T B Muphy (2008) Pasimonious Gaussian mixtue models Statistics and Computing, 18, P D McNicholas, T B Muphy, A F McDaid & D Fost (2010) Seial and paallel implementations of model-based clusteing via pasimonious Gaussian mixtue models Computational Statistics and Data Analysis In pess, doi: /jcsda

16 A P Mitchell (1994) Contol of meiotic gene expession in sacchaomyces ceevisiae Micobiology and Molecula Biology Reviews, 58, J Pan & G MacKenzie (200) On modelling mean-covaiance stuctues in longitudinal studies Biometika, 90, D K Paule & N M Laid (2000) A mixtue model fo longitudinal data with application to assessment of noncompliance Biometics, 56, J Pinheio, D Bates, S DebRoy, D Saka & the R Coe team (2008) nlme: Linea and Nonlinea Mixed Effects Models R package vesion 1-89 M Pouahmadi (1999) Joint mean-covaiance models with applications to longitudinal data: Unconstained paameteisation Biometika, 86, M Pouahmadi (2000) Maximum likelihood estimation of genealised linea models fo multivaiate nomal covaiance matix Biometika, 87, M Pouahmadi, M Daniels & T Pak (2007) Simultaneous modelling of the Cholesky decomposition of seveal covaiance matices Jounal of Multivaiate Analysis, 98, R Development Coe Team (2009) R: A Language and Envionment fo Statistical Computing R Foundation fo Statistical Computing, Vienna, Austia G Schwatz (1978) Estimating the dimension of a model Annals of Statistics, 6, 1 8 T E Tipping & C M Bishop (1999) Mixtues of pobabilistic pincipal component analyses Neual Computation, 11, J C Wakefield, C Zhou & S G Self (200) Modelling gene expession ove time: Cuve clusteing with infomative pio distibutions In J M Benado, M J Bayai, J O Bege, A P Dawid, D Heckeman, A F M Smith & M West, editos, Bayesian Statistics, volume 7, pages Oxfod Univesity Pess, Oxfod Received??? Accepted??? Paul D McNICHOLAS: pmcnicho@uoguelphca Depatment of Mathematics & Statistics, Univesity of Guelph Guelph, Ontaio Canada, N1G 2W1 Thomas Bendan MURPHY: bendanmuphy@ucdie School of Mathematical Sciences, Univesity College Dublin Belfield, Dublin 4 Ieland 15