ON THUSTONE'S MODEL FO PAIED COMPAISONS AND ANKING DATA Alber Maydeu-Olivares Dep. of Psychology. Universiy of Barcelona. Paseo Valle de Hebrón, 171. 08035 Barcelona (Spain). Summary. We invesigae by means of a simulaion sudy wheher sandard srucural equaion modeling echniques are suiable for esimaing an unresriced Thursonian model o: (a) muliple judgmen paired comparisons daa, and (b) ranking daa. We poin ou ha Thursone's (197) original model is no a proper model for muliple judgmen paired comparisons daa as i assigns ero probabiliy o all inransiive paired comparisons paerns. To fi muliple judgmen paired comparisons daa one mus employ Takane's (1987) exension of Thursone's original model, or alernaively, a Thursonian correlaion srucure model, which we inroduce. We found ha for some models as few as 100 observaions suffice o obain accurae parameer esimaes, sandard errors and goodness of fi ess when 7 simuli are modeled (1 binary variables). For oher models, however, sample sies of 1000 observaions are needed. All in all, he procedure invesigaed appears o be an aracive choice o esimae hese models. Keywords: limied informaion, comparaive daa, random uiliy models. 1. Inroducion In 197 Thursone suggesed a class of models for fiing paired comparison daa ha has been highly influenial in he lieraure (see Bock & Jones, 1968). In 1931 Thursone suggesed his model could also be suiable for ranking daa (afer rankings are ransformed ino paired comparisons). Thursone's model is simply a mulivariae normal densiy wih an srucured mean vecor and covariance marix ha has been dichoomied. Thus, i is somewha naural o consider is esimaion using exising procedures for srucural equaion modeling for dichoomous variables (Muhén, 1978, 1993). In his paper we invesigae how well an unresriced Thursonian model for paired comparisons and ranking daa can be esimaed using srucural equaion modeling procedures for dichoomous variables. To do so, we firs inroduce Thursone's original model using marix noaion. Using his noaion, i is easy o see ha Thursone's original model is a proper model for ranking daa bu i is no a proper model for muliple judgmen paired comparisons daa. Nex, we 519
inroduce wo modificaions of Thursone's model ha address his problem. We conclude our presenaion wih some simulaion resuls o illusrae he performance of srucural equaion modeling procedures o esimae hese models.. Thursone's model Thursone (197) inroduced a model for paired comparisons characeried by hree assumpions: 1. Whenever a pair of simuli is presened o a subjec i elicis a coninuous preference (uiliy funcion, or in Thursone's erminology, discriminal process) for each simulus.. The simulus whose value is larger a he momen of he comparison will be preferred by he subjec. 3. These unobserved preferences are normally disribued in he populaion. We assume each individual from a random sample responds o all possible n( n 1) n = paired comparisons, where n denoes he number of simuli being compared. This has been ermed muliple judgmen sampling procedure (Bock & Jones, 1968). Thus, we obain a ñ-dimensional vecor of binary variables y from each responden such ha 1 if objec i is chosen yi, j= (1) 0 if objec j is chosen for all pairs {i, j}. Then, leing denoe he n-dimensional vecor of coninuous ~ N µ, Σ, we wrie preferences in Thursone's model such ha ( ) y * = A, () where A is a ñ n design marix where each column corresponds o one of he simuli, and each row o one of he paired comparisons. For example, when n = 4, A is Finally, 1 1 0 0 1 0 1 0 1 0 0 1 A =. (3) 0 1 1 0 0 1 0 1 0 0 1 1 50
y i, j 1 if yi, j 0 = 0 if yi, j< 0 (4) These equaions describe in marix noaion Thursone's (197) model applied o muliple judgmen sampling paired comparisons daa. To analye ranking daa, Thursone (1931) suggesed ransforming each ranking paern ino a vecor of ñ binary variables using 1 if objec i is ranked above objec j yi, j= (5) 0 if objec i is ranked below objec j and applying his model for paired comparisons daa o hese daa. Under Thursone's model he probabiliy of any paired comparisons paern is ( y µ * Σ * ) * * Pr yi, j = φn :, dy (6) y y i, j µ = A µ y * Σ * = AΣ y A (7) where φn ( ) denoes a ñ-variae normal densiy funcion and is he mulidimensional recangular region formed by he produc of inervals i, j ( ) ( ) 0, if yi, j= 1 =. (8),0 if yi, j = 0 We can perform a change of variable of inegraion in (6) and sandardie y * using * * = D( y µ y * ) = ( Diag ( Σ * y )) As a resul, µ * = 0 and τ = µ = µ y * 1 D. (9) D DA Ρ = Σ = ( Σ ) D D D A A D (10) * * y where τ denoes a vecor of hresholds τ i,j, and he off-diagonal elemens of Ρ * are erachoric correlaions. As a resul, (6), can be equivalenly rewrien as where inervals * * Pr yi, j = n( :, Ρ * φ 0 ) d (11) i, j is now a mulidimensional recangular region formed by he produc of 51
i, j ( τ i j ) ( τ i j),, if yi, j= 1 =. (1),, if yi, j = 0 Equaions (10), (11), and (1) define in fac a class of models as µ and Σ can be resriced in various ways. Takane (1987) provides an excellen overview of resriced Thursonian models. Here we shall concenrae on he unresriced Thursonian model, a model where only minimal idenificaion resricions are imposed on µ and Σ. A very ineresing feaure of Thursone's original model is ha since A is of rank n - 1, Σ * has rank n 1. As a resul, Thursone's model assigns ero probabiliies y o all he ñ - n! inransiive paired comparisons paerns (hose paerns ha do no correspond o ranking paerns). Thus, Thursone's model is no a plausible model for muliple judgmen paired comparisons daa, bu i may be a suiable model for ranking daa (Maydeu-Olivares, 1999). 3. Thursonian models appropriae for muliple judgmen paired comparisons We now describe wo differen soluions o he problem of specifying a Thursonian model ha assigns non-ero probabiliies o all ñ binary paerns. 3.1 The Thursone-Takane model Takane (1987) proposed adding a random error o each paired comparison so ha * = + y A e. Furhermore, he assumed ha N µ, Σ 0 e 0 0 Ω (13) where Ω is a diagonal marix wih elemens ω i, j. Under his model, he probabiliy of any paired comparisons paern is given also by (6) and (8) bu now µ * = A µ y and Σ * = AΣ y A + Ω. Furhermore, performing he change of variable of inegraion (9), we can wrie hese paern probabiliies as (11) and (1) where insead of (10) we have = DA Ρ * = ( Σ + Ω ) τ µ D A A D (14) Takane (1987) also proposed an ineresing special case of his model in which i is assumed ha Ω = ω I. 5
3. Thursonian correlaion srucure models The Thursone and Thursone-Takane models are mean and covariance srucure models in he sense ha hey impose consrains on he mean vecor and covariance marix of y *. However, as only he binary choices are observed, he covariance marix of y * can only be esimaed from a correlaion marix. Moreover, o esimae he parameers of hese models one mus resor o pre and pos-muliply he covariance srucure by he inverse of a diagonal marix of model-based sandard deviaions. This resuls in complex non-linear resricions on he hresholds and erachoric correlaions. An alernaive soluion o he problem of specifying a proper model for muliple judgmen paired comparison daa wihin a Thursonian framework is o specify resricions on he means and correlaions of y * while leaving he srucure for he variances of y * unspecified. Now, he mean and covariance srucures for y * implied by Thursone's model are µ * = A µ y and Σ * = AΣ y A. Thus, we shall assume µ * = A µ y. Also, by parsimony we shall assume Ρ = Off ( AΣ ) y * A, where Off( ) denoes he resricions imposed on he off-diagonal elemens of a correlaion marix. Tha is, we assume ha he resricions on he correlaions among y * have he same funcional form as he resricions of Thursone's model on he covariances among y *. Then, he probabiliy of any paired comparisons paern under his Thursonian mean and correlaion srucure model is ( y µ * Ρ * ) n * * Pr yl = φn :, dy (15) y y l = 1 wih limis of inegraion (8). These paern probabiliies are unchanged when we ransform y * * * using = y µ y *. As a resul, µ * = 0 and = A Ρ ( Σ ) τ µ = Off A * A (16) where now he paern probabiliies are given by (11), (1) and (16). This model is more resricive han he Thursone-Takane model. However, unlike (14), he resricions in (16) are linear. 4. Limied informaion esimaion and esing We shall now discuss minimal idenificaion resricions for he models jus described. These minimal idenificaion resricions yield unresriced models. Given he comparaive naure of he daa, in all cases i is necessary o se he locaion of he elemens of µ and he locaion of he elemens in each of he rows (columns) of Σ. Arbirarily, we se µ n = 0 and Σ = Ρ (a marix wih ones along is diagonal). In addiion, in he Thursonian ranking model i is necessary o fix 53
one of he elemens of Ρ (Dansie, 1986). Arbirarily, we se ρ = nn, 1 0. Also, in he Thursone-Takane model, if Ω is assumed o be diagonal i is necessary o se he locaion for is elemens; arbirarily we se ωn = 1. Alernaively, if Ω = ω I is assumed, we impose he consrain ω = 1. Now, we collec he observed paired comparisons paerns in a ñ-way binary coningency able. We have seen ha he models under consideraion assume ha he coningency able arises by dichoomiing a ñ-dimensional mulivariae sandard normal densiy according o a se of hresholds while imposing consrains -given by Eq. (10), (14), and (16)- on he hresholds and erachoric correlaions. Muhén (1978, 1993) has proposed well esablished mehods for esimaing dichoomied mulivariae normal srucural models from he univariae and bivariae marginals of he coningency able. Here, we invesigae wheher Muhén's (1993) approach is suiable for hese ype of models by means of a simulaion sudy. In his esimaion procedure, firs each hreshold is esimaed separaely from he univariae marginals of he coningency able via. Then, each of he erachoric correlaions of he coningency able is esimaed separaely from he bivariae marginals of he coningency able given he esimaed hresholds. Finally, he model parameers are esimaed from he esimaed hresholds and erachoric correlaions using an unweighed leas squares (ULS) funcion. His mehod yields asympoically correc sandard errors and ess of he srucural resricions imposed by he model on he hresholds and erachoric correlaions. The laer are obained by adjusing he minimum of he ULS fiing funcion by is mean (T s ) or by is mean and variance (T a ). ecenly, Maydeu-Olivares (001) has proposed a similar adjusmen o he sum of squared residuals o he univariae and bivariae margins of he able, which gives us a limied informaion es of he overall resricions imposed by he model on he coningency able. The degrees of freedom available for esing when fiing paired comparisons models is n ( n + 1) r = q, where q is he number of model parameers. When fiing ranking daa i is necessary o adjus for he number of redundancies among he n univariae and bivariae marginals which arise from having n! srucural empy cells in he coningency able. The number of degrees of freedom in his n ( n + 1 n 1 ) x case is r = q (Maydeu-Olivares, 1999). x= We provide in Table 1 he simulaion resuls obained in esimaing a model for 7 simuli. Hence ñ = 1 binary variables were modeled. Sample sie was 500. The rue parameers were µ = ρ = ( 0.5,0, 0.5,0,0.5, 0.5,0 ), ( 0.8,0.7,0.6,0.8,0.7,0.6,,0.8,0.7,0.6) 54
where ρ denoes he elemens below he diagonal in Ρ. In he Thursone-Takane model where Ω was assumed o be diagonal, we used as rue values Ω = I. 1000 replicaions were used for each of he hree models esimaed. In his able we poin ou he larges relaive bias (in %) observed in he parameer esimaes and sandard errors. We also provide he empirical rejecion raes for he es saisics a α = 0.05. They should be as close as possible o 0.05. Finally, we also provide he value of D KS, he Kolmogorov-Smirnov one-sample es. The null hypohesis of a chi-square disribuion for he empirical disribuion of he es saisic is rejeced a α = 0.05 if D KS > 1.35. As can be seen in his able, for his paricular choice of rue generaing values and sample sie: 1. The performance of he ess saisics considered depends on he model. In all cases he mean and variance adjused es saisics of overall resricions mached well is reference chi-square disribuion as indicaed by D KS.. Accurae parameer esimaes and sandard errors (larges relaive bias < 10 %) were obained for all models bu for he Thursone-Takane model when Ω is assumed o be diagonal. Table 1. Simulaion resuls model maximum relaive bias (%) D KS saisic ( a α = 0.05) par. esimaes sandard errors srucur. ess overall ess µ Ρ Ω µ Ρ Ω T s T a T s T a (A) Thursone's <1 <1 - -3-4 - 3.9 (13.9).3 (4.) 7.9 (6.3) 1. (6.4) (B) TT, Ω diagonal -5 9-8 -17 1.0 (5.5) 1.8 (.) 7.8 (3.0) 0.8 (5.6) (C) TT, Ω = ω I 1-1 - -5 4-1. (6.1).1 (.4) 8.5 (5.) 0.9 (5.0) (D) correlaion sruc. <1 <1 - -1-6 -. (9.4). (3.8) 7.8 (4.6) 1.1 (4.3) In Thursone's model we fixed ρ 7,6 =.6 (he rue value) o calculae relaive bias Thursone's model was fied o ranking daa, he oher hree models o paired comparisons TT, Thursone-Takane model;, rejecion rae in %, should be close o 0.05. Addiional simulaion sudies wih hese rue generaing values reveal ha accurae parameer esimaes, sandard errors and goodness of fi ess can be obained for model (B) wih 1000 observaions, wih model (A) wih 300 observaions, and wih models (C) and (D) wih as few as 100 observaions. 55
5. Discussion and conclusions Classical esimaion procedures for Thursonian models (e.g., Bock & Jones, 1968) esimae he model parameers using only he univariae means of he binary variables. These procedures are clearly unsaisfacory because: 1. The sample means are no independen. Sandard errors and ess of he goodness of fi of he model compued under an independence assumpion will be incorrec.. Unresriced Thursonian models and many resriced models are no idenified from univariae informaion only. However, any Thursonian model can be idenified as soon as bivariae informaion is employed. We have shown ha using univariae and bivariae informaion i is possible o obain accurae parameer esimaes, sandard errors and ess of goodness of fi o Thursonian models for paired comparisons and ranking daa wih very small samples, even in large models. We have also poined ou ha Thursone's original model is suiable for ranking daa, bu ha i is no a proper model for muliple judgmen paired comparisons daa. We have discussed wo classes of classes of Thursonian models suiable for hese daa: a correlaion srucure model, and a covariance srucure model (he Thursone-Takane model). These wo models are no equivalen because he Thursone-Takane model is no scale invarian. Alhough he choice beween a Thursonian covariance vs. a correlaion srucure model should be subsanively moivaed, i is also imporan o consider model fi and esimaion issues in choosing beween hese models. eferences Bock,.D. & Jones, L.V. (1968). The measuremen and predicion of judgmen and choice. San Francisco: Holden-Day. Dansie, B.. (1986) Normal order saisics as permuaion probabiliy models. Applied Saisics, 35, 69-75. Maydeu-Olivares, A. (1999). Thursonian modeling of ranking daa via mean and covariance srucure analysis. Psychomerika, 64, 35-340. Maydeu-Olivares, A. (001). Limied informaion esimaion and esing of Thursonian models for paired comparison daa under muliple judgmen sampling. Psychomerika, 66, 09-8. Muhén, B. (1978). Conribuions o facor analysis of dichoomous variables. Psychomerika, 43, 551-560. Muhén, B. (1993). Goodness of fi wih caegorical and oher non normal variables. In K.A. Bollen & J.S. Long [Eds.] Tesing srucural equaion models (pp. 05-34). Newbury Park, CA: Sage. Takane, Y. (1987). Analysis of covariance srucures and probabilisic binary choice daa. Communicaion and Cogniion, 0, 45-6. Thursone, L.L. (197). A law of comparaive judgmen. Psychological eview, 79, 81-99. Thursone, L.L. (1931). ank order as a psychological mehod. Journal of Experimenal Psychology, 14, 187-01. 56