1 Alpha f Deleted and Loss n Crteron Valdty Appeared n Brtsh Journal of Mathematcal and Statstcal Psychology, 2008, 6, Alpha f Item Deleted: A Note on Crteron Valdty Loss n Scale Revson f Maxmsng Coeffcent Alpha Tenko Raykov Mchgan State Unversty Author Note: I am grateful to B. Muthen for nstructve comments on constrant evaluaton, as well as to the Edtor and two anonymous Referees for valuable crtcsm on an earler draft of the paper, whch contrbuted consderably to ts mprovement. Correspondence on ths paper may be addressed to Tenko Raykov, Measurement and Quanttatve Methods, Mchgan State Unversty, East Lansng, MI 48824, USA; emal: RUNNING HEAD: ALPHA IF ITEM DELETED AND VALIDITY LOSS
2 Alpha f Deleted and Loss n Crteron Valdty 2 Abstract Ths note s concerned wth a valdty-related lmtaton of the wdely avalable and used ndex alpha f tem deleted n the process of constructon and development of multplecomponent measurng nstruments. Attenton s drawn to the fact that ths statstc can suggest dspensng wth such scale components, whose removal leads to loss n crteron valdty whle maxmsng the popular coeffcent alpha. As an alternatve, a latent varable modellng approach s dscussed that can be used for pont and nterval estmaton of composte crteron valdty (as well as relablty) after deleton of sngle components. The method can also be utlsed to test conventonal or mnmum level hypotheses about assocated populaton change n measurement qualty ndces. Keywords: coeffcent alpha, crteron valdty, nterval estmaton, latent varable modellng, multple-component measurng nstrument, relablty
3 Alpha f Deleted and Loss n Crteron Valdty 3 Alpha f Item Deleted : A Note on Loss of Crteron Valdty n Scale Development If Maxmsng Coeffcent Alpha Multple-component measurng nstruments are hghly popular n psychology and the behavoural scences. Before beng wdely used they typcally need to undergo a process of development through possbly repeated revsons that am to ensure hgh psychometrc qualty of fnally recommended scales, n partcular hgh relablty and valdty. A rather frequently used statstc for these purposes n emprcal research s Cronbach s coeffcent alpha (α; e.g., Cronbach, 95), and especally the ndex alpha f tem deleted that represents the ncrement or drop n the sample value of α f dspensng wth a scale component. Recently, however, Raykov (2007a) showed that n certan crcumstances that do not appear rare n behavoural research, ths ndex can suggest the deleton of such nstrument components whose removal leads to maxmal ncrement n α but entals consderable loss n composte relablty. Ths results from the fact that α n general ncorrectly evaluates scale relablty already at the populaton level (e.g., Novck & Lews, 967; Zmmerman, 972), and ponts out the possblty that whle seekng components to remove n order to maxmse coeffcent alpha, a psychologst can n fact serously compromse relablty of an nstrument beng developed. The present note deals wth an addtonal aspect of ths potentally serous lmtaton of the popular alpha coeffcent, and n partcular of the wdely utlsed statstc alpha f tem deleted. The remander ndcates that crteron valdty can smlarly decrease as a result of removng a component from a tentatve scale whle maxmsng coeffcent alpha, even f data were avalable from an entre studed populaton. As an alternatve to ths statstc, therefore, an extenson of the latent varable modellng procedure n Raykov (2007a) s recommended that yelds pont and nterval estmaton of crteron valdty, n addton to that of relablty, after dspensng wth sngle components. The method provdes ranges of plausble populaton values for these measurement qualty ndces followng any component s deleton, and can be used for testng conventonal as well as mnmum level hypotheses about them.
4 Alpha f Deleted and Loss n Crteron Valdty 4 Loss n Crteron Valdty When Deletng Components to Maxmse Coeffcent Alpha Ths dscusson s based on the assumpton that a set of congenerc measures s gven, denoted X, X 2,, X p (p > 2; Jöreskog, 97), that s, X = + ε = γ + β ξ + ε () T holds ( =,, p), where T,...,, T2 Tp and ε, ε 2,..., ε p are respectvely ther true and error scores, and ξ desgnates the common latent dmenson evaluated by the measures (e.g., ξ = T can be taken; Lord & Novck, 968). For dentfablty reasons, Var(ξ) = s also set, where Var(.) denotes varance n a studed populaton, and wth respect to ε t s only requred that ther covarance matrx Ψ be postve defnte (e.g., Zmmerman, 975; =,, p). Assume also that a crteron varable, C, s pre-specfed. In the rest of ths artcle, the relablty of the composte Y = X + X X p wll be of nterest as well as that of closely related versons of t, along wth ther crteron valdty as reflected n the correlaton coeffcent Corr (Y, C) (e.g., Crocker & Algna, 986). 2 In the process of nstrument constructon and development, behavoural scentsts commonly follow the wde-spread practce of repeatedly examnng the change n coeffcent alpha after sngle component removal, typcally referred to as alpha f tem deleted. Ths procedure s based on the sample value of the gan or loss n α occurrng f say the th component s dropped from a tentatve scale: Y, = αy αy, α, (2) where
5 Alpha f Deleted and Loss n Crteron Valdty 5 α Y = Cov( X, X p p Var( Y ) ) (3) s coeffcent alpha for the scale Y, and α Y,- denotes ths coeffcent for the composte Y - = Y X,.e., represents alpha f ths tem X s deleted ( =,, p). The estmates of α Y, are furnshed by wdely crculated software, e.g., SPSS, SAS, STATISTICA, and are at present nearly routnely utlsed n the behavoural and socal scences for purposes of nstrument revson ( =,, p). Use of ths procedure s based on the tact, but n general ncorrect (see above), assumpton that α represents the change n relablty followng deleton of the th component ( =,, p). On ths presumpton, a wdely adhered to practce n emprcal research s to nspect the ndex alpha f tem deleted for each component n a tentatve scale, n an effort to dentfy a way of maxmally enhancng relablty va sngle tem deleton; then scholars commonly proceed wth the scale verson whch results from droppng the component assocated wth the hghest α Y, such that α Y > αy Y,, ( =,, p). As shown recently n Raykov (2007a), however, ths procedure cannot be generally trusted because of two mportant reasons. On the one hand, t depends crtcally on the sample estmate of the gan or drop n coeffcent alpha due to component removal, and n addton α n general ncorrectly evaluates scale relablty already at the populaton level, as mentoned earler. Consequently, a researcher relyng on the ndex alpha f tem deleted could decde to proceed wth such a revson of a tentatve composte, whch s assocated wth maxmal ncrease n α but n actual fact leads to lower relablty, even f data were avalable from an entre populaton. Ths lmtaton of the popular statstc alpha f tem deleted turns out to have further consequences to that ust ndcated. Specfcally, dspensng wth a component for whch α Y, > αy can lead also to loss n crteron valdty, a maor aspect of what may well be consdered the bottom lne n behavoural measurement. To see ths, from
6 Alpha f Deleted and Loss n Crteron Valdty 6 Equaton () follows Y k = k = ( β ) ξ + ε, (4) = and hence (e.g., Lord & Novck, 968) Corr Cov[ C,( k β ) ξ + k ε ] σ = = ξc = ξc ( Y, C) = = = ρ k k Y = ωy, (5) σ C k 2 k σ C Var( C) Var[( β ) ξ + ε ] ( β ) + θ = = = = k β σ say, where σ ξc, σ C, and ρ Y denote the latent-crteron covarance, crteron standard devaton and scale relablty, respectvely, whle θ = Var(ε ) ( =,, p). Now denote by ω and ρ correspondngly the crteron valdty and Y, Y, relablty of a tentatve scale from whch the th component s dropped ( =,, p). As shown n Raykov (2007a), there exst theoretcally and emprcally relevant settngs that do not appear rare n psychologcal research, where removal of a component assocated wth the maxmal ncrease n coeffcent alpha entals n fact a loss n relablty. 3 Let n such a settng the kth component possess accordngly the propertes that () α > α Y, k Y, () α Y, k s hghest for all k ( k p), and () ρ Y, k < ρ Y. From the nequalty n () and Equaton (5) (wth ts correspondng modfcaton for the so-revsed composte), t obvously follows ω σ ξc ξc Y, = Corr( Y X k, C) = ρy, k < ρy = Corr( Y, C) = ωy, (6) σ C σ C σ that s, after dspensng wth ts kth component the revsed scale has actually lower crteron valdty than ts mmedately precedng verson from whch t s obtaned. Equaton (6) lets one further observe that the amount by whch crteron valdty wll be
7 Alpha f Deleted and Loss n Crteron Valdty 7 compromsed n ths way, ω Y ωy,, depends on the assocated loss n relablty ndex and the correlaton between latent and crteron varables. Therefore, at least n the crcumstances outlned n Raykov (2007a; see Footnote 3), the resultng trmmed scale score, Y X k, wll have lower crteron valdty. Thus t s possble that a psychologst nvolved n nstrument development who follows the wdespread practce of removng a component from a tentatve scale, whch s assocated wth the hghest ncrease n coeffcent alpha, n actual fact arrves at a revsed scale that has consderably nferor crteron valdty (n addton to such relablty) relatve to ts verson before droppng that component. When ths practce s adhered to across several consecutve revsons, as s commonly the case n emprcal research, due to accumulaton of ths negatve effect t s obvous that the end verson may have substantally lower crteron valdty as well as relablty compared to an ntal scale. A Latent Varable Modellng Approach to Evaluaton of Measurement Qualty Followng Sngle Component Deleton In order to resolve these potentally serous defcences of the popular statstc alpha f tem deleted, the latent varable modellng approach n Raykov (2007a) can be extended to accomplsh pont and nterval estmaton of crteron valdty after deleton of any component from a tentatve scale. Ths procedure s not concerned wth the statstc alpha f tem deleted but s nstead entrely based on the coeffcent of crteron valdty, as well as that of relablty, for the composte resultng from deletng the th component of a gven scale ( =,.., p). To ths end, Equaton () s frst consdered as defnng a latent varable model (e.g., Muthén, 2002), and then 2p external parameters (new parameters, or auxlary parameters) are ntroduced (cf. Raykov, 2007b). The frst p of them, denoted π, π 2,, π p, are defned as the crteron valdty coeffcents of the verson resultng after deletng the th component (see Equaton (5)):
8 Alpha f Deleted and Loss n Crteron Valdty 8 π σ ξc = σ C k β =, k 2 k ( β + ) θ = = = Y, ω, (7) whle the second set of p external parameters, π p+, π p+2,, π 2p, are defned as the relablty coeffcents of the correspondng scale versons p 2 ( β ) = π p+ = k = k Y, 2 ( β ) + θ = = ρ, (8) ( =,, p). (Wth error covarances, the denomnators n the rght-hand sde of Equatons (7) and (8) are extended by the sum of non-zero error covarance estmates; e.g., McDonald, 999.) It s emphassed that (7) and (8) are not model parameters but are functons of the latter, and hence can be estmated once those are so. When the maxmum lkelhood (ML) method s used for model fttng purposes, due to the nvarance property of ML, the rght-hand sdes of (7) and (8) wrtten n terms of the partcpatng parameter estmates represent correspondngly the ML estmates of crteron valdty and scale relablty after removng the th component ( =,, p). Therefore, the latter estmates share all desrable large-sample propertes of ML estmates consstency, unbasedness, normalty and effcency (e.g., Rao, 973). The estmates (7) and (8) do not address the queston of how close they are to the respectve populaton crteron valdty and relablty coeffcents after dspensng wth a gven component from a tentatve scale, whch are the actual coeffcents of nterest. To ths end, a standard error and confdence nterval for these quanttes s needed. Usng the delta method (e.g., Rao, 973), an approxmate standard error and confdence nterval was furnshed n Raykov (2007a) for scale relablty followng deleton of any
9 Alpha f Deleted and Loss n Crteron Valdty 9 component, and the same method can be used here to also render an approxmate standard error and confdence nterval for the crteron valdty of each scale verson obtaned n ths way. Denote frst the (2p - 2) x vector of parameters n model () after deletng the th scale component by ψ = (ψ,,ψ -, ψ +,, ψ p, ψ p+,, ψ p+-, ψ p++,, ψ 2p ), where prmng stands for transposton and the notaton ψ = β,, ψ p- = β p, ψ p = θ,, ψ 2p = θ p s used for ease of reference (see Equaton (); =,, p). The frst-order Taylor expanson of the crteron valdty coeffcent (7) around the populaton parameter ψ 0,- = ( 0,,..., β0,, β0,,..., β0, p, θ0,,..., θ0,, θ0,,..., θ0, p ) β + + = ( 0,,..., ψ 0,, ψ 0,,..., ψ 0, p, ψ 0, p,..., ψ 0, p, ψ 0, p,..., ψ 0, 2 p ) ψ s 2 p ˆ p, ω p, ( ψ 0, ) + Dˆ ( ψˆ, ψ 0,, ) =,, p+ ˆ ω, (9) where denotes approxmately equal and Dˆ = ˆ ω p, ψ s the partal dervatve of ˆω p, wth respect to ts th argument, taken at the parameter estmate pont ( =,, -, +,, p+-, p++,, 2p; =,, p). (The explct expressons for these dervatves can be rendered followng well-known rules for dfferentaton, but are actually not needed for the purposes of ths note, as ndcated below.) Hence, an approxmate standard error for crteron valdty followng removal of the th component s obtaned from Equaton (9) as: ˆ ω p ˆ, ω p, S. Eˆ.( ω p, ) = Cov( ψˆ ), (0) ψ ψ
10 Alpha f Deleted and Loss n Crteron Valdty 0 ˆ ω p where ( D D D D D D D ), = ˆ, ˆ,..., ˆ, ˆ,..., ˆ,..., ˆ, ˆ,..., ˆ ψ D 2 + p p+ p+ + 2 p s the row vector of above mentoned dervatves and Cov ψ ˆ ) s the covarance matrx of pertnent ( parameter estmators, evaluated at the model soluton ( =,, p; cf. Raykov, 2007a). Wth ths standard error, an approxmate 00(-δ)%-confdence nterval (0 < δ < ) for crteron valdty after droppng the th component results as follows by captalsng on the asymptotc normalty of the latent varable model parameter estmator (e.g., Muthén, 2002): (max(0, ˆω p, - z -δ/2 S. Eˆ.( ω p, ) ), mn (, ˆω p, + z -δ/2 S. Eˆ.( ω p, ))), () where z -δ/2 denotes the δ/2th quantle of the standard normal dstrbuton whle max(.,.) and mn(.,.) stand for the larger and smaller numbers followng n parentheses, respectvely ( =,, p). The confdence nterval () provdes a range of plausble values, at a confdence level δ, for the populaton crteron valdty of the composte of all components but the th ( =,, p). From the dualty between hypothess testng and confdence nterval (e.g., Hays, 994) t follows that () could also be used to test, as well known, conventonal hypotheses at a sgnfcance level - δ about the crteron valdty (or, for the same matter, relablty) coeffcent after the th component s dropped ( =,, p). Moreover, () can be used to test mnmum level hypotheses about ths coeffcent. Such a hypothess states that after dspensng wth a component from a gven composte, the crteron valdty (or relablty) s equal to at least w 0 say (0 < w 0 < ), where w 0 s a substantvely desrable threshold for crteron valdty (or relablty) that a psychologst requres the composte to attan before beng recommended for wder use. Accordngly, the pertnent null hypothess s H 0 : ω - w 0 (or H 0 : ρ - w 0 ), wth correspondng alternatve hypothess H : ω - < w 0 (or H : ρ - < w 0 ) ( =,, p). Ths hypothess s tested by examnng whether the left-endpont of the correspondng confdence nterval s
11 Alpha f Deleted and Loss n Crteron Valdty entrely above the threshold, n whch case the null hypothess s consdered retanable. Otherwse the alternatve hypothess s accepted and further nstrument revson may be called for n order to accomplsh the desred mnmal level of valdty (or relablty). Emprcal mplementaton The descrbed approach can be mplemented n behavoural research wth the ncreasngly popular latent varable modellng program Mplus (Muthén & Muthén, 2006). Ths software ncorporates recent advances n numercal optmzaton, whch allow one to utlse readly the delta method applcaton outlned n the precedng secton for obtanng approxmate standard errors and confdence ntervals. Specfcally, fttng model () wth the added 2p external parameters n Equatons (7) and (8), upon a request for confdence nterval evaluaton, yelds pont as well as nterval estmates of these parameters,.e., for the crteron valdty (and relablty) coeffcents after removng any component from a tentatve scale. (The code accomplshng ths goal, wth annotatons, s provded n Appendx where t s appled wth data used n the llustraton secton.) It s emphassed that ths approach yelds nterval estmates of crteron valdty (or relablty) followng sngle component removal, as well as of an ntally consdered scale, whereas there s no counterpart nterval estmate avalable when one adheres to the wdely followed practce of usng the ndex alpha f tem deleted for scale revson purposes. The dscussed procedure s also drectly applcable n settngs wth mssng data that are frequently encountered n behavoural research dealng wth scale constructon and development. The method s then straghtforwardly employed va use of full nformaton maxmum lkelhood or multple mputaton f ther assumptons are plausble vz. data mssng at random and normalty (e.g., Muthén & Muthén, 2006; Lttle & Rubn, 2002). Last but not least, the proposed approach can be repeatedly used on scale versons resultng from precedng measure removal, n the search of yet further mprovement n ther crteron valdty (and relablty) followng deleton of any of ther own components. Fnal recommendatons regardng composte revson should be based,
12 Alpha f Deleted and Loss n Crteron Valdty 2 however, on results from a replcaton study on an ndependent sample from the same populaton, due to the possblty of captalzaton on chance. Illustraton on Data In order to demonstrate the possblty that the wdely used statstc alpha f tem deleted can suggest msleadng avenues of scale mprovement that are n fact assocated wth pronounced loss n crteron valdty, smulated multnormal data are employed n ths secton. These data wll also allow llustraton of the utlty of the dscussed approach to evaluaton of crteron valdty (and relablty) after sngle component deleton. To ths end, multvarate, zero-mean normal data were generated for N = 500 cases and k = 5 components Y through Y 5 accordng to the model X = ξ + ε (2) X 2 = ξ + ε 2 X 3 = ξ + ε 3 X 4 = ξ + ε 4 X 5 = 6 ξ + ε 5, where ξ was standard normal and the error terms ε through ε 5 were ndependent zeromean normal varables wth varance.3 each; the crteron varable was generated as havng correlaton of.80 wth ξ. The resultng covarance matrx s presented n Table. Insert Table about here Conventonal scale analyss on the ntal composte contanng all 5 components Y through Y 5 (.e., of the scale score Y = Y + + Y 5 ) reveals an estmated alpha coeffcent of.702. The wdely used statstc alpha f tem deleted ndcates then that alpha wll be maxmsed f the last component, Y 5, s removed from ths composte. Specfcally, accordng to that statstc, ths removal would yeld a four-component scale
13 Alpha f Deleted and Loss n Crteron Valdty 3 wth an alpha of.749 that s more than. hgher than the alpha resultng from droppng nstead any of the other four components (.e., Y to Y 4 ) from the ntal composte. It s stressed that alpha f tem deleted suggests here droppng the most relable component of all fve, n order to maxmse alpha. Indeed, Equatons (2) and mmedately followng dscusson mply that relablty of each of the frst four components s under.50, whle that of Y 5 s n excess of.95. To see the effect of deletng the last component on crteron valdty and relablty, the latent varable modelng approach of ths artcle s appled. Frst, fttng the congenerc model () (wth the 2p external parameters, whch do not affect model ft as they do not have any mplcatons on the covarance structure), one obtans acceptable goodness of ft ndces: ch-square = 3.73, degrees of freedom (df) = 9, p-value (p) =.55, root mean square error of approxmaton (RMSEA) =.030 wth a 90%-confdence nterval (0,.063). The crteron valdty and relablty of the fve versons of the ntal scale, whch result after each of ts components s dropped n turn, as well as of that startng scale are presented n Table 2 along wth correspondng standard errors and confdence ntervals. Insert Table 2 about here As seen from Table 2, removal of Y 5 as suggested by the statstc alpha f tem deleted n fact leads to a substantal decrement n crteron valdty from.773 (ntal scale) to.687 (composte of frst four components only), that s a drop by more than 0%. (Usng the data generaton parameters, ths crteron valdty loss s found to be equal to.096 n the populaton.) Smlarly, relablty drops from.938 to.742,.e., by more than 20%. (In the same way, ths relablty decrement s found to be.27 n the populaton.) These effects represent pronounced losses n measurement qualty, whch result f one were to follow the wde-spread practce of deletng the sngle component whose removal maxmses coeffcent alpha. Note also that precson of estmaton, as udged by the wdth of the assocated confdence ntervals, also drops f one were to dspense wth the last component followng that popular procedure. Further, from Table 2 t s seen that deleton of any of the frst four components nstead does not have a notable effect on the
14 Alpha f Deleted and Loss n Crteron Valdty 4 pont estmate of crteron valdty or relablty, whle leadng to some relatvely mnmal loss of estmaton precson. Ths demonstraton exemplfes the pont that adherng to the wdely used statstc alpha f tem deleted for purposes of scale revson can be assocated wth a marked loss n crteron valdty and relablty, two measurement qualty ndces of specal relevance for psychology and the behavoural scences. Concluson For a number of decades, a wde-spread practce has been followed by behavoural scentsts nvolved n nstrument development. Accordngly, the sample values of the popular coeffcent alpha before and after sngle component removal have receved crtcal attenton n an effort to fnd ways of revsng tentatve scales so as to maxmally enhance ther relablty. In partcular, the ndex alpha f tem deleted has been rather frequently nspected for ths purpose. The present note hghlghts a valdtyrelated lmtaton of ths statstc. The artcle shows that dspensng wth a scale component to maxmally ncrease coeffcent alpha, can n fact ental consderable loss n crteron valdty, a maor aspect of behavoural measurement qualty. In addton to a recent demonstraton n Raykov (2007a) that such a revson path can lead to loss n composte relablty, ths note further cautons psychologsts engaged n nstrument development that use of alpha f tem deleted can be serously msleadng n more than one mportant way. As an alternatve, the note dscusses a latent varable modellng procedure that provdes pont and nterval estmates of both crteron valdty and relablty followng deleton of each component n a tentatve scale. In addton, the outlned approach allows smultaneous examnaton of the factoral structure of a gven set of measures consdered as ts components. Moreover, the method s straghtforwardly applcable n cases wth mssng data usng maxmum lkelhood or multple mputaton, when ther assumptons are plausble (vz. mult-normalty and data mssng at random), whch s qute often the case n emprcal contexts where nstrument development s conducted. The dscussed procedure, beng based on latent varable modellng that s grounded n an asymptotc theory (e.g., Muthén, 2002), yelds most trustworthy results wth large samples, and smlarly wth (approxmately) contnuous components. Further,
15 Alpha f Deleted and Loss n Crteron Valdty 5 beng concerned wth crteron valdty, the proposed method may yeld lmted nformaton about other relevant types of valdty of measurement (e.g., Crocker & Algna,986). In addton, ts results depend on the choce of a crteron varable, whch should be made n emprcal research based on detaled knowledge of a substantve doman of concern. Fnally, as presented n ths note, the procedure utlses the assumpton of congenerc measures, but t s stressed that t s readly extended to the case of more than a sngle underlyng source of latent varablty (see Footnote ).
16 Alpha f Deleted and Loss n Crteron Valdty 6 References Bollen, K. A. (989). Structural equatons wth latent varables. New York: Wley. Crocker, L., & Algna, J. (986). Introducton to classcal and modern test theory. Fort Worth, TX: Harcourt Brace Jovanovch. Cronbach, L. J. (95). Coeffcent alpha and the nternal structure of a test. Psychometrka, 6, Hays, W. L. (994). Statstcs. Fort Worth, TX: Harcourt Brace Jovanovch. Jöreskog, K. G. (97). Statstcal analyss of sets of congenerc tests. Psychometrka, 36, Lttle, R. J., & Rubn, D. B. (2002). Statstcal analyss wth mssng data. New York: Wley. Lord, F., & Novck, M. (968). Statstcal theores of mental test scores. Readngs, MA: Addson-Wesley. McDonald, R. P. (999). Test theory. A unfed treatment. Mahwah, NJ: Erlbaum. Muthén, B. O. (2002). Beyond SEM: General latent varable modelng. Behavormetrka, 29, 8-7. Muthén, L. K., & Muthén, B. O. (2006). Mplus user s gude. Los Angeles, CA: Muthén & Muthén. Novck, M. R., & Lews, C. (967). Coeffcent alpha and the relablty of composte measurement. Psychometrka, 32, -3. Rao, C. R. (973). Lnear statstcal nference and ts applcatons. New York: Wley. Raykov, T. (2007a). Relablty f deleted, not alpha f deleted : Evaluaton of scale relablty followng component deleton. Brtsh Journal of Mathematcal and Statstcal Psychology, 60, Raykov, T. (2007b). Estmaton of revson effect on crteron valdty of multple-component measurng nstruments. Multvarate Behavoral Research, 42, Zmmerman, D. W. (972). Test relablty and the Kuder-Rchardson formulas: Dervaton from probablty theory. Educatonal and Psychologcal Measurement, 32, Zmmerman, D. W. (975). Probablty measures, Hlbert spaces, and the axoms of classcal test theory. Psychometrka, 30,
17 Alpha f Deleted and Loss n Crteron Valdty 7 Table Covarance matrx of fve congenerc measures (N = 500) Varable Y Y 2 Y 3 Y 4 Y 5 C Y Y Y Y Y C Note. N = sample sze, C = crteron varable.
18 Alpha f Deleted and Loss n Crteron Valdty 8 Table 2 Pont and nterval estmates of crteron valdty and relablty of composte resultng after ndcated component s dropped from ntal scale wth all fve components, and for that ntal scale DM CV SE CI(CV) R SE CI(R) Y (.739,.807) (.924,.960) Y (.739,.807) (.923,.959) Y (.739,.807) (.924,.960) Y (.740,.808) (.925,.96) Y (.648,.727) (.707,.777) None (.739,.807) (.925,.952) Note. DM = dropped measure; CV = crteron valdty, R = relablty; SE = standard error, CI(CV) and CI(R) = 95%-confdence nterval of crteron valdty and of relablty, respectvely. Entres n row None pertan to estmates and standard errors for crteron valdty and relablty of the ntal scale wth all fve components (.e., when none of the latter s removed).
19 Alpha f Deleted and Loss n Crteron Valdty 9 Footnotes If p = 2, addtonal dentfyng restrctons wll be needed, such as ndcator loadng equalty (true score-equvalent measures) and/or error varance equalty (e.g., parallel measures; Lord & Novck, 968). Snce the locaton parameters, γ 2 γ k are not consequental for relablty n the settng underlyng ths γ,..., paper, for convenence they are all assumed equal to zero (e.g., Bollen, 989). The developments n ths note can be drectly generalsed to the case where more than a sngle latent dmenson s evaluated by a consdered set of measures, followng the correspondng approach n McDonald (999; omega coeffcent). 2 The procedure dscussed below s readly extended to the case when C s a latent varable wth at least two ndcators. As s common n latent varable modellng, C s also assumed unrelated to the error terms n the observed measures X,, X p (e.g., Bollen, 989). 3 As outlned n Raykov (2007a), at least the followng general setup belongs to these emprcal settngs, wth a sngle latent varable ξ and p = q + ndcators (p > 2; see Footnote ): X = β ξ + ε ( =,, q), X q+ = γ ξ + ε q+, where γ > β s suffcently large, Var(ε + ) Var(ε ) = θ ( =,, q; obvously, wthout lmtaton of generalty one can also presume that β > 0; n the last q+ equatons consderaton of component ntercepts s dspensed wth as they are nconsequental for relablty; see Footnote ). As shown n the last cted source, deleton of the last component n ths setup, whch leads to the hghest ncrement n coeffcent alpha, entals substantal loss of relablty. (Note that ths setup descrbes a case of q+ congenerc measures, of whch the frst q are parallel whle the last one s the most relable of all; see also Appendx 2 n that source.)
20 Alpha f Deleted and Loss n Crteron Valdty 20 Appendx Mplus Code for Evaluaton of Crteron Valdty and Relablty After Sngle Component Deleton TITLE: EVALUATION OF CRITERION VALIDITY/RELIABILITY AFTER COMPONENT DELETION DATA: FILE = <fle name>! PROVIDES NAME OF RAW DATA FILE. VARIABLE: NAMES = Y-Y6;! ATTACHES LABELS TO OBSERVED VARIABLES MODEL: KSI BY Y* (P)! THIS AND NEXT 4 LINES DEFINE THE COMPONENTS Y2* (P2)! AND ATTACH TO THEM PARAMETER SYMBOLS TO BE Y3* (P3)! USED BELOW (SEE MODEL CONSTRAINT SECTION). Y4* (P4) Y5* (P5); Y* (P6);! THIS AND NEXT 4 LINES DEFINE THE ERROR VARIANCES Y2* (P7);! AND ATTACH TO THEM PARAMETER SYMBOLS TO BE USED Y3* (P8);! BELOW (SEE MODEL CONSTRAINT SECTION). Y4* (P9); Y5* (P0); C BY Y6*; C WITH KSI* (P);! C IS THE CRITERION VARIABLE FIXES LATENT VARIANCE AT, FOR MODEL IDENTIFICATION MODEL CONSTRAINT: NEW(PI_ PI_2 PI_3 PI_4 PI_5 PI_6 PI_7 PI_8 PI_9 PI_0 PI_ PI_2);! INTRODUCES THE AUXILIARY PARAMETERS π, π 2,, π 2 (SEE EQ. (7), (8)) PI_6=(P2+P3+P4+P5)**2/ ((P2+P3+P4+P5)**2+P7+P8+P9+P0);! = RELIABILITY W/OUT ST COMPONENT PI_7=(P+P3+P4+P5)**2/ ((P+P3+P4+P5)**2+P6+P8+P9+P0);! = RELIABILITY W/OUT 2ND COMPONENT PI_8=(P+P2+P4+P5)**2/ ((P+P2+P4+P5)**2+P6+P7+P9+P0);! = RELIABILITY W/OUT 3RD COMPONENT PI_9=(P+P2+P3+P5)**2/ ((P+P2+P3+P5)**2+P6+P7+P8+P0);! = RELIABILITY W/OUT 4TH COMPONENT PI_0=(P+P2+P3+P4)**2/ ((P+P2+P3+P4)**2+P6+P7+P8+P9);! = RELIABILITY W/OUT 5TH COMPONENT PI_2=(P+P2+P3+P4+P5)**2/ ((P+P2+P3+P4+P5)**2+P6+P7+P8+P9+P0);! = RELIABILITY WITH ALL COMP. PI_=P*SQRT(PI_6);! = CRITERION VALIDITY W/OUT ST COMPONENT. PI_2=P*SQRT(PI_7);! = CRITERION VALIDITY W/OUT 2ND COMPONENT. PI_3=P*SQRT(PI_8);! = CRITERION VALIDITY W/OUT 3RD COMPONENT.