Validity evidence based on internal structure

Transcription

1 Pscothema 014, Vol. 6, No. 1, do: /pscothema ISSN CODEN PSOTEG Copyrght 014 Pscothema Valdty evdence based on nternal structure Joseph Ros and Crag Wells Unversty of Massachusetts Amherst (USA) Abstract Background: Valdty evdence based on the nternal structure of an assessment s one of the fve forms of valdty evdence stpulated n the Standards for Educatonal and Psychologcal Testng of the Amercan Educatonal Research Assocaton, Amercan Psychologcal Assocaton, and Natonal Councl on Measurement n Educaton. In ths paper, we descrbe the concepts underlyng nternal structure and the statstcal methods for gatherng and analyzng nternal structure. Method: An ndepth descrpton of the tradtonal and modern technques for evaluatng the nternal structure of an assessment. Results: Valdty evdence based on the nternal structure of an assessment s necessary for buldng a valdty argument to support the use of a test for a partcular purpose. Conclusons: The methods descrbed n ths paper provde practtoners wth a varety of tools for assessng dmensonalty, measurement nvarance and relablty for an educatonal test or other types of assessment. Keywords: valdty, standards, dmensonalty, measurement nvarance, relablty. Resumen Evdenca de valdez basada en la estructura nterna. Antecedentes: la evdenca de valdez basada en la estructura nterna de una evaluacón es una de las cnco formas de evdencas de valdez estpuladas en los Standards for Educatonal and Psychologcal Testng de la Amercan Educatonal Research Assocaton, Amercan Psychologcal Assocaton, and Natonal Councl on Measurement n Educaton. En este artículo descrbmos los conceptos que subyacen a la estructura nterna y los métodos estadístcos para analzarla. Método: una descrpcón detallada de las técncas tradconales y modernas para evaluar la estructura nterna de una evaluacón. Resultados: la evdenca de valdez basada en la estructura nterna de una evaluacón es necesara para elaborar un argumento de valdez que apoye el uso de un test para un objetvo partcular. Conclusones: los métodos descrtos en este artículo aportan a los profesonales una varedad de herramentas para evaluar la dmensonaldad, nvaranza de la medda y fabldad de un test educatvo u otro tpo de evaluacón. Palabras clave: valdez, standards, estructura nterna, dmensonaldad, nvaranza de la medda, fabldad. The Standards for Educatonal and Psychologcal Testng (Amercan Educatonal Research Assocaton [AERA], Amercan Psychologcal Assocaton, & Natonal Councl on Measurement n Educaton, 1999) lst fve sources of evdence to support the nterpretatons and proposed uses of test scores: evdence based on test content, response processes, nternal structure, relatons to other varables, and consequences of testng. Accordng to the Standards, evdence based on nternal structure, whch s the focus of ths paper, pertans to the degree to whch the relatonshps among test tems and test components conform to the construct on whch the proposed test score nterpretatons are based (p. 13). There are three basc aspects of nternal structure: dmensonalty, measurement nvarance, and relablty. When assessng dmensonalty, a researcher s manly nterested n determnng f the nter-relatonshps among the tems support the ntended test scores that wll be used to draw nferences. For example, a test that ntends to report one composte score should be predomnately undmensonal. For measurement nvarance, t s useful to provde evdence that the tem characterstcs Receved: August 7, 013 Accepted: October 10, 013 Correspondng author: Joseph Ros School of Educaton Unversty of Massachusetts Amherst Amherst (Estados Undos) e-mal: jaros@educ.umass.edu (e.g., tem dscrmnaton and dffculty) are comparable across manfest groups such as sex or race. Lastly, relablty ndces provde evdence that the reported test scores are consstent across repeated test admnstratons. The purpose of the present paper s to descrbe basc methods for provdng evdence to support the nternal structure of a test (e.g., achevement tests, educatonal surveys, psychologcal nventores, or behavoral ratngs) wth respect to assessng dmensonalty, measurement nvarance, and relablty. Assessng dmensonalty Assessng test dmensonalty s one aspect of valdatng the nternal structure of a test. Factor analyss s a common statstcal method used to assess the dmensonalty of a set of data (Bollen, 1989; Brown, 006; Klne, 010; Thompson, 004). There are several factor analytc methods avalable for analyzng test dmensonalty; however, ths paper wll focus solely on confrmatory factor analyss, whch s the most comprehensve means for comparng hypotheszed and observed test structures. Conf rmatory factor analyss Confrmatory factor analyss (CFA) s a type of structural equaton model (SEM) that examnes the hypotheszed 108

2 Valdty evdence based on nternal structure relatonshps between ndcators (e.g., tem responses, behavoral ratngs) and the latent varables that the ndcators are ntended to measure (Bollen, 1989; Brown, 006; Klne, 010). The latent varables represent the theoretcal construct n whch evdence s collected to support a substantve nterpretaton. In comparson to exploratory factor analyss (EFA), a basc feature of CFA s that the models are specfed by the researcher a pror usng theory and often prevous emprcal research. Therefore, the researcher must explctly specfy the number of underlyng latent varables (also referred to as factors) and whch ndcators load on the specfc latent varables. Beyond the attractve feature of beng theoretcally drven, CFA has several advantages over EFA such as ts ablty to evaluate method effects and examne measurement nvarance. CFA provdes evdence to support the valdty of an nternal structure of a measurement nstrument by verfyng the number of underlyng dmensons and the pattern of tem-tofactor relatonshps (.e., factor loadngs). For example, f the hypotheszed structure s not correct, the CFA model wll provde poor ft to the data because the observed nter-correlatons among the ndctors wll not be accurately reproduced from the model parameter estmates. In ths same ven, CFA provdes evdence of how an nstrument should be scored. If a CFA model wth only one latent varable fts the data well, then that supports the use of a sngle composte score. In addton, f the latent structure conssts of multple latent varables, each latent varable may be consdered a subscale and the pattern of factor loadngs ndcates how the subscores should be created. If the mult-factor model fts the data well, and the construct s ntended to be multdmensonal, then that s evdence supportng the nternal structure of the measurement nstrument. Furthermore, for mult-factor models, t s possble to assess the convergent and dscrmnant valdty of theoretcal constructs. Convergent valdty s supported when ndcators have a strong relatonshp to the respectve underlyng latent varable. Dscrmnant valdty s supported when the relatonshp between dstnct latent varables s small to moderate. In fact, CFA can be used to analyze multtratmultmethod (MTMM; Campbell & Fsk, 1959) data (Kenny, 1976; Marsh, 1989). Three sets of parameters are estmated n a CFA model. For one, the factor loadngs, whch represent the strength of the relatonshp between the ndcator and ts respectve latent varable and may be consdered a measure of tem dscrmnaton, are estmated. In CFA, the factor loadngs are fxed to zero for ndcators that are not hypotheszed to measure a specfc latent varable. When standardzed, and no cross-loadngs exst (.e., each ndcator loads on one latent varable), the factor loadngs may be nterpreted as correlaton coeffcents. The varance and covarance coeffcents for the latent varables are also estmated. However, the varance for each latent varable s often fxed to one to establsh the scale of the latent varable. Fxng the varance for each latent varable to one produces a standardzed soluton. Lastly, the varance and covarance coeffcents for the measurement errors (.e., unque varance for each ndcator) are estmated. When the measurement errors are expected to be uncorrelated, the covarance coeffcents are fxed to zero. To examne the nternal structure of a measurement nstrument, the CFA model s evaluated for model ft and the magntude of the factor loadngs and correlatons among the latent varables are examned. Model ft determnes f the hypotheszed model can reproduce the observed covarance matrx (.e., covarance matrx for the ndcators) usng the model parameter estmates. If the model s specfed ncorrectly (e.g., some ndcators load on other latent varables) then the model wll not ft the data well. Although there are several approaches to assess model ft, such as hypothess testng, the most common method uses goodnessof-ft ndces. There are a plethora of goodness-of-ft ndces avalable for a researcher to use to judge model ft (see Bollen, 1989; Hu & Bentler, 1999). It s advsable to use a few of the ndces n evaluatng model ft. Some of the more commonly used ndces are the comparatve ft ndex (CFI), Tucker-Lews ndex (TLI), root mean square error of approxmaton (RMSEA), and standardzed root mean square resdual (SRMR). Suggested cutoff values are avalable to help researchers determne f the model provdes adequate ft to the data (e.g., See Hu & Bentler, 1999). A model that does not ft the data well must be re-specfed before nterpretng the parameter estmates. Although there are numerous CFA models that one can ft to the sample data, n ths paper we descrbe and llustrate the ncreasngly popular bfactor model. Bfactor model The bfactor model (also referred to as the nested or generalspecf c model) frst ntroduced by Holznger and Swneford (1937) has seen a drastc ncrease n popularty wthn the SEM and tem response theory (IRT) lterature over the past few years. Once overshadowed by alternatve multdmensonal models, such as the correlated-factors and second-order models, advances n parameter estmaton, user-frendly software, and novel applcatons (e.g., modelng dfferental tem functonng (Fukuhara & Kamata, 011; Jeon, Rjmen, & Rabe-Hesketh, 013), dentfyng local dependence (Lu & Thssen, 01), evaluatng construct shft n vertcal scalng (L & Lsstz, 01), to name a few) have led to a renewed nterest n the model. However, applcatons of the bfactor model have been lmted n the feld of psychology, whch some have suggested s due to a lack of famlarty wth the model and a lack of apprecaton of the advantages t provdes (Rese, 01). Therefore, the objectve of the current secton s to provde a general descrpton of the confrmatory canoncal bfactor model, note some of the advantages and lmtatons assocated wth the model, and dscuss technques for determnng model selecton when comparng undmensonal and bfactor models. General descrpton of bfactor model. The bfactor model s a multdmensonal model that represents the hypothess that several constructs, as ndcated each by a subset of ndcators, account for unque varance above and beyond the varance accounted for by one common construct that s specfed by all ndcators. More specfcally, ths model s composed of one general and multple specf c factors. The general factor can be conceptualzed as the target construct a measure was orgnally developed to assess, and accounts for the common varance among all ndcators. In contrast, specfc factors pertan to only a subset of ndcators that are hghly related n some way (e.g., content subdoman, tem type, locally dependent tems, etc.), and account for the unque varance among a subset of ndcators above and beyond the varance accounted for by the general factor. Wthn the confrmatory model, each ndcator loads on the general factor and on one and only one specfc factor. Allowng ndcators to cross-load on multple specfc factors leads to questonable parameter estmates, and s lmted by the small degrees of freedom avalable n the model. As 109

3 the specfc factors are nterpreted as the varance accounted for above and beyond the general factor, an orthogonal (uncorrelated) assumpton s made for the relatonshps between the general and specfc factors. Furthermore, the covarances among the specfc factors are set to 0 to avod dentfcaton problems (Chen, Hayes, Carver, Laurenceau, & Zhang, 01). The resdual varances of the ndcators are nterpreted as the varance unaccounted for by ether the general or specfc factors (see Fgure 1). Wthn the feld of psychology, ths model has been appled to study a number of constructs, such as depresson (Xe et al., 01), personalty (Thomas, 01), ADHD (Martel, Roberts, Gremllon, von Eye, & Ngg, 011), and posttraumatc stress dsorder (Wolf, Mller, & Brown, 011). Advantages of the bfactor model. The bfactor model possesses the followng four advantages over other multdmensonal models (e.g., the second-order model): 1) the doman specfc factors can be studed ndependently from the general factor, ) the relatonshp between the specfc factors and ther respectve ndcators can be evaluated, 3) nvarance can be evaluated for both the specfc and general factors ndependently, and 4) relatonshps between the specfc factors and an external crteron can be assessed above and beyond the general factor (Chen, West, & Sousa, 006). The ablty to study the specfc factors ndependently from the general factor s mportant n better understandng theoretcal clams. For example, f a proposed specfc factor dd not account for a substantal amount of varance above and beyond the general factor, one would observe small and non-sgnfcant factor loadngs on the specfc factor, as well as a non-sgnfcant varance of the specfc factor n the bfactor model. Ths would notfy the researcher that the hypotheszed specfc factor does not provde unque varance beyond the general factor, whch would call for a modfcaton of the theory and the test specfcatons. A closely related advantage of the bfactor model s the ablty to drectly examne the strength of the relatonshp between the specfc factors and ther respectve ndcators. Such an assessment provdes a researcher wth nformaton regardng the approprateness of usng partcular tems as ndcators of the specfc factors. If a relatonshp s weak, one can conclude that the tem may be approprate solely as an ndcator of the general factor. The last two advantages deal drectly wth gatherng valdty evdence to support a theoretcal ratonale. More specfcally, wthn the bfactor model one has the ablty to evaluate nvarance for both the specfc and general factors ndependently. Ths would allow researchers to drectly compare means of the latent factors (both the specfc and general factors), f scalar nvarance s met, across dstnctve subgroups of examnees wthn the populaton (See Levant, Hall, & Rankn, 013). Lastly, the bfactor model s advantageous n that one can study the relatonshps between the specfc factors and an external crteron or crtera above and beyond the general factor. Ths applcaton of the bfactor model could be partcularly attractve for gatherng evdence based on relatons to other varables (convergent and dscrmnant evdence, as well as test-crteron relatonshps) for multdmensonal measures. Lmtatons of the bfactor model. Although the bfactor model provdes numerous advantages, t s also has some lmtatons. As noted by Rese, Moore, and Havland (010), there are three major reasons for lmtng the applcaton of the bfactor model n practce: 1) nterpretaton, ) model specfcaton, and 3) restrctons. The frst major lmtng factor for practtoners s relatng the bfactor model to ther respectve substantve theores. More specfcally, the bfactor model assumes that the general and specfc factors are orthogonal to one another, whch may be too restrctve or make lttle sense n adequately representng a theoretcal model. For example, f one were studyng the role of varous workng memory components on readng sklls, t would be dffcult to assume the relatonshp between these two constructs s orthogonal. Instead, competng multdmensonal models, such as the correlated-trats or second-order models would be more attractve as the restrctve orthogonalty assumpton s not requred. Ths s one of the major N (0.0, 1.0) General I1 I I3 I4 I5 I6 I7 I8 I9 I10 I11 I1 Specfc Factor 1 Fgure 1. Bfactor model path dagram Specfc Factor Specfc Factor 3 N (0.0, 1.0) N (0.0, 1.0) N (0.0, 1.0) Factor structure a 11 a a 1 a 0 0 a 31 a a 41 a a 51 0 a 53 0 a 61 0 a 63 0 a 71 0 a 73 0 a 81 0 a 83 0 a a 94 a a 104 a a 114 a a

4 Valdty evdence based on nternal structure reasons why the bfactor model has seen lttle applcaton to noncogntve measures. A closely related lmtaton of the bfactor model s model specfcaton. Rese et al. (010) advsed that for stable parameter estmaton one should have at least three group factors, for each group factor there should be at least three ndcators, and the number of ndcators should be balanced across all group factors. The queston then becomes, can I stll apply the bfactor model f my theoretcal representaton s lackng n one of these areas? The answer s t depends. For one, wthn a SEM framework one should always have at least three ndcators per latent construct for dentfcaton purposes. Furthermore, the requrement of possessng at least three specfc factors holds true n the secondorder model, where t s requred that there are at least three frstorder factors that load onto the second-order factor (Chen, Hayes, Carver, Laurenceau, & Zhang, 01). If these frst two condtons are not met, one should not apply the bfactor model. In terms of the last condton, havng an unequal number of ndcators across specfc factors wll mpact relablty estmates of the subscales; however, keepng ths mnd, one can stll ft the model. Lastly, the bfactor model requres an addtonal restrctve assumpton beyond orthogonalty, whch s that each ndcator load on one general factor and one and only one specfc factor. Allowng tems to cross-load on multple specfc factors would lead to untrustworthy tem parameter estmates. Such a restrcton on the structure of the multdmensonalty may lmt the applcaton of the bfactor model. However, ths s one of the major reasons why Rese (01) promoted the use of exploratory bfactor analyss, whch allows for ndcators to cross-load on specfc factors (For a detaled dscusson on exploratory bfactor analyss see Jennrch & Bentler, 011). Such analyses would allow researchers to better understand the structure of the data before applyng confrmatory procedures, whch s partcularly vtal wth the restrctve assumptons that are nherent n the confrmatory canoncal bfactor model. Model selecton. Consderng plausble rval hypotheses s an mportant part of gatherng evdence to support the valdty of scored-based nferences (Amercan Educatonal Research Assocaton, Amercan Psychologcal Assocaton, & Natonal Councl on Measurement n Educaton, 1999). In terms of evdence based on nternal structure, rval hypotheses nclude alternatve theoretcal models. For example, when a measure s hypotheszed to compose one general and multple specfc factors, as s the case wth the bfactor model, t s mperatve to consder alternatve score nterpretatons. One such alternatve hypothess s that reportng separate scores for the general and specfc factors s unnecessary as the score varance can be captured by one promnent dmenson. That s, although a model may demonstrate adequate model ft for a multdmensonal model, practcal and techncal consderatons (e.g., lack of adequate relablty on the subscales, desre to employ undmensonal IRT applcatons, etc.) may dctate that reportng a undmensonal model s good enough or preferred. In ths case, one would be comparng two competng models, the bfactor and undmensonal models. To determne whch model best represents the sample data the followng four technques wll be dscussed: 1) comparson of model ft statstcs, ) rato of varance accounted for by the general factor over the varance accounted for by the specfc factors, 3) the degree to whch total scores reflect a common varable, and 4) the vablty of reportng subscale scores as ndcated by subscale relablty. An emprcal example s provded followng a dscusson of these four technques. Tradtonally wthn the SEM framework, model ft statstcs are employed to determne the adequacy of a model. For example, to determne the ft of confrmatory models, heurstc gudelnes are appled to popular ndces, such as CFI, TLI, RMSEA, and SRMR. After obtanng model ft for both undmensonal and bfactor models, one can drectly compare the two competng models va the change n CFI (ΔCFI) ndex as the undmensonal model s herarchcally nested wthn the bfactor model (Rese, 01). Ths ndex s generally preferred to the tradtonal Chsquare dfference test as CFI has been demonstrated to provde stable performance wth varous condtons, such as sample sze, amount of nvarance, number of factors, and number of tems (Meade, Johnson, & Braddy, 008). In contrast, the Ch-square statstc s notorously known for beng hghly senstve to sample sze. The CFI s calculated as: CFI = CFI M1 - CFI MO (1) where CFI M1 s equal to the CFI value obtaned for model 1, and CFI MO s equal to the CFI value obtaned for model 0. Based on smulaton analyses, Cheung and Rensvold (00) have recommended that a CFI.01 supports the nvarance hypothess. Ths approach for assessng whether data are undmensonal enough s qute popular wthn the SEM framework (Cook & Kallen, 009). However, such an approach does not shed lght on the amount of varance accounted for by the general factor over that accounted for by the specfc factors nor does t provde nformaton regardng the vablty of reportng a composte score or separate scores on the specfc factors. Use of ft ndces lmts one s assessment of determnng the techncal adequacy of reportng multdmensonal scores that may be adequately represented by a undmensonal model. Ths asserton s reflected n recent work by Rese, Schenes, Wdaman, and Havland (013) who have demonstrated that use of ft ndces to determne whether data are undmensonal enough s not optmal f the data have a multdmensonal bfactor structure. Ths research llustrated that f tem response data are bfactor, and those data are forced nto a undmensonal model, parameter bas (partcularly n structural parameters that depend on loadng bas) s a functon of the expected common varance (ECV) and percentage of uncontamnated correlatons (PUC), whereas model ft ndces are a poor ndcator of parameter bas. ECV, whch provdes a rato of the strength of the general to group factors, s defned as follows: I G =1 G ECV = I G + s1 + s + + I s n I s I s G 1 =1 =1 =1 =1 sn () where I G = total number of tems loadng onto the general factor, I s1 = the number of tems loadng on specfc factor 1, I s = the number of tems loadng on specfc factor, I sn = the number of tems loadng on specfc factor n, λ = the squared factor loadngs of the G general factor, λ S 1 = the squared factor loadngs of specfc factor 1, λ S = the squared factor loadngs of specfc factor, and λ S n = the squared factor loadngs of specfc factor n. As the ECV value ncreases to 1, there s evdence to suggest that a strong general dmenson s present n the bfactor data. Although ths value can be used as an ndex of undmensonalty, 111

5 ts nterpretaton s moderated by PUC. That s, PUC moderates the effects of factor strength on basng effects when applyng a undmensonal model to bfactor data (Rese, Schenes, Wdaman, & Havland, 013). PUC can be defned as the number of uncontamnated correlatons dvded by the number of unque correlatons: I G I G 1 I s 1 I s1 1 + Is I 1 s + + I s n I sn 1 PUC = I G ( I G 1) (3) where I G = the number of tems loadng on the general factor, I s1 =the number of tems loadng on specfc factor 1, I s = the number of tems loadng on specfc factor, I sn = the number of tems loadng on specfc factor n. When PUC values are very hgh (>.90), unbased undmensonal estmates can be obtaned even when one obtans a low ECV value (Rese, 01). More specfcally, when the PUC values are very hgh, the factor loadngs of the undmensonal model wll be close to those obtaned on the general factor n the bfactor model. In addton to ECV and PUC values, researchers can compute relablty coeffcents to determne f composte scores predomnately reflect a sngle common factor even when the data are bfactor. As noted by Rese (01), the presence of multdmensonalty does not dctate the creaton of subscales nor does t run the nterpretablty of a unt-weghted composte score. Instead, researchers must make the dstncton between the degree of undmensonalty and the degree to whch total scores reflect a common varable. Ths latter assessment can be accomplshed by computng coeffcent omega herarchcal, whch s defned as: H = ( G ) + S1 ( G ) + ( S ) + + ( Sn ) + where λ G = the factor loadng for tem on the general factor, λ S1 = the factor loadng for tem on specfc factor 1, λ S = the factor loadng for tem on specfc factor, λ Sn = the factor loadng for tem on specfc factor n, and θ = the error varance for tem. Large ω H values ndcate that composte scores prmarly reflect a sngle varable, thus provdng evdence that reportng a undmensonal score s vable. Lastly, f ths evaluaton proves to be nconclusve one can compute the relablty of subscale scores once controllng for the effect of the general factor. Ths relablty coeffcent, whch Rese (01) termed omega subscale (ω s ), can be computed as follows: ( Sn ) s = ( G ) + Sn + Hgh values ndcate that the subscales provde relable nformaton above and beyond the general factor, whereas low values suggest that the subscales are not precse ndcators of the specfc factors. (4) (5) An llustraton To llustrate the basc concepts of usng CFA to assess nternal structure and model selecton, we examned a survey measurng student engagement (SE). The survey was comprsed of 7 four-pont Lkert-type tems and was admnstered to 1,900 partcpants. Based on theory and prevous research, the survey was hypotheszed to measure four latent varables: self-management of learnng (SML), applcaton of learnng strateges (ALS), support of classmates (SC), and self-regulaton of arousal (SRA). Nne of the tems loaded on SML, ten tems loaded on ALS, sx tems loaded on SC, and three tems loaded on SRA. All four latent varables as well as the measurement errors were expected to be uncorrelated n the measurement model. Alternatvely, a undmensonal model was also ft to the sample data to determne whether the general student engagement dmenson could account for the majorty of the score varance. Parameter estmaton was conducted n Mplus, verson 5 (Muthén & Muthén, 007) applyng the weghted least squares wth mean and varance adjustment (WLSMV) estmator to mprove parameter estmaton wth categorcal data. Adequate model ft was represented by CFI and TLI values >.95, as well as an RMSEA value <.06 (Hu & Bentler, 1999). Table 4 provdes the standardzed factor loadng estmates for both the undmensonal and bfactor models. Results demonstrated nadequate model ft to the sample data for the undmensonal model as ndcated prmarly by a small CFI value, CFI=.80, TLI=.97, and RMSEA=.07. In contrast, model ft was drastcally mproved when fttng the bfactor model, CFI=.94, TLI=.99, RMSEA=.04, and a ΔCFI ndex of.14. Examnaton of the factor loadngs (Table 1) demonstrated statstcally sgnfcant factor loadngs of moderate strength for tems 7 and 9 on SML, tems 10, 1, and 16 on ALS, tems 1, 0,, and 7 on SC, and all tems on SRA. These fndngs suggest that the specfc factors accounted for a sgnfcant amount of varance for many of the tems above and beyond the varance accounted for by the general factor. Based solely on model ft statstcs, one would conclude that the data were not undmensonal enough and that a bfactor model best represented the sample data for the models evaluated. However, as mentoned before, model ft statstcs do not provde nformaton related to the parameter bas that comes about by representng bfactor data wth a undmensonal representaton. The frst step n examnng parameter bas that s brought about by applyng a undmensonal model to bfactor data s to evaluate ECV and PUC. In ths example, the sum of the squared factor loadngs was 9.13, 0.40, 0.63, 0.91, and 0.73 for the SE, SML, ALS, SC, and SRA factors, respectvely (see Table 1). Applyng these values to equaton, ECV was calculated as follows: 9.13 ECV = =.79 (6) The results demonstrated that the rato of the strength of the general to group factors was.79, whch suggested that a very strong general factor was present. However, as mentoned, the nterpretaton of ECV s medated by PUC. In ths example, the number of unque correlatons was [(7 6)/] = 351. As there were 8, 10, 6, and 3 tems that loaded on each specfc factor, respectvely, the number of correlatons for tems wthn group 11

6 Valdty evdence based on nternal structure factors was [((8 7)/) + ((10 9)/)+ ((6 5)/)+ ((3 )/)] = 91. Therefore, the number of uncontamnated correlatons was = 60, and the proporton of uncontamnated correlatons was 60/351=.74, whch s moderate-hgh wth extreme values beng represented by anythng >.90. Although the PUC value was not as hgh as one would hope, ts value s dependent on the number of group factors. For example, hgher PUC values would be obtaned f ncreasng the number of group factors from 3 to 9, whch would produce [((3 )/) 9]= 7 uncontamnated correlatons and a proporton of uncontamnated correlatons of (34/351)=.9. Nevertheless, n comparng the factor loadngs between the general factor from the bfactor model and the undmensonal factor loadngs there was a hgh degree of smlarty, r=.88, whch demonstrated that the varance accounted for by the general factor was mpacted mnmally wth the ncluson of the specfc factors (see Table 1). Such a fndng Table 1 Factor loadngs for undmensonal and bfactor models Undmensonal Bfactor Item λ SE λ SE λ SML λ ALS λ SC λ SRA θ ( λ ) ( λ) Note: λ SE = factor loadng for the student engagement factor, λ SML = factor loadng for the self-management of learnng factor, λ ALS = factor loadng for the applcaton of learnng strateges factor, λ SC = factor loadng for the support of classmates factor, λ SRA = factor loadng for the self-regulaton of arousal factor, and θ = tem resdual varance (only reported for bfactor model due to relablty coeffcent calculatons) <.05 n combnaton wth the ECV and PUC results suggested that a strong general factor was present n the bfactor data. The next step was to evaluate the degree to whch a total score reflected a common varable. Ths was accomplshed by frst computng the squared sums of the factor loadngs, whch were 44.61, 1.30, 4.37, 4.58, and 1.90 for the SE, SML, ALS, SC, and SRA factors, respectvely. In addton, the sum of the resdual varance across all 7 tems was equal to These values were then appled to equaton 5 as follows: H = =.90 (7) The results demonstrated an omega herarchcal of.90, whch suggested that a very hgh amount of the varance n summed scores could be attrbuted to the sngle general factor. The last step of the analyss was to compute the relablty of the subscales by controllng for the general factor varance. The omega subscale relabltes were calculated for the four specfc factors as follows: 1.30 SML = =.004 (8) 4.37 ALS = =.0 (9) 4.58 SC = =.0 (10) 1.90 SRA = =.007 (11) As can be seen, the relabltes of the scores for the specfc factors after controllng for the varance accounted for by the general factor were extremely low. Such low relablty estmates demonstrate that reportng scores on the specfc factors would provde unrelable nformaton. In summarzng the results of assessng the undmensonal and bfactor models tested n ths example, one would conclude that although unque factors assocated wth the ndvdual scales were present, the nformaton that they provded was of neglgble consequence. That s, from a practcal standpont, reportng multdmensonal scores would be nvald as the techncal adequacy was lackng, due to a strong general factor, a hgh amount of varance beng accounted for n summed scores by the general factor, and extremely low relablty estmates for scores on the specfc factors. Ths example demonstrates the need for researchers to go beyond the use of model ft statstcs n decdng whether to employ a multdmensonal representaton as often a undmensonal model can be more adequate. Assessng measurement nvarance One socetal concern related to measurement s the lack of test farness for dstnct subgroups wthn the populaton. Although the evaluaton of farness ncorporates legal, ethcal, poltcal, phlosophcal, and economc reasonng (Camll, 006), from 113

7 a psychometrc perspectve, one can defne farness as a lack of systematc bas (measurement nvarance). Bas s a techncal term that comes about when there are systematc defcences n the test that lead to dfferental nterpretaton of scores by subgroup. From ths perspectve, the man concern n evaluatng bas s to determne whether knowledge of an examnee s group membershp nfluences the examnee s score on the measured varable (e.g., an tem, subdoman, or test), gven the examnee s status on the latent varable of nterest (Mllsap, 011). If group membershp s found to mpact score-based nferences, one would conclude that the measure contans construct-rrelevant varance. If not, one would conclude that the measure demonstrates equvalence (nvarance) across subgroups. Therefore, for a test to be far (from a psychometrc perspectve) one must demonstrate measurement nvarance across all dstnctve subgroups beng evaluated. Ths asserton s reflected n Standard 7.1 of the 1999 Standards, whch states: the same forms of valdty evdence collected for the examnee populaton as a whole should also be collected for each relevant subgroup (Amercan Educatonal Research Assocaton, Amercan Psychologcal Assocaton, & Natonal Councl on Measurement n Educaton, 1999, p. 80). There are numerous statstcal approaches for assessng measurement nvarance. These methods can be categorzed nto three dstnctve groups: 1) lnear measurement models, ) non-lnear measurement models, and 3) observed score methods (Mllsap, 011). Furthermore, these approaches can be broken down nto methods that examne nvarance at the scale- and temlevels (Zumbo, 003). Scale-level analyses are prmarly concerned wth the degree of nvarance observed wthn common factor analytc models across groups. In contrast, tem-level analyses (dfferental tem functonng (DIF)) evaluate nvarance for each tem ndvdually. The lterature on DIF s extensve and spans more than 40 years. As a result, the man focus of ths secton wll be on descrbng the multple-group confrmatory factor analytc method for assessng nvarance at the scale-level. For a general ntroducton to DIF, as well as the varous methods avalable for analyzng tem-level nvarance, the reader s referred to Srec and Ros (013). Multple Group Conf rmatory Factor Analyss (MGCFA) MGCFA s a theory-drven method used to evaluate formal hypotheses of parameter nvarance across groups (Dmtrov, 010). MGCFA s advantageous to use when establshng construct comparablty as t allows for: 1) smultaneous model fttng across multple groups, ) varous levels of measurement nvarance can be assessed, 3) the means and covarances of the latent constructs are dsattenuated (.e., controls for measurement error), and 4) drect statstcal tests are avalable to evaluate cross-group dfferences of the estmated parameters (Lttle & Slegers, 005). Conductng MGCFA requres a number of herarchcal steps, whch depend on the desred nferences that the researcher s nterested n. These herarchcal steps can be descrbed as frst establshng a baselne model separately for each group, and then systematcally evaluatng herarchcally nested models to determne the level of nvarance present across groups. Ths systematc process s known as sequental constrant mposton as model parameters across groups are allowed to be freely estmated wth greater constrants on the parameters beng placed as adequate model ft for less restrcted models s obtaned. Comparson of herarchcally nested models can be conducted va the CFI ndex. Levels of measurement nvarance There are varous levels of nvarance; however, for the purposes of ths paper, we wll only dscuss confgural, metrc, scalar, and strct factoral nvarance; however, t should be noted that there are other forms of equvalence, such as nvarance of temunqueness (See Dmtrov, 010). The most basc and necessary condton for group comparsons s confgural nvarance, whch assesses whether there s conceptual equvalence of the underlyng varable(s) across groups (Vandenberg & Lance, 000). From a factor analytc perspectve, confgural nvarance s reflected n the use of dentcal ndcators to measure the same latent construct(s) of nterest across groups. A more restrctve form of nvarance s metrc equvalence, whch assumes both confgural nvarance and equvalent strengths between the ndcators and latent varable (factor loadngs) across groups. Attanment of metrc equvalence denotes equal measurement unts of the scale desgned to measure the latent construct across groups. Ths form of equvalence allows for ndrect comparsons as the score ntervals are equal across groups but the measurement unts do not share the same orgn of the scale. As a result, drect comparsons of group means are not vald. To make drect comparsons of latent group means, t s necessary to attan scalar equvalence. Ths form of nvarance subsumes both confgural and metrc equvalence, as well as assumes that the scales of the latent construct possess the same orgn, whch s ndcated by equal ntercepts across groups. Lastly, when one s concerned wth the equvalence of covarances among groups for a number of latent factors wthn the model, strct factoral nvarance s of nterest. For a detaled example of conductng a scale-level measurement nvarance analyss, the reader s referred to Dmtrov (010). Relablty: Internal consstency Internal consstency relablty represents the reproducblty of test scores on repeated test admnstratons takng under the same condtons and s operatonally defned as the proporton of true score varance to total observed score varance (Crocker & Algna, 1986). Although there are several methods for estmatng relablty of a composte or subscale score such as splt-half relablty, coeffcent α (Cronbach, 1951) s arguably the most commonly used statstc. Cronbach (1951) demonstrated that coeffcent α s the average of all possble splt-half relablty values for a test and s computed as follows: I ˆ = I I 1 1 =1 s s x (1) I represents the number of tems; s represents the varance of scores for tem ; and s represents the test score varance. x 114

8 Valdty evdence based on nternal structure Despte the wdespread use of coeffcent α, t s not wthout ts lmtatons. For example, n most cases when the measurement errors are uncorrelated (except for the tau-equvalent condton), coeffcent α wll often underestmate relablty (Crocker & Algna, 1987; Lord & Novck, 1968). When the measurement errors are correlated, for example due to method effects or tems that share a common stmulus, coeffcent α can ether underestmate or overestmate relablty (Raykov, 001). To address these lmtatons, CFA can be used to provde a more accurate estmate of relablty. Relablty can be estmated from the parameter estmates n a CFA model as follows: Y = + VAR( ) + COV, j, j (13) λ represents the unstandardzed factor loadng; VAR(δ ) represents the measurement error varance; and COV(δ,δ j ) represents the covarance n measurement errors. Essentally, the numerator represents true score varance and equals the squared sum of the unstandardzed factor loadngs. The denomnator Table Factor loadngs to llustrate relablty estmaton Item Unstandardzed λ Standardzed λ represents the total observed score varance and ncludes the true score varance, error varance and any non-zero correlated measurement errors. To llustrate how to compute relablty usng a CFA model, we utlzed the factor loadngs for a sx tem subscale (see Table ). The relablty for a subscale can be computed usng the model parameter estmates. For example, for ths one subscale, the true varance equals the squared sum of the unstandardzed factor loadngs: ( ) = 4.11 (14) The total varance of the subscale s = 6.81 (15) Therefore, the relablty estmate based on the CFA model s 4.11/6.81 = In comparson to coeffcent α, whch equaled 0.80 for the subscale, the relablty estmate based on the CFA model parameter estmates was larger most lkely because the tau equvalence condton was not met. Concluson The need to gather evdence that supports the valdty of score-based nferences s mperatve from scentfc, ethcal, and legal perspectves. In ths artcle we provded a general revew of methodologcal procedures to evaluate one form of valdty evdence, nternal structure, by specfcally focusng on assessment of dmensonalty, measurement nvarance, and relablty wthn a factor analytc framework. In addton, an overvew of the bfactor model, as well as technques that go beyond ft ndces for determnng model selecton, was llustrated. The methods outlned n ths paper, when appled approprately, wll assst researchers n gatherng evdence to strengthen the valdty of ntended scored-based nferences. References Amercan Educatonal Research Assocaton, Amercan Psychologcal Assocaton, & Natonal Councl on Measurement n Educaton (1999). The standards for educatonal and psychologcal testng ( nd ed.). Washngton, DC: Amercan Educatonal Research Assocaton. Arfn, W.N., Yusoff, M.S.B., & Nang, N.N. (01). Confrmatory factor analyss (CFA) of USM Emotonal Quotent Inventory (USMEQ-) among medcal degree program applcants n Unverst Sans Malaysa (USM). Educaton n Medcne Journal, 4(), e1-e. Bollen, K.A. (1989). Structural equaton models wth latent varables. New York, NY: Wley. Brown, T.A. (006). Conf rmatory factor analyss for appled research. New York, NY: Gulford Press. Camll, G. (006). Test Farness. In R.L. Brennan (Ed.), Educatonal measurement (4th ed.) (pp. 1-56). Westport, CT: Amercan Councl on Educaton/Praeger. Campbell, D.T., & Fske, D.W. (1959). Convergent and dscrmnant valdaton by multtrat-multmethod matrx. Psychologcal Bulletn, 56, Chen, F.F., Hayes, A., Carver, C.S., Laurenceau, J.P., & Zhang, Z. (01). Modelng general and specfc varance n multfaceted constructs: A comparson of the bfactor model to other approaches. Journal of Personalty, 80(1), Chen, F.F., Sousa, K.H., & West, S.G. (005). Testng measurement nvarance of second-order factor models. Structural Equaton Modelng, 1(3), Chen, F.F., West, S.G., & Sousa, K.H. (006). A comparson of bfactor and second-order models of qualty of lfe. Multvarate Behavoral Research, 41(), Cheung, G.W., & Rensvold, R.B. (00). Evaluatng goodness-of-ft ndexes for testng measurement nvarance. Structural Equaton Modelng, 9(), Cook, K.F., & Kallen, M.A. (009). Havng a ft: Impact of number of tems and dstrbuton of data on tradtonal crtera for assessng IRT s undmensonalty assumptons. Qualty of Lfe Research, 18, Dmtrov, D.M. (010). Testng for factoral nvarance n the context of construct valdaton. Measurement and Evaluaton n Counselng and Development, 43(), Fukuhara, H., & Kamata, A. (011). A bfactor multdmensonal tem response theory model for dfferental tem functonng analyss on testlet-based tems. Appled Psychologcal Measurement, 35(8), Holznger, K.J., & Swneford, F. (1937). The b-factor method. Psychometrka,,

9 Assessng stress n cancer patents: A second-order factor analyss model for the Perceved Stress Scale. Assessment, 11(3), Hu, L., & Bentler, P.M. (1999). Cutoff crtera for ft ndexes n covarance structure analyss: Conventonal crtera versus new alternatves. Structural Equaton Modelng, 6(1), Jennrch, R.I., & Bentler, P.M. (011). Exploratory b-factor analyss. Psychometrka, 76(4), Jeon, M., Rjmen, F., & Rabe-Hesketh, S. (013). Modelng dfferental tem functonng usng a generalzaton of the multple-group bfactor model. Journal of Educatonal and Behavoral Statstcs, 38(1), Kenny, D.A. (1976). An emprcal applcaton of confrmatory factor analyss to the multtrate-multmethod matrx. Journal of Expermental Socal Psychology, 1, Klne, R.B. (010). Prncples and practce of structural equaton modelng (3 rd ed.). New York, NY: Gulford Press. Krause, N., & Hayward, R.D. (013). Assessng stablty and change n a second-order confrmatory factor model of meanng n lfe. Journal of Happness Studes, Levant, R.F., Hall, R.J., & Rankn, T.J. (013). Male Role Norms Inventory- Short Form (MRNI-SF): Development, confrmatory factor analytc nvestgaton of structure, and measurement nvarance across gender. Journal of Counselng Psychology, 60(), L, Y., & Lsstz, R.W. (01). Explorng the full-nformaton bfactor model n vertcal scalng wth construct shft. Appled Psychologcal Measurement, 36(1), 3-0. Lttle, T.D., & Slegers, D.W. (005). Factor analyss: Multple groups wth means. In B. Evertt & D. Howell (Eds.), Encyclopeda of statstcs n behavoral scence (pp ). Chchester, UK: Wley. Lu, Y., & Thssen, D. (01). Identfyng local dependence wth a score test statstc based on the bfactor logstc model. Appled Psychologcal Measurement, 36(8), Marsh, H.W. (1989). Confrmatory factor analyses of multtratmultmethod data: Many problems and a few solutons. Appled Psychologcal Measurement, 13, Martel, M.M., Roberts, B., Gremllon, M., von Eye, A., & Ngg, J.T. (011). External valdaton of bfactor model of ADHD: Explanng heterogenety n psychatrc comorbdty, cogntve control, and personalty trat profles wthn DSM-IV ADHD. Journal of Abnormal Chld Psychology, 39(8), Meade, A.W., Johnson, E.C., & Braddy, P.W. (008). Power and senstvty of alternatve ft ndces n tests of measurement nvarance. Journal of Appled Psychology, 93(3), Mllsap, R.E. (011). Statstcal approaches to measurement nvarance. New York, NY: Routledge. Muthén, L.K., & Muthén, B.O. (007). Mplus user s gude (5 th ed.). Los Angeles, CA: Muthén & Muthén. Rese, S.P. (01). The redscovery of bfactor measurement models. Multvarate Behavoral Research, 47(5), Rese, S.P., Moore, T.M., & Havland, M.G. (010). Bfactor models and rotatons: Explorng the extent to whch multdmensonal data yeld unvocal scale scores. Journal of Personalty Assessment, 9(6), Rese, S.P., Schenes, R., Wdaman, K.F., & Havland, M.G. (013). Multdmensonalty and structural coeffcent bas n structural equaton modelng: A bfactor perspectve. Educatonal and Psychologcal Measurement, 73(1), 5-6. Salguero, M.F., Smth, P.W.F., & Vera, M.D.T. (013). A mult-process second-order latent growth curve model for subjectve well-beng. Qualty & Quantty: Internatonal Journal of Methodology, 47(), Srec, S.G., & Ros, J.A. (013). Decsons that make a dfference n detectng dfferental tem functonng. Educatonal Research and Evaluaton, 19(-3), Thomas, M.L. (01). Rewards of brdgng the dvde between measurement and clncal theory: Demonstraton of a bfactor model for the Bref Symptom Inventory. Psychologcal Assessment, 4(1), Thompson, B. (004). Exploratory and conf rmatory factor analyss. Washngton, DC: Amercan Psychologcal Assocaton. Vandenberg, R.J., & Lance, C.E. (000). A revew of synthess of the measurement nvarance lterature: Suggestons, practces, and recommendatons for organzatonal research. Organzatonal Research Methods, 3(1), Wang, M., Schalock, R.L., Verdugo, M.A., & Jenaro, C. (010). Examnng the factor structure and herarchcal nature of the qualty of lfe construct. Amercan Journal on Intellectual and Developmental Dsabltes, 115(3), Wolf, E.J., Mller, M.W., & Brown, T.A. (011). The structure of personalty dsorders n ndvduals wth posttraumatc stress dsorder. Personalty Dsorders: Theory, Research, and Treatment, (4), Xe, J., B, Q., L, W., Shang, W., Yan, M., Yang, Y., Mao, D., & Zhang, H. (01). Postve and negatve relatonshp between anxety and depresson of patents n pan: A bfactor model analyss. Plos ONE, 7(10). Zumbo, B.D. (003). Does tem-level DIF manfest tself n scale-level analyses? Implcatons for translatng language tests. Language Testng, 0(),