The Pennsylvania State University. The Graduate School. Department of Educational Psychology, Counseling, and Special Education

Transcription

1 The Pennsylvania State University The Graduate School Department of Educational Psychology, Counseling, and Special Education THE PERFORMANCE OF MODEL FIT MEASURES BY ROBUST WEIGHTED LEAST SQUARES ESTIMATORS IN CONFIRMATORY FACTOR ANALYSIS A Dissertation in Educational Psychology by Yu Zhao 2015 Yu Zhao Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy May 2015

2 The dissertation of Yu Zhao was reviewed and approved* by the following: Pui-Wa Lei Associate Professor of Education Dissertation Advisor Chair of Committee Hoi K. Suen Distinguished Professor of Educational Psychology Jonna M. Kulikowich Professor of Education Aleksandra Slavkovic Associate Professor of Statistics Robert J. Stevens Professor of Educational Psychology Program Coordinator of Educational Psychology *Signatures are on file in the Graduate School

3 ABSTRACT Despite the prevalence of ordinal observed variables in applied structural equation modeling (SEM) research, limited attention has been given to model evaluation methods suitable for ordinal variables, thus providing practitioners in the field with few guidelines to follow. This dissertation represents a first attempt to thoroughly examine the performance of five fit measures χ 2 statistic, Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean Square Residual (SRMR) produced by the mean- and variance-corrected Weighted Least Squares (WLSMV) estimator from Mplus 7 and the Diagonally Weighted Least Squares (DWLS) estimator from LISREL 9.1, both of which are forms of Robust Weighted Least Squares (RWLS) estimator designed to accommodate ordinal and nonnormal observed variables, in Confirmatory Factor Analysis (CFA) model evaluation, under various realistic sample, data, and model conditions, especially when different types and degrees of model misspecification occur. This study also empirically examined the applicability of the most widely used cut-off criteria of the fit indices proposed by Hu and Bentler (1999) in RWLS estimation with ordinal variables. Results showed that in evaluating the goodness-of-fit of CFA models with ordinal variables, fit measures generated by Mplus WLSMV seemed to be more effective and reliable than those produced by LISREL DWLS across studied conditions. The WLSMV fit measures generally maintained good Type I error control and were powerful enough to detect moderate model misspecification, provided that the model was not too large. The DWLS fit measures, on the other hand, were susceptible to influences of small sample size and could be largely inflated or deflated when a small sample was used to evaluate a large model. In addition, Hu and Bentler s (1999) cut-off criteria, despite of their popularity among applied SEM researchers, were not iii

4 universally applicable in RWLS model evaluation, mainly because all of the fit indices examined varied systematically with the size of the proposed model. Recommendations are made by the end of the dissertation, based on the results of the current study, on practical issues pertaining to reallife CFA model evaluation with ordinal observed variables, such as minimum sample size required and how to use information provided by the RWLS fit measures to make model-data fit decisions, while taking into consideration the sample, data, and model characteristics specific to researchers own studies. iv

5 TABLE OF CONTENTS List of Figures... vii List of Tables... ix Acknowledgements... x Chapter 1 Introduction Background Factors that affect the performance of the model fit measures Estimation methods Model misspecification Sample size and model size Scale and distribution of the observed variables Evaluation of model fit indices Limitations of past research Statement of the problem Hypotheses Significance and contribution of the study Chapter 2 Literature Review The Confirmatory Factor Analysis model Model specification Model estimation Model evaluation Empirical study results of the RWLS-based model fit measures Findings on model fit measures Findings on cut-off criteria for the model fit indices Purpose of the study Chapter 3 Method Design factors Estimation methods Model Size Sample size Level of nonnormality Number of variable categories Model misspecification Data generation Data analysis v

6 Chapter 4 Results Convergence rates The χ 2 statistics CFI TLI RMSEA SRMR Chapter 5 Discussion Mplus WLSMV vs. LISREL DWLS The RWLS fit statistics and indices Model size effect Cut-off criteria for the fit indices Types and degrees of model misspecification Conclusion Recommendations Examples of applying recommendations in practice Example 1: Planning sample size with existing instruments (recommendation #2) Example 2: Using χ 2 statistic with fit indices to justify decision (recommendation #4) Example 3: Treating SRMR with caution (recommendation #5) Limitations and future directions Appendix A: Population Parameters of the Models Appendix B: Examples of Programming Codes Appendix C: Tables of the Convergence Rates, Estimated Mean Values and Rejection Rates of the RWLS Fit Measures Appendix D: Examples of RMSEAs from LISREL 9.1, LISREL 8.8, and Mplus Bibliography vi

7 LIST OF FIGURES Figure 1-1: Percentage distribution of the 50 CFA studies that used 2- to 6+-point categorical variables... 9 Figure 1-2: Percentage distribution of the 50 CFA studies that used different sample sizes Figure 1-3: Percentage distribution of the 50 CFA studies that used different N:t ratios Figure 3-1: Relative frequency distribution of model sizes (# of indicators) from the 50 CFA studies Figure 3-2: Misspecification of the number of factors Figure 3-3: Misspecification of the cross-loadings Figure 3-4: Misspecification of the error correlations Figure 3-5: Misspecification of the factor correlation Figure 4-1: Mplus convergence rates by MTD and sample size Figure 4-2: Mplus convergence rates by variable scale and sample size Figure 4-3: Mplus convergence rates by variable distribution and sample size Figure 4-4: LISREL convergence rates by model size, variable scale, and sample size Figure 4-5: LISREL convergence rates by model size, variable distribution, and sample size Figure 4-6: Mplus chi-square rejection rates by MTD and sample size Figure 4-7: Mplus chi-square rejection rates by model size Figure 4-8: LISREL S-B scaled chi-square rejection rates by MTD and sample size Figure 4-9: LISREL S-B adjusted chi-square rejection rates by MTD and sample size Figure 4-10: Distributions of the Mplus and the LISREL CFIs by studied factors (correct model) Figure 4-11: Mplus CFI by MTD and model size Figure 4-12: LISREL CFI by MTD and model size Figure 4-13: LISREL CFI by sample size vii

8 Figure 4-14: Distributions of the Mplus and the LISREL TLIs by studied factors (correct model) Figure 4-15: Mplus TLI by MTD and model size Figure 4-16: LISREL NNFI by MTD and model size Figure 4-17: LISREL NNFI by sample size Figure 4-18: Distributions of the Mplus and the LISREL RMSEAs by studied factors (correct model) Figure 4-19: Mplus RMSEA by MTD and model size Figure 4-20: LISREL RMSEA by model size and sample size Figure 4-21: LISREL RMSEA by MTD Figure 4-22: Distributions of the Mplus and the LISREL SRMRs by studied factors (correct model) Figure 4-23: Mplus SRMR by sample size Figure 4-24: Mplus SRMR by MTD and model size viii

9 LIST OF TABLES Table 1-1: Summary of 50 CFA Empirical Studies... 4 Table 3-1: Summary of the simulation design Table 3-2: List of cut points used in the study Table 4-1: Proportion of Variance Due to Studied Factors: Convergence Rates Table 4-2: Proportion of Variance Due to Studied Factors: Chi-square Rejection Rates Table 4-3: Proportion of Variance Due to Studied Factors: CFI Table 4-4: Proportion of Variance Due to Studied Factors: TLI Table 4-5: Proportion of Variance Due to Studied Factors: RMSEA Table 4-6: Proportion of Variance Due to Studied Factors: SRMR Table 5-1: Summary of some of the recommendations ix

10 ACKNOWLEDGEMENTS This dissertation would not have been possible without many people. First and foremost, I would like to express my deepest gratitude to my academic and dissertation advisor, and a good friend, Dr. Pui-Wa Lei, for her excellent guidance, support, patience, and encouragement during my entire doctoral study. She not only ignited my interests in education-related methodological and statistical studies, but also saw me through the struggling stages with basic mathematical theories and psychometric concepts. Her guidance was strategic while flexible, allowing me the room to grow as an independent researcher. She believed in my abilities more than I did in myself, and challenged me with higher level academic courses and advanced research topics I would not have otherwise completed. She is also a cheerful friend, who has always been able to look at things from a positive side to enlighten my days. I am so fortunate to have Dr. Lei and could not have asked for a better advisor. I also owe my thanks to my committee members, Dr. Hoi Suen, Dr. Jonna Kulikowich, and Dr. Aleksandra Slavkovic. Dr. Suen has been a role model for my future career. He is not just an accomplished researcher, but also an amazing instructor, who has the ability to enliven abstract methodological theories and concepts, making his measurement classes fun to attend. He is also a supportive and reassuring friend to me. Dr. Kulikowich constantly inspired me with her passion and persistency on connecting methodological studies to real-life educational research. Dr. Slavkovic is my dissertation committee member as well as my master s advisor in applied statistics. She taught me how to be professional and precise not only as an educational researcher, but also as a statistician. Their insightful and constructive feedback on my dissertation and other related research projects, and flexibility to work along a timeline that is most suitable to me are much appreciated. x

11 Special thanks must go to other professors with whom I have worked closely during my doctoral studies, including Dr. Lisa Lenze, Dr. Paul Morgan, and Khanjan Mehta. They have provided me with invaluable opportunities to apply the methodological theories I have learned into real-life practices. They mentored me with their expertise, treated me with patience, and respected me as an immature but independent researcher. It has been my honor to work with them. I am indebted to my family and friends for many reasons. I want to thank my parents for supporting me, and each one of my decisions along the way. They saw my doctoral studies and my dissertation as the upmost important endeavor at this stage of my life, and tried their best to help me with other things so that I could focus on this academic goal. My husband, Kang, has been my biggest moral support. He encouraged me when I felt uninspired, comforted me when I was frustrated, complimented me for each small achievement, and embraced me during every little setback. I would not have come so far without his endless love. And my precious daughter Khloe, who has been so sweet and understanding, brightens up my days and brings so much joy to my life since she was born. My thanks also go to the friends I ve worked with during my doctoral studies and those I have not had a chance to work with but who have always stood by my side. Thank you all for being part of my life. xi

12 To my husband Kang Zhao my daughter Khloe Zhao my parents Zhihong Sun and Qiusheng Zhao and my son who is still in my belly xii

13 Chapter 1 Introduction Background Structural equation modeling (SEM) has been widely used in social and behavioral research to model relationships among multiple variables, observed and/or latent. It is a family of versatile statistical modeling tools that encompass exploratory modeling (e.g., exploring direct effects, Asparouhov & Muthén, 9), confirmatory modeling (e.g., confirming factor structures), cross-sectional modeling (e.g., testing factor invariance across groups), longitudinal modeling (e.g., modeling latent growth over time), and many more. SEM is a covariance structure analysis technique. Covariance structure analysis tests theories with correlated variables that are represented in a system of equations, which describe the unidirectional and bidirectional influences of several variables on each other (Bentler & Bonett, 1980, p.588). At the heart of the covariance structure analysis, as its name indicates, lays the correlations and covariances of the variables. Researchers derive models from substantive theories and fit models to empirical data to see if their models can explain inter-correlations or covariances among variables. Confirmatory factor analysis (CFA) is a member of the general SEM family. The multiple-indicator measurement models as analyzed in CFA represent half the basic rationale of analyzing covariance structures in SEM (Kline, 2011, p.230), with the other half being the analysis of structural models. CFA plays an important role in SEM analysis because many structural equation models include measurement models. As Thompson (4) noted, (i)t makes 1

14 little sense to relate constructs within an SEM model if the factors specified as part of the model are not worthy of further attention (p.110). The method of CFA involves analyzing measurement models in which not only the number of latent factors, but also their specific relations to the indicators are pre-specified based on substantive theories. Theory plays a paramount role in the specification, testing, and interpretation of the models. Because of the theory-driven nature of the predictions, much stronger inferences can be made from confirmatory rather than exploratory models (Curran, 1994). CFA is a popular technique that can be used to test a variety of hypothesis about a set of observed variables. For many applied researchers, it is a primary tool for construct validation (e.g., Gupta, Ganster, & Kepes, 2013; McDermott et al., 2011; Prati, 2012; Van Eck, Finney, & Evans, 2010), scale refinement (e.g., Immekus & Imbrie, 2010; Rowe, Kim, Baker, Kamphaus, & Horne, 2010), assessment of measurement invariance (e.g., Libbrecht, Lievens, & Schollaert, 2010; Randall & Engelhard, 2010; Segeritz & Pant, 2013), or even theory development (e.g., Greiff, Wüstenberg, & Funke, 2012; Kahraman, De Champlain, & Raymond, 2012). It is also used extensively in simulation studies (e.g., Beauducel & Herzberg, 6; DiStefano, 2010; Flora & Curran, 4; Lei, 9; Muthén, du Toit, & Spisic, 1997; Yu, 2, etc.). Most applications of CFA involve five consecutive steps: model specification, identification, estimation, evaluation, and (possibly) modification/respecification. Since CFA is inherently a confirmatory modeling technique, relationships between observed variables and their latent constructs need to be specified a priori, using substantive justifications. Pre-specified CFA models have to be identified so that unique set of parameter estimates is possible when the model is estimated. After data is collected, proper estimation methods need to be selected depending on characteristics of the data and the model. Different estimation methods will all generate a set of fit 2

15 statistics and indices, which can be used to evaluate if the model fits the empirical data well. If the fit of the model as evaluated by fit statistics and indices is less than acceptable, then either the model needs to be modified post hoc, or a completely different model needs to be re-specified. Although all steps involved in CFA application are essential to the success of the analysis, arguably, model estimation and evaluation are more important (Bentler & Bonett, 1980; Hu & Bentler, 1999; Yuan, 5). Traditionally, parameters in structural equation models are estimated by maximum likelihood (ML) estimation and the model-data fit is evaluated by chi-square tests. However, because some of the assumptions required by ML estimation and chi-square tests in SEM, such as multivariate normality of the observed variables and exact fit of the hypothesized model to the population covariance matrix, are not realistic in practice (Bollen, 1989), other estimators (e.g., weighted least squares, or WLS) and fit indices (e.g., approximate fit indices) that have more relaxed assumptions were developed. Ever since Bentler and Bonett (1980) introduced using fit indices 1 to compare and evaluate different models, many fit indices have been developed and examined to serve this purpose. With the advance of such indices, applied SEM researchers started to rely more on fit indices instead of chi-square tests for decisions of model fit. For example, a random review of 50 CFA empirical studies published recently (from ) in major journals of educational and psychological research (e.g., Applied Psychological Measurement, Applied Measurement in Education, Educational and Psychological Measurement, Journal of Applied Psychology, Psychological Assessment, etc., see table 1-1) revealed that all of these papers used information provided by the fit indices (e.g., comparative fit index, root mean square of approximation, etc.) 1 Here the fit indices refers to the approximate fit indices and do not include the χ 2 statistic. The model fit measures can be roughly classified into two categories: (1) model test statistics, such as the χ 2 statistic, which evaluates the proposed model using statistical hypothesis testing; and (2) fit indices, which are considered mainly as descriptive. The current paper will follow this wording convention to distinguish the fit statistic from the fit indices. See Yuan (5) for more details. 3

16 Table 1-1 Summary of 50 CFA Empirical Studies # First author Year Sample size Number of indicators Scale of indicators Estimation method Fit measures Cut-off criteria Software 1. Adelson & 5-point ML χ 2, CFI, TLI, RMSEA Amos 2. Benson ML χ 2, CFI, RMSEA, SRMR, AIC, BIC Hu & Bentler, 1999 Amos 3. Berzonsky point ML χ 2, RMSEA, SRMR Hu & Bentler, 1999 LISREL 4. Bowden /680 ML/MLM χ 2, CFI, TLI, RMSEA, SRMR, Mplus ECVI 5. Brown point ML χ 2, CFI, RMSEA, SRMR Hu & Bentler, 1999 LISREL 6. Burrow- χ & 10 5-point ML, CFI, NFI, Sánchez RMSEA Byrne, 2010 Amos 7. Cai Dichotomous χ DWLS, CFI, RMSEA, & 5-point SRMR, AIC Hu & Bentler, 1999 LISREL 8. Cieslak point ML CFI, RMSEA, SRMR Amos 9. Coates point WLSMV χ 2, CFI, RMSEA Hu & Bentler, 1999 Mplus 10. Curseu point ML χ 2, CFI, TLI, Yuan & Bentler, RMSEA 4 Amos 11. Dedrick point ML CFI, RMSEA, SRMR, BIC Hu & Bentler, 1999 Mplus 12. Dunn point χ 2, CFI, GFI, RMSEA, SRMR Hu & Bentler, 1999 Mplus 4

17 Table 1-1 Continued # First author Year Sample size Number of indicators Scale of indicators Estimation method Fit measures Cut-off criteria Software 13. Esbjørn point WLSMV CFI, TLI, RMSEA Schreiber, Nora, Stage, Barlow, & Mplus King, France point ML χ 2, CFI, RMSEA, SRMR Hu & Bentler, 1999 LISREL 15. Gagné / & 7-point RML χ 2, CFI, RMSEA Y a. 16. Greiff dichotomous WLSMV χ 2, CFI, TLI, RMSEA, Hu & Bentler, 1999 Mplus 17. Gupta point χ 2, RMSEA, SRMR Amos 18. Hardré point χ 2, CFI, RMSEA Schumacker & Lomax, Immekus / point WLSMV χ 2, CFI, RMSEA, SRMR Hu & Bentler, 1999 Mplus 20. Joseph & 50 5-point χ 2, CFI, TLI, RMSEA, SRMR LISREL 21. Kahraman point ML χ 2, CFI, TLI, AIC Y. Mplus 22. Koster point ML GFI, RMSEA Jaccard & Wan, 1996 LISREL χ 23. Kuenssberg point MLM 2, CFI, TLI, Hu & Bentler, 1999 Mplus RMSEA, SRMR 24. Lac point RML χ 2, CFI, IFI, RMSEA Browne & Cudeck, 1993; MacCallum, Browne, & Sugawara, Lakin dichotomous WLSMV χ 2, CFI, RMSEA Kline, 4 Mplus 26. Leite dichotomous WLSMV χ 2, CFI, TLI, RMSEA, SRMR Mplus EQS 5

18 Table 1-1 Continued # First author Year Sample size Number of indicators Scale of indicators Estimation method Fit measures Cut-off criteria Software 27. Libbrecht point RML χ 2, CFI, IFI, RMSEA Y. EQS 28. Little point WLSMV χ 2, CFI, RMSEA, Hu & Bentler, 1999; SRMR Byrne, 6 Mplus 29. Loke point WLSMV χ 2, CFI, TLI, RMSEA Hu & Bentler, 1999 Mplus 30. Martin point RML χ 2, CFI, RMSEA, Browne & Cudeck, SRMR, AIC 1993 Mplus 31. McDermott point CFI, RMSEA Hu & Bentler, Merz point MLM χ 2, RMSEA, SRMR Hu & Bentler, 1999 Mplus 33. Mesmer- χ point RML, CFI, IFI, Magnus RMSEA Hu & Bentler, 1999 EQS 34. Myers point WLSMV χ 2, CFI, TLI, RMSEA, SRMR Mplus 35. NG point χ 2, CFI, NNFI, AGFI, PGFI, PNFI, LISREL RMSEA, SRMR 36. Prati & 5-point WLSMV χ 2, CFI, NFI, RMSEA Mplus 37. Randall / dichotomous WLSMV χ 2, CFI, RMSEA Byrne, 6; Kline, 5 Mplus 38. Rowe point χ 2, CFI, RMSEA, SRMR Hu & Bentler, 1999 Amos 39. Ryser / point RML χ 2, CFI, RMSEA, SRMR Yu & Muthén, 2 LISREL 40. Schroeders dichotomous WLSMV χ 2, CFI, RMSEA, WRMR Yu, 2 Mplus 6

19 Table 1-1 Continued # First author Year 41. Segeritz 2013 Sample size 1/1400 /13000 Number of indicators 42. Su Takishima- Lacasa Scale of indicators Estimation method 45 RML Continuoussummed ML point WLSMV 44. Teo point ML 45. Thomas / point ML 46. Van den Broeck point WLSMV 47. Van Eck point Williams, M. Williams, S /235/ 424 RML/ WLSM/ WLSMV 39 & 26 5-point ML / point 50. Xu / point RML Fit measures Cut-off criteria Software χ 2, CFI, TLI, RMSEA, SRMR χ 2, CFI, IFI, TLI, RMSEA, SRMR, AIC χ 2, CFI, TLI, RMSEA χ 2, CFI, TLI, RMSEA, SRMR χ 2, CFI, RMSEA, SRMR χ 2, CFI, TLI, RMSEA Browne & Cudeck, 3 Hu & Bentler, 1999 Hu & Bentler, 1999; Browne & Cudeck, 1993 Hu & Bentler, 1999 Hu & Bentler, 1999 Hu & Bentler, 1999 χ 2, CFI, RMSEA Yu & Muthén, 2 χ 2, CFI, NNFI, RMSEA, SRMR, AIC χ 2, CFI, RMSEA, SRMR χ 2, CFI, RMSEA, SRMR Note. a This means that the paper employed cut-off values of some sort, but did not specify sources. Schermelleh-Engel et al., 3 Weston & Gore, 6 Hu & Bentler, 1999; Browne & Cudeck, 1993; Byrne, 8 LISREL Mplus Amos LISREL Mplus LISREL/ Mplus SAS EQS 7

20 to support their model fit decisions. Although most of these papers reported the values of the χ 2 statistics and their associated p values, few of them considered the chi-square test as evidence of model-data fit. The advantages of fit indices are that they are single summary measures that are supposed to provide information about the overall model fit. They are easy to understand because many of them vary along the continuum of 0 to 1. However, despite of these advantages, robustness studies discovered that the performance of fit indices can be differentially affected by factors such as the estimation method used, sample size, scale, and distribution of the variables, and other factors that may not have been completely understood by researchers. The instability of a fit index essentially means that it can give different or even contradictory information of model fit when, for example, different estimation methods are employed to analyze the same model, or a same model is tested with samples of different sizes. What makes the matter even more complicated is the evaluation of the fit indices. That is, after researchers obtain fit indices from SEM analysis softwares, how do they know if the number means a close fit or not? There are some general guidelines being offered (e.g., Bentler & Bonnett, 1980; Browne & Cudeck, 1993; Hu & Bentler, 1999; Marsh & Hau, 1996; Rigdon, 1996; Yu, 2) as to the ranges of values that constitute an acceptable model fit for different fit indices. These guidelines are employed in most of the SEM applied studies published in major educational and psychological journals (i.e., 41 out of the 50 papers reviewed used cut-off values for decision making, see table 1-1). However, only a few simulation studies have systematically examined the appropriateness of these proposed cut-off values under limited sets of conditions. With such a small number of simulated conditions, the external validity of the proposed cut-off values is questionable. 8

21 In social and behavioral science research, it is common that a set of ordinal-scaled items (e.g., items with five categories: 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree) is used to measure one or more psychological constructs. For example, the 50 CFA empirical studies reviewed showed that 82% of these randomly chosen studies analyzed ordinal variables with five or fewer categories (see figure 1-1). These ordinally scaled items result in observed variables that are coarse and crude categorization of the latent continuous variables (Finney & DiStefano, 2013). While Finney and DiStefano (2013) pointed out that all ordinal variables are inherently non-continuous, whether or not a variable with many categories can be treated as continuous in analysis seems to be a subjective matter (e.g., the default of LISREL is to treat observed variables with 15 or fewer categories as ordered categorical variables). All in all, in many SEM applications, variables with five or fewer categories are considered ordinal instead of continuous, and ML estimation is not recommended for analyzing ordinal variables (Finney & DiStefano, 2013). 47% 21% 18% 12% 2% 2-point 3-point 4-point 5-point 6+point Figure 1-1. Percentage distribution of the 50 CFA studies that used 2- to 6+-point categorical variables. 9

22 The mean- and variance-corrected weighted least squares (WLSMV) and the diagonally weighted least squares (DWLS) are estimation techniques developed to accommodate ordinal variables without the computational intensity of the full WLS estimation. The WLSMV is implemented in Mplus (Muthén & Muthén, ) and the DWLS is built in in LISREL (Jöreskog & Sörbom, 2012). Both estimators are forms of robust weighted least squares (RWLS), which does not make any distributional assumptions. RWLS is commonly used with polychoric correlations to better estimate associations of the latent response variables underlying categorical observed variables. The WLSMV and the DWLS differ in the weight matrices they employ and the calculation of the χ 2 statistics (which is elaborated in chapter 2). As a result, the fit indices produced by the two estimators may also vary, though the values are most likely to be asymptotically equivalent (Muthén, 5, 6). Initial research findings about the two estimators are encouraging. The WLSMV produced χ 2 statistic that had Type I error (i.e., rejecting a correctly specified model) rates close to the nominal level of 5% with relatively small samples (e.g., N > 250 when the model was not too complex), when the model was correctly specified (Bandalos, 8; Beauducel & Herzberg, 6; Flora & Curran, 4; Lei, 9; Muthén et al., 1997). The DWLS was also a significant improvement over the WLS in terms of convergence rates, and the DWLS χ 2 statistic had acceptable Type I error rates when the sample size was smaller than that required by the WLS (DiStefano, 2010; Yang-Wallentin, Jöreskog, & Luo, 2010). However, due to the fact that the development of the two estimators is relatively recent (in 1997 and 0, respectively, for the WLSMV and the DWLS), studies regarding their performance are rather limited, and some observations have only been preliminary. For example, little is known about the performance of the two estimators when the model is misspecified. Given the large number of applied CFA 10

23 studies that utilize ordinal data, and the high probability that a model is misspecified to some extent in real-life research (Curran, 1994; MacCallum, 1995), much more studies are needed to comprehensively assess the functioning of the two estimators and their associated fit measures (Finney & DiStefano, 2013). Hu and Bentler (1999) once discussed two pressing issues pertaining to model evaluation faced by methodological and applied researchers: The first pressing issue is determination of adequacy of fit indexes under various data and model conditions often encountered in practice. These conditions include sensitivity of fit index to model misspecification, small sample bias, estimation method effect, effects of violation of normality and independence, and bias of fit indexes resulting from model complexity. The second pressing issue is the selection of the rules of thumb conventional cutoff criteria for given fit indexes used to evaluate model fit (p.4). Keeping the two issues in mind, this study sought to gain a better understanding of the performance of five fit measures, χ 2 statistic, Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean Square Residual (SRMR), using the WLSMV and the DWLS estimators, in small samples, large models, and highly nonnormal variables, with correct and misspecified models. In addition, this study also empirically examined the behavior of the most widely used cut-off values proposed by Hu and Bentler (1999), under broader conditions than the ones simulated in the original study, as an effort to investigate the external validity of the cut-off values. 11

24 Factors that affect the performance of the model fit measures Hu and Bentler (1998) pointed out that there are four major problems involved in using fit indices for model evaluation: sensitivity of fit indices to model misspecification, estimation method effect, small-sample bias, and effects of violation of normality and independence (p.427). The first issue is related to the main practical point for using fit indices, that is, the abilities of the fit indices to discriminate well- from badly-fitted models (Maiti & Mukherjee, 1991). The latter three are all natural consequences of the fact that (fit) indices typically are based on chi-square tests (Hu & Bentler, 1998, p.427). As will be discussed briefly below, and with more details in chapter 2, the adequacy of chi-square test may depend on particular assumptions it requires. Violation of these assumptions may affect the performance of a chi-square test, and subsequently the fit indices. This section discusses these possible influences on the model fit measures and how they may relate to model evaluation under RWLS estimation. Estimation methods Choice of estimation methods affects the quality of model parameter estimates, their associated standard error estimates, and overall model fit statistics in CFA modeling (Lei & Wu, 9). There are many estimation methods available for CFA. Each method is derived under a specific set of assumptions that must be met for proper estimation. Violation of these assumptions may affect the performance of the fit measures produced by the estimation methods, thus leading to inaccurate information and invalid conclusions about the fit of the model. The RWLS estimation is based upon full WLS estimation, which is a distribution-free method. Furthermore, the RWLS estimation analyzes polychoric correlations, which take into consideration the categorical nature of the ordinal observed variables. Two implicit assumptions 12

25 about the RWLS estimation are that the model is properly specified and the sample is sufficiently large. When these assumptions are met, the test statistics follow a chi-square distribution. Previous research have found that when the model was properly specified, both WLSMV and DWLS χ 2 statistics had good control of Type I error at a relatively small sample size (between 250 and 400, e.g., Bandalos, 9; DiStefano, 2010; Lei, 9; Yang-Wallentin, 2010). However, no consistent conclusions can be reached when the estimators are used with misspecified models, and little is learned about the performance of the RWLS fit indices. Model misspecification The performance of the RWLS-based χ 2 statistics and fit indices in misspecified models received inadequate amount of attention from methodological researchers. A correctly specified model is one that matches well with the population from which the sample is drawn. A model is said to be misspecified if it is either overparameterized, or underparameterized, or both (Hu & Bentler, 1998). Overparameterized models estimate one or more parameters when their population values are zero, whereas underparameterized models fix one or more parameters to zero when their population values are in fact non-zero. MacCallum (1995) observed that most models specified in applied research are misspecified to some extent. Therefore, simulation study results about model fit measures based on properly specified models may have limited generalizability in practice. A model can be misspecified in many ways. For CFA models, Brown (6) listed three ways that model misspecification can occur in practice: (1) misspecified numbers of factors; (2) misspecified indicators or factor loadings; and (3) misspecified correlated errors. 13

26 Misspecification of the numbers of factors can happen when a researcher either does not have a complete understanding of the latent factors that affect the observed variables, or is not aware of the fact that a set of observed variables are affected by some factors other than those modeled. The former usually takes place when a theory is just forming. For example, when investigating the factor structure of the Psychopathy Checklist: Youth Version (PCL:YV), Jones, Cauffman, Miller, and Mulvey (6) noted that (a)s this review demonstrates, the factor structure of the PCL:YV is far from clear (p.35); and Ward (6) stated that BDI-II factor structure has not, however, been completely consistent in the several investigations that have been reported (p. 81), as he compared different factor structure models for the Beck Depression Inventory--II. When the latent factor structure is unclear to researchers, it is likely that they mistakenly model too many or too few factors. The latter situation occurs when a researcher has overlooked a second factor that is affecting a subset of observed variables he/she studies. For example, the existence of a method effect due to positive and negative wording in self-esteem measurement has been repeatedly documented by several authors (as cited in Bandalos, 8). However, a researcher who is not aware of such a method effect may overlook this factor when he or she analyzes self-esteem questionnaires. If too many factors are specified, it is likely that two or more factors are highly correlated, so that they have poor discriminant validity. Cohen, Cohen, West, and Aiken (3) suggested that a factor correlation of.85 or higher might signify problematic discriminant validity. So misspecification of too many factors can be easily detected by researchers. If too few factors are modeled, CFA will fail to reproduce the observed relationships among the indicators; parameter estimates will be affected, which will in turn affect the theoretical interpretation of the results. However, there is no direct way of knowing it. Modification indices may suggest correlated 14

27 errors but that might not be the real source of mis-fit. Therefore, it is important to know if the overall model fit indices are sensitive enough to pick up this kind of model misspecification. The most common form of misspecification in CFA is omission of cross-loadings, because a standard CFA model assumes no cross-loadings. Subsequently, cross-loadings that exist in the population may be overlooked intentionally or unintentionally in practice. In some situations, a researcher may intentionally ignore cross-loadings when a simple structure CFA model is preferred (e.g., Greiff et al., 2012, deleted all items with cross-loadings from the Space Shuttle measure in order to analyze a clean CFA model). Or a researcher can simply be not aware of the multidimensionality of the items. For example, an item that measures mathematical ability can also assess reading ability. For researchers who analyze achievement tests on which both abilities are required, failing to recognize the effect of reading ability on the success of the mathematical items may cause missing cross-loadings. Suppose the observed variable X 1 loads on two latent factors F 1 and F 2 in the population, simulation study (Brown, 6) showed that omission of a cross-loading (e.g., X 1 F 2 ) may result in inflation of the estimated factor correlation between F 1 and F 2, and the estimated factor loading of X 1 F 1, and underestimation of the other factor loadings on F 1. When these factor loadings and correlation are interpreted, some effects may be overstated while others may be understated. Depending on the magnitude and the importance of the effects, this type of model misspecification can pose serious or nonessential problems in different scientific contexts. In a CFA model, when all error variances are specified to be independent of one another, a researcher is making a claim that all covariations among indicators are accounted for by the set of latent factors in the model and all measurement errors are random. Correlated errors are specified when some of the covariances between indicators are not explained by the latent factors, 15

28 but rather by some other omitted common causes. One such common cause is the method effect discussed earlier. Depending on the purpose of the analysis and the underlying theoretical justification, method effect can be modeled as error covariances among indicators that are measured with the same method instead of as a separate factor (Brown, 6). Correlated error variances can be misspecified in two ways: estimating a non-existent error covariance and omitting an existing error covariance. Empirical examples of such are difficult to document because in practice whether error covariances truly exist is not known. As Brown (6) described, estimating a non-existent error covariance by mistake is readily detectable by non-significant statistical or clinical results (e.g., a non-significant test statistic or a very small parameter estimate of the path). More difficulties reside with detecting salient correlated errors that are mistakenly omitted in the solution. Brown (6) showed that in some cases, this type of misspecification may not be captured by overall model fit measures. Researchers have to look into the standardized residuals and modification indices to catch the mistake. However, in reality, some researchers may not proceed to check out the standardized residuals and the modification indices when the overall model fit indices indicate an acceptable model fit. Consequences of this type of model misspecification include overestimation of the factor loadings of the indicators whose errors are supposed to co-vary, and underestimation of the factor loadings of the other indicators that load on the same factor. Therefore, it is imperative to examine to what extent are the overall model fit measures sensitive to this kind of model misspecification. Another type of model misspecification that is not discussed by Brown (6), but is commonly encountered in personality research, is the misspecification of the factor correlations. In personality psychology, it is commonly believed that the Big Five personality factors (i.e., 16

29 openness, conscientiousness, extraversion, agreeableness, and neuroticism) are independent to one another. So personality researchers specify and test orthogonal factor models of various personality measures. However, it has been repeatedly shown that when the Big Five is operationalized, factors in personality measures are usually correlated to one another (see Saucier, 2, for a more detailed discussion). In these cases, it is important that the fit indices catch such mis-fit of the model to the data, so that researchers are aware of the existence of the factor correlations, whether it is due to imperfect theory or unsatisfactory measurement. Sample size and model size Ding, Velicer, and Harlow (1995) once described a good fit index as being only sensitive to model misspecification, but robust against factors that are irrelevant to the correctness of the specified model (p.120). Sample size and model size are such factors that are irrelevant to the correctness of the specified model. As a large sample technique, SEM s minimum recommended sample size is (Hoogland & Boomsma, 1998; Kline, 5). What makes the matter more complicated is that the minimum sample sizes required often depends on the size of the model tested. For example, cases may seem large enough for a 6-indicator-2-factor CFA model, but will probably not suffice when a 50-indicator-10-factor CFA model is tested. Problems arise when a relatively small sample is used to estimate a large number of parameters. Sampling error may be too large and parameter estimates may not be reliable. The recommended ratio of sample size (N) to number of parameters estimated (t) is at least 10:1 (Hoogland & Boomsma, 1998; Kline, 5; Nunnally, 1967). However, it has been observed that in applied CFA studies, it is not rare that a small sample is used to estimate a relatively large model. Figures 1-2 and 1-3 display the percentage 17

30 distribution of the sample sizes and the N:t ratios used in the 50 applied CFA studies. As can be seen from the graphs, 17% of these studies used a sample smaller than cases and 27% used a ratio of N:t less than 5:1. Indeed, in reality, it is not easy for individual researchers to collect large samples, even when the number of indicators they analyze is decently large. 39% 17% 24% 20% <= > Figure 1-2. Percentage distribution of the 50 CFA studies that used different sample sizes. 41% 27% 32% < 5:1 5:1-10:1 > 10:1 Figure 1-3. Percentage distribution of the 50 CFA studies that used different N:t ratios. 18

31 As discussed earlier, both WLSMV and DWLS are built upon WLS, which requires a very large sample size to provide stable estimates. In addition, both WLSMV and DWLS only produce accurate model test statistics when correct asymptotic variances of the sample correlations are calculated. Although research (Beauducel & Herzberg, 6; Flora & Curran, 4; Yang-Wallentin et al., 2010) have shown that neither WLSMV nor DWLS needed a sample that is as large as required by WLS to perform well, it is very likely that these estimators will not behave properly in the presence of an extremely small sample. Therefore, it is important to find out if the model fit measures produced by WLSMV and DWLS can be trusted when the sample is small relative to the model. Scale and distribution of the observed variables Micceri (1989) observed that most data collected from achievement and other measures are not normally distributed, which is generally true in the field of educational and psychological studies, where ordinal observed variables are often used to measure underlying psychological constructs. Among the 50 CFA studies that are reviewed, most studies that specifically mentioned that they checked distributions of the observed variables reported having nonnormal variables, with some reporting severe nonnormality. Observed variables that are nonnormal or categorical should not post serious threats to RWLS-based fit measures because the RWLS estimation takes into account the categorical nature of the ordinal variables and is distributionfree. However, previous research on RWLS fit statistics and indices showed that they were affected differentially by nonnormal distributions and the scale of the observed variables. For example, observed ordinal variables with more severe nonnormality were found to affect convergence rates of the WLSMV and the DWLS estimators (e.g., DiStefano, 2010; Lei, 9) 19

32 and Type I error control of the overall model fit measures (e.g., Lei, 9; Yu, 2), with higher rates of nonconvergence and inflated Type I error rates associated with more severe nonnormality. Nonnormality and the scale of the observed variables also affected Type II error (i.e., accepting a misspecified model) rates with more Type II errors associated with increasing nonnormality and decreasing numbers of variable categories (Bandalos, 8). Therefore, it would be interesting to find out whether the WLSMV and the DWLS can produce acceptable overall model fit statistics/indices in commonly encountered less than ideal conditions, such as nonnormality and small sample sizes with different types of model misspecifications. Evaluation of model fit indices An important issue that is closely relevant to model assessment is the evaluation of the model fit indices. Hu and Bentler did a highly influential simulation study in 1999, in which they evaluated different cut-off values for a group of better performing fit indices under different data and sample conditions (specifics are discussed in chapter 2) with correct and misspecified models. They proposed a set of viable cut-off values for the fit indices examined, based on the fact that these cut-off values generally resulted in minimal incorrect rejection and acceptance rates 2. They believed that these rule-of-thumb cut-off values would help researchers to arrive at a more objective decision regarding model-data fit (Hu & Bentler, 1999). Although Hu and Bentler (1998, 1999) repeatedly stressed that the application of these cut-off values should be restricted to conditions that are similar to the ones studied in their papers 2 Hu and Bentler called these Type I and Type II error rates. Although strictly speaking, the incorrect rejection and acceptance rates of a fit index are not Type I and Type II error rates as the fit indices are mainly used as descriptive statistics, this study will follow the convention of the field and loosely term the incorrect rejection of a true model as Type I error, the incorrect acceptance of a misspecified model as Type II error, and the correct rejection of a misspecified model as power. 20

33 (mainly, ML estimation and continuous variables), these tentative cut-off values became extremely popular as golden rules in SEM practice. Among the 50 CFA studies reviewed, 41 (82%) specifically claimed that they used cut-off criteria of some sort when making model-fit decisions, of which 23, or 56%, of the decisions were based on Hu and Bentler s (1999) suggested cut-off values, despite the fact that almost all of these studies analyzed ordered categorical data, and 62% of them explicitly reported that an estimation method other than the regular ML estimation was used for the analysis (see table 1-1). On the one hand, researchers do need some criteria to assess the goodness of their proposed model. On the other hand, applying these rule-of-thumb cut-off values as golden rules without considering the sample, data, and model conditions specific to the study may lead to incorrect interpretations. The overgeneralization and overinterpretation of Hu and Bentler s (1999) cut-off criteria may partly be due to the fact that there are few other studies that researched and examined the appropriateness of the cut-off values in conditions that are not covered by Hu and Bentler (1999). Therefore, more research is needed to assess the workability of these cut-off thresholds under different conditions, such as with the RWLS estimation and ordinal variables. Limitations of past research Compared to research on traditional estimation methods such as ML estimation, studies that have looked at the WLSMV and the DWLS estimators and their associated fit measures are rather limited. While some general guidelines (e.g., minimum sample size required, cut-off values for different fit indices, etc.) have been offered for ML estimation, equivalencies have not been established for RWLS estimations. Although a few consistent conclusions can be drawn 21

34 from past research, many questions remain unanswered about these two estimators. This section discusses several limitations in the existing research that the current study attempts to address. First of all, although model misspecification is one of the biggest concerns of applied researchers when they specify and test their models, few simulation studies have incorporated model misspecification as one of their conditions when they examined the performance of the RWLS fit statistics and indices. The sensitivity of the fit measures to model misspecification can be empirically examined in simulation studies using the power of the fit statistic/index. The power of a fit index is the probability that the fit index indicates a bad fit when a model is indeed misspecified. Lei (9) introduced a small dosage of model misspecification in her simulation study examining the performance of WLSMV chi-square test. She found that the empirical power of the WLSMV chi-square test was generally low unless sample size was very large ( ). It was unclear, however, if the low power of the WLSMV chi-square test was inherent to the estimator or just due to the small size of model misspecification. Yu (2) also looked at the performance of the WLSMV chi-square test and other fit indices when the model was misspecified. Contrary to Lei (9), Yu (2) found that the power of the WLSMV-base fit statistic and indices was high, especially when the misspecification was regarding omitted factor covariances. However, when the misspecification was regarding omitted cross-loadings, the power of the fit measures was low. Bandalos (8) also attempted to address the issue of model misspecification with the WLSMV estimation. She examined the power of the χ 2 statistic, CFI, and RMSEA when two latent factors in the population were specified to be one in the sample. She found very low power associated with the WLSMV fit statistic and indices, and the power seemed to decrease as the variable nonnormality increased. Overall, it is unclear what might have caused the high and the low power of the RWLS fit measures in these different studies. Is it the 22