# Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide

Save this PDF as:

Size: px
Start display at page:

Download "Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide"

## Transcription

2 Practical Assessment, Research & Evaluation, Vol 17, No 3 Page 2 estimation of a test s reliability, in turn, might potentially entail some misinformed inferences, such as discarding a test due to its seemingly low reliability. Purpose of the paper In this paper, we provide a tutorial for when and how to calculate ordinal reliability coefficients rather than non-ordinal coefficients, such as Cronbach s alpha for the very common scenario that one s data come from measurements based on ordinal response scales (e.g., rating scales or Likert-type response formats). We focus on ordinal alpha, which was introduced by Zumbo, Gadermann, and Zeisser (2007), and which was shown to estimate reliability more accurately than Cronbach s alpha for binary 1 and ordinal response scales. We focus on alpha because it is the most widely used reliability coefficient, and because it is useful to use a familiar scenario as a concrete example. We note, however, that our discussion applies to other reliability coefficients as well. In other words, the rationale for using an ordinal version of a reliability coefficient is not restricted to alpha, but is equally valid for other reliability coefficients, such as McDonald s omega or Revelle s beta (please see Zinbarg et al. (2005) for a description of the omega and beta reliability coefficients). The main purpose of this paper is to (i) provide researchers and practitioners with the psychometric and conceptual rationale for when, why, and how to use ordinal reliability coefficients, (ii) present an empirical example that illustrates the degree to which Cronbach s alpha and its ordinal equivalent can differ, and (iii) present step-by-step practical instructions and an example for how to compute ordinal alpha and other reliability coefficients, such as ordinal beta and ordinal omega, in the freely available software package R ( Coefficient alpha for ordinal data: psychometric rationale Ordinal alpha is conceptually equivalent to Cronbach s alpha. The critical difference between the two is that ordinal alpha is based on the polychoric correlation matrix, described in detail below, rather than the Pearson covariance matrix, and thus more accurately estimates alpha for measurements involving ordinal data. 1 A special case of coefficient alpha is KR-20, which is computed from binary data. In general, the computation of coefficient alpha involves the matrix of correlations or covariances among all items of a scale. For Cronbach s alpha, the Pearson covariance matrix is routinely used; for example, as a default in statistical software programs, such as SPSS and SAS. An important assumption for the use of Pearson covariances is that data are continuous, and if this assumption is violated, the Pearson covariance matrix can be substantively distorted (e.g., Flora & Curran, 2004). In social science measurement, it is very common to use the Likert-type item response format. (For example, respondents are asked to indicate their level of agreement with an item by choosing one of a given number of ordered response categories, e.g., with five categories ranging from strongly agree to strongly disagree.) The data arising from such items are not continuous, but ordinal; however, they are often treated as if they were continuous; that is, they are treated as if the data had been measured on an interval scale with desired distributional properties (p. 443, Olsson, 1979). It has been shown that the Pearson correlation coefficient severely underestimates the true relationship between two continuous variables when the two variables manifest themselves in a skewed distribution of observed responses. A tetrachoric/polychoric correlation, on the other hand, more accurately estimates the relationship of the underlying variables (Carroll, 1961). Accordingly, for ordinal data, the method of choice is to use the polychoric correlation matrix. Based on this argument, Zumbo et al. (2007) introduced a coefficient alpha for ordinal data ordinal alpha that is derived from the polychoric correlation matrix. It needs to be noted that ordinal alpha, in line with the longstanding psychometric tradition of interpreting ordinal responses as manifestations of an underlying variable, is focusing on the reliability of the unobserved continuous variables underlying the observed item responses. Using a polychoric matrix for calculating alpha thus represents an underlying variable approach to covariance modeling of ordinal data. That is, when using a polychoric correlation matrix, an item s observed responses are considered manifestations of respondents exceeding a certain number of thresholds on that item s underlying continuum. Conceptually, the idea is to estimate the thresholds and model the observed crossclassification of response categories via the underlying continuous item response variables. Formally, the observed ordinal response for item j with C response categories, where the response option c = 0, 1, 2,, C-1, is defined by the underlying variable y* such that

3 Practical Assessment, Research & Evaluation, Vol 17, No 3 Page 3 y j = c if τ c < y j < τ c+ 1, where τ c, τ c + 1 are the thresholds on the underlying continuum, which are typically spaced at non-equal intervals and satisfy the constraint = τ 0 < τ1 < L < τ c 1 < τ c =. The underlying distribution does not necessarily have to be normally distributed, although it is commonly assumed so due to its well understood nature and beneficial mathematical properties (cf. Liu, Wu, & Zumbo, 2010). In summary, ordinal reliability coefficients may differ from their non-ordinal counterparts because of their scaling assumptions. The non-ordinal coefficients focus on the reliability of the observed scores by treating the observed item responses as if they were continuous, whereas the ordinal coefficients focus on the reliability of the unobserved continuous variables underlying the observed item responses. In this way, the ordinal coefficients are nonparametric reliability coefficients in a nonlinear classical test theory sense (Lewis, 2007). Review of findings from a simulation study Zumbo et al. (2007) present findings from a simulation study that compared ordinal alpha and Cronbach s alpha for all combinations of the following conditions: (i) The theoretical reliability of a test was simulated so that it was.4,.6,.8, or.9. As Zumbo et al. (2007) note, the theoretical reliability was determined in reference to the underlying continuum of variation. (ii) The number of response options of the items was set to 2, 3, 4, 5, 6, or 7. (iii) The amount of skewness of the data was set to 0 or 2. For all conditions, the number of items (p) was set to 14. We reanalyzed the data from Zumbo et al. s (2007) simulation study, which invoked a paradigm introduced by Zumbo and Zimmerman (1993), specifying the underlying continuous scale as the reference for the theoretical reliability. Figure 1 illustrates the degree of underestimation of the theoretical reliability by * Cronbach s alpha and ordinal alpha, for those different conditions, respectively. In Figure 1, the degree to which ordinal alpha as well as Cronbach s alpha accurately estimate or underestimate the theoretical reliability of the underlying variable is illustrated in terms of an attenuation index, which is calculated by the following formula (Equation 1): Percent attenuation = [100 * (alpha theoretical reliability) / theoretical reliability)] In equation 1, alpha denotes either ordinal alpha or Cronbach s alpha. When alpha is equal to the theoretical reliability, the term becomes zero, indicating no attenuation. The more alpha diverges from the theoretical reliability, the closer the term gets to (-100), which would indicate the highest possible degree of attenuation. The graphs in Figure 1 indicate that ordinal alpha provides a suitable estimate of the theoretical reliability, regardless of (i) the magnitude of the theoretical reliability, (ii) the number of scale points, and (iii) the skewness of the scale point distributions. The accuracy of Cronbach s alpha, on the other hand, decreases (i) as the skewness of the scale items increases, (ii) as the number of response options becomes smaller, and (iii) as the theoretical reliability of the scale is lower. The findings from the simulation study thus corroborate the general psychometric recommendation to use a polychoric correlation matrix for ordinal data, and they indicate that ordinal alpha is an unbiased estimator of the theoretical reliability for ordinal data (at least for scenarios like or similar to those tested in the simulation study). If one assumes that the observed item responses are manifestations of a continuous underlying item response variable, particular care should be taken in the interpretation of Cronbach s alpha, especially when one has very few item response options and/or highly skewed observed item responses. In those cases, Cronbach s alpha is a substantially attenuated estimate of the lower bound of the reliability of the underlying item response variables, whereas ordinal alpha tends to estimate this reliability more accurately as the polychoric correlations correct for the attenuation caused by the scaling of the items (cf. Carroll, 1961).

4 Practical Assessment, Research & Evaluation, Vol 17, No 3 Page 4 Figure 1: Percent attenuation of Cronbach s alpha and ordinal alpha Illustration of an example We now provide an example that illustrates the potential for a large discrepancy between ordinal alpha and Cronbach s alpha, using real data. The data are from a sample of 43,644 kindergarten children (48% girls; M age = 5.7 years), and were collected with the Early Development Instrument (EDI; Janus & Offord, 2007). The EDI is a teacher-administered rating scale with 103 ordinal/binary items on children s developmental status in Kindergarten (see also Guhn, Janus, Hertzman, 2007; Guhn, Zumbo, Janus, & Hertzman, 2011). For our example, we used data from the physical independence subscale of the EDI. This subscale is composed of 3 binary items, which are scored as 0 (no) and 1 (yes). From a statistical point of view, the binary case can be thought of as a special case of ordinal data, and therefore the same methods apply here. Table 1 shows the means, standard deviations, skewness, and kurtosis for the 3 items. Table 2 shows the Pearson correlations, Pearson covariances, and polychoric/tetrachoric correlations for the three items, the average correlations/covariance, and the respective alphas. Table 3 shows the factor loadings, communalities, and uniquenesses for the 3 items, and the alphas, calculated based on a factor analysis model, and using the matrix of the (i) Pearson correlations, (ii) Pearson covariances, or (ii) polychoric correlations 2, 2 We note that the term polychoric correlation refers to all correlations based on ordinal variables that measure an (assumably) continuous underlying variable. In the special case that this involves two dichotomous/binary variables, the term tetrachoric correlation is used. Because the tetrachoric correlation is a special case of the polychoric correlation, calculating a polychoric correlation for binary variables is, in fact, equivalent to calculating a tetrachroic correlation; see Uebersax, 2006.

8 Practical Assessment, Research & Evaluation, Vol 17, No 3 Page 8 coefficients) or test-retest reliability by using the polychoric correlation with the respective equations. For future research, it will be of particular interest to better understand the interdependent, interacting effects that a scale s number of items, number of item response options, skewness, kurtosis, and factor structure have on ordinal alpha and Cronbach s alpha, and the discrepancy between them. Our data suggest a diminishing return model with regard to the number of items and number of response options, and also indicate that item skewness is associated with the attenuation of Cronbach s alpha. However, the exact nature of the multivariate relationship between these factors remains to be determined. We would like to conclude with a note of caution and with an endorsement of a unitary, holistic approach to validation. In a recent review (Cizek, Rosenberg, & Koons, 2008), it was found that a majority of articles in the social sciences that report on the validity of tests rely on none (7%), one (29%), or two (33%) sources of evidence for validity. Cronbach s alpha is the most commonly reported piece of validity evidence for tests (reported in 76% of the cases). This practice is not in line with current recommendations provided by the large scientific and professional associations in the psychological and educational fields (e.g., American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999). Furthermore, such practice is not in line with current scholarly thinking in the areas of reliability analysis, generalizability theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972; see also Brennan, 2001; Shavelson & Webb, 1991; Shavelson, Webb, & Rowley, 1989), and holistic perspectives on validity theory (e.g., Lissitz, 2009; Zumbo, 2007). Rather, a unitary, holistic perspective on validity emphasizes the importance (i) of uncovering and understanding multiple sources of measurement variance, and (ii) of validating the interpretations, meanings, inferences, and social consequences that are attributed to or based on measurement scores. In line with this thinking, we recommend using ordinal reliability coefficients for binary and Likert-type and mixed response data as one of several sources of information on a scale s reliability and validity. References American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association. Bentler, P. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, doi: /s Bernaards, C. A., & Jennrich, R. I. (2005) Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement, 65, doi: / Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag. Carroll, J. B. (1961). The nature of data, or how to choose a correlation coefficient. Psychometrika, 26, doi: /BF Cizek, G. J., Rosenberg, S., & Koons, H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, doi: / Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3 rd ed.). Maywah, NJ: Erlbaum. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, doi: / Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, doi: /BF Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, doi: / Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York: Wiley. Elosua Oliden, P., & Zumbo, B. D. (2008). Coeficientes de fiabilidad para escalas de respuesta categoŕica ordenada. Psicothema, 20, Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, doi: / X Ford, J. K., MacCallum, R. C., & Tait, M. (1986). The application of exploratory factor-analysis in applied psychology a critical review and analysis. Personnel Psychology, 39, doi: /j tb00583.x

11 Practical Assessment, Research & Evaluation, Vol 17, No 3 Page 11 Appendix B Syntax for calculating ordinal alpha and other ordinal reliability coefficients in R library(psych) # This activates the R package 8 psych (Revelle, 2011) for the current session in R. data(bfi) # This loads the dataset bfi contained in the R package psych. attach(bfi) # This attaches the dataset bfi to the current session in R. bfi5items<-data.frame(n1,n2,n3,n4,n5) # This creates a new dataset, labeled bfi5items, containing only five (ordinal) variables, N1 to N5, of the original 15-item dataset bfi. describe(bfi5items) # This describes the dataset bfi5items, providing descriptives, such as n, mean, sd, min, max, range, skew, and kurtosis. bfi5items # This displays the object/dataset called bfi5items. polychoric(bfi5items) # This provides the polychoric correlation matrix for the dataset bfi5items. cor(bfi5items, y=null, use="complete.obs", method=c("pearson )) cov(bfi5items, y=null, use="complete.obs", method=c("pearson )) skew(bfi5items) kurtosi(bfi5items) scree(bfi5items) examplename<-polychoric(bfi5items) alpha(examplename\$rho) alpha(bfi5items) # This calculates the Pearson correlation matrix for the dataset bfi5items, only taking into account cases with complete data ( complete.obs ). # This calculates the Pearson method covariance matrix for the dataset bfi5items, only taking into account cases with complete data ( complete.obs ). # This provides the skewness for all items in the bfi5items dataset. # This provides the kurtosis 9 for all items in the bfi5items dataset. # This provides the scree plots of the eigenvalues for a factor analysis and a principal component analysis for the dataset bfi5items. # This saves the polychoric correlation matrix, and corresponding tau values, under the name examplename. You may choose any name to save the matrix. (Note: R will not produce any output for this step.) # This provides (raw and standardized) alpha, and corresponding item statistics, based on the data set or matrix that is specified in brackets. (The \$rho command specifies that only the correlation matrix is used for the calculation, disregarding the tau values that are saved in conjunction with the matrix.) In the output of this calculation, alpha represents ordinal alpha, because it is based on the polychoric correlation matrix for the bfi5items dataset saved under the name examplename. One should obtain the following results as part of the R output: raw_alpha =.84; std.alpha =.84; average_r =.51. (Please note that raw alpha and standardized alpha are the same when they are calculated from a correlation matrix.) # This provides raw/cronbach s and standardized alpha of the object specified in brackets. In this case, the object is a data 8 As stated in the previous section, a package in R has to be installed once first, before it can be loaded for a current session in R. Please refer to Appendix A. 9 Please note that, in R, the command for kurtosis is spelled without the final s (i.e.: kurtosi).

12 Practical Assessment, Research & Evaluation, Vol 17, No 3 Page 12 matrix (bfi5items), and R therefore calculates raw/cronbach s and standardized alpha, respectively, from the Pearson covariance and the Pearson correlation matrices of the data set. This step, in combination with the previous one, will allow one to compare ordinal alpha with raw/cronbach s alpha. One should obtain the following results as part of the R output: raw_alpha =.81; std.alpha =.81; average_r =.47. fa(bfi5items) # This provides the factor loadings (MR1), communalities (h 2 ), and uniquenesses (u 2 ) for a 1-factor solution of the bfi5items data matrix. fa(examplename\$rho) # This provides the factor loadings (MR1), communalities (h 2 ), and uniquenesses (u 2 ) for a 1-factor solution of the polychoric correlation matrix that was saved under the name examplename. guttman(examplename\$rho) # This provides alternative estimates of reliability for the data matrix that is specified in brackets (i.e., examplename\$rho). In the R output, these estimates are labeled as beta, Guttman bounds L1, L2, L3 (alpha), L4 (max), L5, L6 (smc), TenBerge bounds mu0, mu1, mu2, mu3, alpha of the first PC (=principal component), and the estimated greatest lower bound based upon communalities. Since the specified data matrix is, in this case, a polychoric correlation matrix, all the reliability estimates represent ordinal versions. (We note that the guttman syntax command includes alpha (=L3) as one of the reliability estimates however, the alpha syntax command provides additional item characteristics, such as the item-total correlations, that may be of interest to the user.) [Further details and references with regard to the different reliability coefficients featured in the guttman command can be found in Revelle, 2011.] guttman(bfi5items) # Equivalent to the command above, this provides a list of alternative estimates of reliability for the data matrix specified in brackets. Since bfi5items is a raw data matrix, the reliability estimates represent, in this case, Pearson correlation based reliability estimates. omega(examplename\$rho) # This provides the ordinal versions of the reliability coefficients omega (hierarchical, asymptotic, and total), because their calculation is based on the polychoric correlation matrix examplename. omega(bfi5items) # This provides omega coefficients for the data matrix bfi5items. (For details, see Revelle, 2011.) Bolded font indicates necessary steps, and regular font indicates steps that are optional, but will help to obtain commonly requested information in the context of calculating ordinal reliability coefficients.

13 Practical Assessment, Research & Evaluation, Vol 17, No 3 Page 13 Citation: Gadermann, Anne M., Guhn, Martin & Bruno D. Zumbo (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation, 17(3). Available online: Acknowledgement We would like to thank Dr. John Fox and Dr. William Revelle, developers of the R software package, for their responses, giving advice on calculations and syntax for polychoric correlations in R. Also, Martin Guhn gratefully acknowledges the financial support provided to him in the form of a postdoctoral research fellowship from the Michael Smith Foundation for Health Research, British Columbia. Bruno Zumbo wishes to acknowledge support from the Social Sciences and Humanities Research Council of Canada (SSHRC) and the Canadian Institutes of Health Research (CIHR) during the preparation of this work. Authors: Anne M. Gadermann Harvard Medical School 180 Longwood Avenue Boston, MA AnneGadermann [at] googl .com Martin Guhn The Human Early Learning Partnership University of British Columbia Suite 440, 2206 East Mall Vancouver, BC, V6T 1Z3, Canada Martin.Guhn [at] ubc.ca Bruno D. Zumbo, Corresponding Author University of British Columbia Scarfe Building, 2125 Main Mall Vancouver, B.C. CANADA V6T 1Z Bruno.Zumbo [at] ubc.ca

### Using R and the psych package to find ω

Using R and the psych package to find ω William Revelle Department of Psychology Northwestern University February 23, 2013 Contents 1 Overview of this and related documents 1 2 Install R and relevant packages

### Installing R and the psych package

Installing R and the psych package William Revelle Department of Psychology Northwestern University August 17, 2014 Contents 1 Overview of this and related documents 2 2 Install R and relevant packages

### Internal Consistency: Do We Really Know What It Is and How to Assess It?

Journal of Psychology and Behavioral Science June 2014, Vol. 2, No. 2, pp. 205-220 ISSN: 2374-2380 (Print) 2374-2399 (Online) Copyright The Author(s). 2014. All Rights Reserved. Published by American Research

### Four Assumptions Of Multiple Regression That Researchers Should Always Test

A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to Practical Assessment, Research & Evaluation. Permission is granted to distribute

### Richard E. Zinbarg northwestern university, the family institute at northwestern university. William Revelle northwestern university

psychometrika vol. 70, no., 23 33 march 2005 DOI: 0.007/s336-003-0974-7 CRONBACH S α, REVELLE S β, AND MCDONALD S ω H : THEIR RELATIONS WITH EACH OTHER AND TWO ALTERNATIVE CONCEPTUALIZATIONS OF RELIABILITY

### Correlation Coefficients: Mean Bias and Confidence Interval Distortions

Journal of Methods and Measurement in the Social Sciences Vol. 1, No. 2,52-65, 2010 Correlation Coefficients: Mean Bias and Confidence Interval Distortions Richard L. Gorsuch Fuller Theological Seminary

### What is Rotating in Exploratory Factor Analysis?

A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment, Research & Evaluation. Permission is granted to

### An introduction to using Microsoft Excel for quantitative data analysis

Contents An introduction to using Microsoft Excel for quantitative data analysis 1 Introduction... 1 2 Why use Excel?... 2 3 Quantitative data analysis tools in Excel... 3 4 Entering your data... 6 5 Preparing

### FACTORANALYSISOFCATEGORICALDATAINSAS

FACTORANALYSISOFCATEGORICALDATAINSAS Lei Han, Torsten B. Neilands, M. Margaret Dolcini University of California, San Francisco ABSTRACT Data analysts frequently employ factor analysis to extract latent

### How to report the percentage of explained common variance in exploratory factor analysis

UNIVERSITAT ROVIRA I VIRGILI How to report the percentage of explained common variance in exploratory factor analysis Tarragona 2013 Please reference this document as: Lorenzo-Seva, U. (2013). How to report

### Exploratory Factor Analysis

Exploratory Factor Analysis ( 探 索 的 因 子 分 析 ) Yasuyo Sawaki Waseda University JLTA2011 Workshop Momoyama Gakuin University October 28, 2011 1 Today s schedule Part 1: EFA basics Introduction to factor

### Coefficients alpha, beta, omega and the glb: comments on Sijtsma

Psychometrika (submitted 9/9/08) Coefficients alpha, beta, omega and the glb: comments on Sijtsma William Revelle Department of Psychology Northwestern University Richard E. Zinbarg Department of Psychology

### Stephen du Toit Mathilda du Toit Gerhard Mels Yan Cheng. LISREL for Windows: PRELIS User s Guide

Stephen du Toit Mathilda du Toit Gerhard Mels Yan Cheng LISREL for Windows: PRELIS User s Guide Table of contents INTRODUCTION... 1 GRAPHICAL USER INTERFACE... 2 The Data menu... 2 The Define Variables

### Data analysis process

Data analysis process Data collection and preparation Collect data Prepare codebook Set up structure of data Enter data Screen data for errors Exploration of data Descriptive Statistics Graphs Analysis

### Reliability Analysis

Measures of Reliability Reliability Analysis Reliability: the fact that a scale should consistently reflect the construct it is measuring. One way to think of reliability is that other things being equal,

### Calculating, Interpreting, and Reporting Cronbach s Alpha Reliability Coefficient for Likert-Type Scales

2003 Midwest Research to Practice Conference in Adult, Continuing, and Community Education Calculating, Interpreting, and Reporting Cronbach s Alpha Reliability Coefficient for Likert-Type Scales Joseph

### SPSS: Getting Started. For Windows

For Windows Updated: August 2012 Table of Contents Section 1: Overview... 3 1.1 Introduction to SPSS Tutorials... 3 1.2 Introduction to SPSS... 3 1.3 Overview of SPSS for Windows... 3 Section 2: Entering

### TIMSS 2011 User Guide for the International Database

TIMSS 2011 User Guide for the International Database Edited by: Pierre Foy, Alka Arora, and Gabrielle M. Stanco TIMSS 2011 User Guide for the International Database Edited by: Pierre Foy, Alka Arora,

### University of Oklahoma

A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to Practical Assessment, Research & Evaluation. Permission is granted to distribute

### An Applied Comparison of Methods for Least- Squares Factor Analysis of Dichotomous Variables Charles D. H. Parry

An Applied Comparison of Methods for Least- Squares Factor Analysis of Dichotomous Variables Charles D. H. Parry University of Pittsburgh J. J. McArdle University of Virginia A statistical simulation was

### Overview of Factor Analysis

Overview of Factor Analysis Jamie DeCoster Department of Psychology University of Alabama 348 Gordon Palmer Hall Box 870348 Tuscaloosa, AL 35487-0348 Phone: (205) 348-4431 Fax: (205) 348-8648 August 1,

### Simple Predictive Analytics Curtis Seare

Using Excel to Solve Business Problems: Simple Predictive Analytics Curtis Seare Copyright: Vault Analytics July 2010 Contents Section I: Background Information Why use Predictive Analytics? How to use

### A Review of Methods. for Dealing with Missing Data. Angela L. Cool. Texas A&M University 77843-4225

Missing Data 1 Running head: DEALING WITH MISSING DATA A Review of Methods for Dealing with Missing Data Angela L. Cool Texas A&M University 77843-4225 Paper presented at the annual meeting of the Southwest

### Data exploration with Microsoft Excel: univariate analysis

Data exploration with Microsoft Excel: univariate analysis Contents 1 Introduction... 1 2 Exploring a variable s frequency distribution... 2 3 Calculating measures of central tendency... 16 4 Calculating

### CHAPTER 4 EXAMPLES: EXPLORATORY FACTOR ANALYSIS

Examples: Exploratory Factor Analysis CHAPTER 4 EXAMPLES: EXPLORATORY FACTOR ANALYSIS Exploratory factor analysis (EFA) is used to determine the number of continuous latent variables that are needed to

### What is Effect Size? effect size in Information Technology, Learning, and Performance Journal manuscripts.

The Incorporation of Effect Size in Information Technology, Learning, and Performance Research Joe W. Kotrlik Heather A. Williams Research manuscripts published in the Information Technology, Learning,

### Correlation and Regression

Dublin Institute of Technology ARROW@DIT Books/Book Chapters School of Management 2012-10 Correlation and Regression Donal O'Brien Dublin Institute of Technology, donal.obrien@dit.ie Pamela Sharkey Scott

### Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization. Learning Goals. GENOME 560, Spring 2012

Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization GENOME 560, Spring 2012 Data are interesting because they help us understand the world Genomics: Massive Amounts

### PSYCHOLOGY Vol. II - The Construction and Use of Psychological Tests and Measures - Bruno D. Zumbo, Michaela N. Gelin, Anita M.

THE CONSTRUCTION AND USE OF PSYCHOLOGICAL TESTS AND MEASURES Bruno D. Zumbo, Michaela N. Gelin and Measurement, Evaluation, and Research Methodology Program, Department of Educational and Counselling Psychology,

### Early Childhood Measurement and Evaluation Tool Review

Early Childhood Measurement and Evaluation Tool Review Early Childhood Measurement and Evaluation (ECME), a portfolio within CUP, produces Early Childhood Measurement Tool Reviews as a resource for those

### Categorization and measurement quality. The choice between Pearson and Polychoric correlations

7. Chapter Categorization and measurement quality. The choice between Pearson and Polychoric correlations Germà Coenders and Willem E. Saris * ABSTRACT This chapter first reviews the major consequences

### Factorial Invariance in Student Ratings of Instruction

Factorial Invariance in Student Ratings of Instruction Isaac I. Bejar Educational Testing Service Kenneth O. Doyle University of Minnesota The factorial invariance of student ratings of instruction across

### Terminating Sequential Delphi Survey Data Collection

A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment, Research & Evaluation. Permission is granted to

### A Brief Introduction to SPSS Factor Analysis

A Brief Introduction to SPSS Factor Analysis SPSS has a procedure that conducts exploratory factor analysis. Before launching into a step by step example of how to use this procedure, it is recommended

### An introduction to LISREL 8.80 for Windows

An introduction to LISREL 8.80 for Windows 1. Introduction LISREL 8.80 for Windows (Jöreskog & Sörbom 2006) is a Windows software product for structural equation modeling, multilevel structural equation

### Getting Started with HLM 5. For Windows

For Windows August 2012 Table of Contents Section 1: Overview... 3 1.1 About this Document... 3 1.2 Introduction to HLM... 3 1.3 Accessing HLM... 3 1.4 Getting Help with HLM... 4 Section 2: Accessing Data

### Introduction Course in SPSS - Evening 1

ETH Zürich Seminar für Statistik Introduction Course in SPSS - Evening 1 Seminar für Statistik, ETH Zürich All data used during the course can be downloaded from the following ftp server: ftp://stat.ethz.ch/u/sfs/spsskurs/

### Two SPSS programs for interpreting multiple regression results. Universitat Rovira i Virgili, Tarragona, Spain

Behavior Research Methods 2010, 42 (1), 29-35 doi:10.3758/brm.42.1.29 Two SPSS programs for interpreting multiple regression results Urbano Lorenzo-Seva, Pere J. Ferrando, and Eliseo Chico Universitat

### Impact of Skewness on Statistical Power

Modern Applied Science; Vol. 7, No. 8; 013 ISSN 1913-1844 E-ISSN 1913-185 Published by Canadian Center of Science and Education Impact of Skewness on Statistical Power Ötüken Senger 1 1 Kafkas University,

### Calculating, Interpreting, and Reporting Estimates of Effect Size (Magnitude of an Effect or the Strength of a Relationship)

1 Calculating, Interpreting, and Reporting Estimates of Effect Size (Magnitude of an Effect or the Strength of a Relationship) I. Authors should report effect sizes in the manuscript and tables when reporting

### PROPERTIES OF THE SAMPLE CORRELATION OF THE BIVARIATE LOGNORMAL DISTRIBUTION

PROPERTIES OF THE SAMPLE CORRELATION OF THE BIVARIATE LOGNORMAL DISTRIBUTION Chin-Diew Lai, Department of Statistics, Massey University, New Zealand John C W Rayner, School of Mathematics and Applied Statistics,

### Using Minitab for Regression Analysis: An extended example

Using Minitab for Regression Analysis: An extended example The following example uses data from another text on fertilizer application and crop yield, and is intended to show how Minitab can be used to

### A Brief Introduction to Factor Analysis

1. Introduction A Brief Introduction to Factor Analysis Factor analysis attempts to represent a set of observed variables X 1, X 2. X n in terms of a number of 'common' factors plus a factor which is unique

### Reporting and Interpreting Scores Derived from Likert-type Scales

Journal of Agricultural Education, 55(5), 30-47. doi: 10.5032/jae.2014.05030 Derived from Likert-type Scales J. Robert Warmbrod 1 Abstract Forty-nine percent of the 706 articles published in the Journal

### MINITAB ASSISTANT WHITE PAPER

MINITAB ASSISTANT WHITE PAPER This paper explains the research conducted by Minitab statisticians to develop the methods and data checks used in the Assistant in Minitab 17 Statistical Software. One-Way

### Running head: ASSUMPTIONS IN MULTIPLE REGRESSION 1. Assumptions in Multiple Regression: A Tutorial. Dianne L. Ballance ID#

Running head: ASSUMPTIONS IN MULTIPLE REGRESSION 1 Assumptions in Multiple Regression: A Tutorial Dianne L. Ballance ID#00939966 University of Calgary APSY 607 ASSUMPTIONS IN MULTIPLE REGRESSION 2 Assumptions

### What Does the Correlation Coefficient Really Tell Us About the Individual?

What Does the Correlation Coefficient Really Tell Us About the Individual? R. C. Gardner and R. W. J. Neufeld Department of Psychology University of Western Ontario ABSTRACT The Pearson product moment

### UNDERSTANDING THE INDEPENDENT-SAMPLES t TEST

UNDERSTANDING The independent-samples t test evaluates the difference between the means of two independent or unrelated groups. That is, we evaluate whether the means for two independent groups are significantly

### Data exploration with Microsoft Excel: analysing more than one variable

Data exploration with Microsoft Excel: analysing more than one variable Contents 1 Introduction... 1 2 Comparing different groups or different variables... 2 3 Exploring the association between categorical

### How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power

STRUCTURAL EQUATION MODELING, 9(4), 599 620 Copyright 2002, Lawrence Erlbaum Associates, Inc. TEACHER S CORNER How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power Linda K. Muthén

### Running head: ONLINE VALUE AND SELF-EFFICACY SCALE

Online Value and Self-Efficacy Scale Running head: ONLINE VALUE AND SELF-EFFICACY SCALE Development and Initial Validation of the Online Learning Value and Self-Efficacy Scale Anthony R. Artino Jr. and

### Chapter 1 Introduction. 1.1 Introduction

Chapter 1 Introduction 1.1 Introduction 1 1.2 What Is a Monte Carlo Study? 2 1.2.1 Simulating the Rolling of Two Dice 2 1.3 Why Is Monte Carlo Simulation Often Necessary? 4 1.4 What Are Some Typical Situations

### Doing Quantitative Research 26E02900, 6 ECTS Lecture 2: Measurement Scales. Olli-Pekka Kauppila Rilana Riikkinen

Doing Quantitative Research 26E02900, 6 ECTS Lecture 2: Measurement Scales Olli-Pekka Kauppila Rilana Riikkinen Learning Objectives 1. Develop the ability to assess a quality of measurement instruments

### Software Solutions - 375 - Appendix B. B.1 The R Software

Appendix B Software Solutions This appendix provides a brief discussion of the software solutions available to researchers for computing inter-rater reliability coefficients. The list of software packages

### Psyc 250 Statistics & Experimental Design. Correlation Exercise

Psyc 250 Statistics & Experimental Design Correlation Exercise Preparation: Log onto Woodle and download the Class Data February 09 dataset and the associated Syntax to create scale scores Class Syntax

### [This document contains corrections to a few typos that were found on the version available through the journal s web page]

Online supplement to Hayes, A. F., & Preacher, K. J. (2014). Statistical mediation analysis with a multicategorical independent variable. British Journal of Mathematical and Statistical Psychology, 67,

### Psychology 205: Research Methods in Psychology

Psychology 205: Research Methods in Psychology Using R to analyze the data for study 2 Department of Psychology Northwestern University Evanston, Illinois USA November, 2012 1 / 38 Outline 1 Getting ready

### Understanding and Using Factor Scores: Considerations for the Applied Researcher

A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment, Research & Evaluation. Permission is granted to

### AUTISM SPECTRUM RATING SCALES (ASRS )

AUTISM SPECTRUM RATING SCALES ( ) Scoring the for Individuals Who Do Not Speak or Speak Infrequently Sam Goldstein, Ph.D. & Jack A. Naglieri, Ph.D. Technical Report #1 This technical report was edited

### Mathematics within the Psychology Curriculum

Mathematics within the Psychology Curriculum Statistical Theory and Data Handling Statistical theory and data handling as studied on the GCSE Mathematics syllabus You may have learnt about statistics and

### Psychology 405: Psychometric Theory Homework on Factor analysis and structural equation modeling

Psychology 405: Psychometric Theory Homework on Factor analysis and structural equation modeling William Revelle Department of Psychology Northwestern University Evanston, Illinois USA June, 2014 1 / 20

### Rachel J. Goldberg, Guideline Research/Atlanta, Inc., Duluth, GA

PROC FACTOR: How to Interpret the Output of a Real-World Example Rachel J. Goldberg, Guideline Research/Atlanta, Inc., Duluth, GA ABSTRACT THE METHOD This paper summarizes a real-world example of a factor

### IBM SPSS Statistics 20 Part 1: Descriptive Statistics

CALIFORNIA STATE UNIVERSITY, LOS ANGELES INFORMATION TECHNOLOGY SERVICES IBM SPSS Statistics 20 Part 1: Descriptive Statistics Summer 2013, Version 2.0 Table of Contents Introduction...2 Downloading the

### Moderation. Moderation

Stats - Moderation Moderation A moderator is a variable that specifies conditions under which a given predictor is related to an outcome. The moderator explains when a DV and IV are related. Moderation

### An introduction to computing and interpreting Cronbach Coefficient Alpha in SAS Chong Ho Yu, Ph.D., Arizona State University, Tempe, AZ

An introduction to computing and interpreting Cronbach Coefficient Alpha in SAS Chong Ho Yu, Ph.D., Arizona State University, Tempe, AZ ABSTRACT In spite of the ease of computation of Cronbach Coefficient

### Polynomial Neural Network Discovery Client User Guide

Polynomial Neural Network Discovery Client User Guide Version 1.3 Table of contents Table of contents...2 1. Introduction...3 1.1 Overview...3 1.2 PNN algorithm principles...3 1.3 Additional criteria...3

### Reliability and Validity Evidence for the GED English as a Second Language Test

Reliability and Validity Evidence for the GED English as a Second Language Test GED Testing Service Research Studies, 2009-4 A Program of the American Council on Education December 2009 American Council

### Measurement Error 2: Scale Construction (Very Brief Overview) Page 1

Measurement Error 2: Scale Construction (Very Brief Overview) Richard Williams, University of Notre Dame, http://www3.nd.edu/~rwilliam/ Last revised January 22, 2015 This handout draws heavily from Marija

### Introduction to Regression and Data Analysis

Statlab Workshop Introduction to Regression and Data Analysis with Dan Campbell and Sherlock Campbell October 28, 2008 I. The basics A. Types of variables Your variables may take several forms, and it

### MISSING DATA TECHNIQUES WITH SAS. IDRE Statistical Consulting Group

MISSING DATA TECHNIQUES WITH SAS IDRE Statistical Consulting Group ROAD MAP FOR TODAY To discuss: 1. Commonly used techniques for handling missing data, focusing on multiple imputation 2. Issues that could

### Εισαγωγή στην πολυεπίπεδη μοντελοποίηση δεδομένων με το HLM. Βασίλης Παυλόπουλος Τμήμα Ψυχολογίας, Πανεπιστήμιο Αθηνών

Εισαγωγή στην πολυεπίπεδη μοντελοποίηση δεδομένων με το HLM Βασίλης Παυλόπουλος Τμήμα Ψυχολογίας, Πανεπιστήμιο Αθηνών Το υλικό αυτό προέρχεται από workshop που οργανώθηκε σε θερινό σχολείο της Ευρωπαϊκής

### Conducting Exploratory and Confirmatory Factor Analyses for Competency in Malaysia Logistics Companies

Conducting Exploratory and Confirmatory Factor Analyses for Competency in Malaysia Logistics Companies Dazmin Daud Faculty of Business and Information Science, UCSI University, Kuala Lumpur, MALAYSIA.

: Table of Contents... 1 Overview of Model... 1 Dispersion... 2 Parameterization... 3 Sigma-Restricted Model... 3 Overparameterized Model... 4 Reference Coding... 4 Model Summary (Summary Tab)... 5 Summary

### Course Description. Learning Objectives

STAT X400 (2 semester units in Statistics) Business, Technology & Engineering Technology & Information Management Quantitative Analysis & Analytics Course Description This course introduces students to

### Package EstCRM. July 13, 2015

Version 1.4 Date 2015-7-11 Package EstCRM July 13, 2015 Title Calibrating Parameters for the Samejima's Continuous IRT Model Author Cengiz Zopluoglu Maintainer Cengiz Zopluoglu

### Chapter 6: The Information Function 129. CHAPTER 7 Test Calibration

Chapter 6: The Information Function 129 CHAPTER 7 Test Calibration 130 Chapter 7: Test Calibration CHAPTER 7 Test Calibration For didactic purposes, all of the preceding chapters have assumed that the

### January 26, 2009 The Faculty Center for Teaching and Learning

THE BASICS OF DATA MANAGEMENT AND ANALYSIS A USER GUIDE January 26, 2009 The Faculty Center for Teaching and Learning THE BASICS OF DATA MANAGEMENT AND ANALYSIS Table of Contents Table of Contents... i

### Semester 1 Statistics Short courses

Semester 1 Statistics Short courses Course: STAA0001 Basic Statistics Blackboard Site: STAA0001 Dates: Sat. March 12 th and Sat. April 30 th (9 am 5 pm) Assumed Knowledge: None Course Description Statistical

### Minitab Guide. This packet contains: A Friendly Guide to Minitab. Minitab Step-By-Step

Minitab Guide This packet contains: A Friendly Guide to Minitab An introduction to Minitab; including basic Minitab functions, how to create sets of data, and how to create and edit graphs of different

### Applications of Structural Equation Modeling in Social Sciences Research

American International Journal of Contemporary Research Vol. 4 No. 1; January 2014 Applications of Structural Equation Modeling in Social Sciences Research Jackson de Carvalho, PhD Assistant Professor

### Common method bias in PLS-SEM: A full collinearity assessment approach

Common method bias in PLS-SEM: A full collinearity assessment approach Ned Kock Full reference: Kock, N. (2015). Common method bias in PLS-SEM: A full collinearity assessment approach. International Journal

### How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power Linda K. Muthén & Bengt O. Muthén Published online: 19 Nov 2009.

This article was downloaded by: [University of California, Los Angeles (UCLA)] On: 19 February 2014, At: 05:09 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954

### 9.2 User s Guide SAS/STAT. Introduction. (Book Excerpt) SAS Documentation

SAS/STAT Introduction (Book Excerpt) 9.2 User s Guide SAS Documentation This document is an individual chapter from SAS/STAT 9.2 User s Guide. The correct bibliographic citation for the complete manual

### Table 2: Factor loadings from an exploratory factor analysis with 4 factors (using ML estimation and an oblimin rotation)

Appendix 1 Exploratory factor analysis We fitted exploratory factor analysis models with 1 to 6 factors using maximum likelihood estimation and an oblimin (oblique) rotation. Some fit statistics for these

### SCIENTIFIC COMPUTING AND PROGRAMMING IN THE CLOUD USING OPEN SOURCE PLATFORMS: AN ILLUSTRATION USING WEIGHTED VOTING SYSTEMS

SCIENTIFIC COMPUTING AND PROGRAMMING IN THE CLOUD USING OPEN SOURCE PLATFORMS: AN ILLUSTRATION USING WEIGHTED VOTING SYSTEMS Mohamed I Jamaloodeen Georgia Gwinnet College School of Science and Technology

### How to Get More Value from Your Survey Data

Technical report How to Get More Value from Your Survey Data Discover four advanced analysis techniques that make survey research more effective Table of contents Introduction..............................................................2

### Linear Models in STATA and ANOVA

Session 4 Linear Models in STATA and ANOVA Page Strengths of Linear Relationships 4-2 A Note on Non-Linear Relationships 4-4 Multiple Linear Regression 4-5 Removal of Variables 4-8 Independent Samples

### Measuring critical thinking, intelligence, and academic performance in psychology undergraduates

The Irish Journal of Psychology 2009 Vol. 30 No. 3-4 pp. 123-131 Copyright 2009 by The Psychological Society of Ireland ISSN 0303-3910 Measuring critical thinking, intelligence, and academic performance

### DISCRIMINANT FUNCTION ANALYSIS (DA)

DISCRIMINANT FUNCTION ANALYSIS (DA) John Poulsen and Aaron French Key words: assumptions, further reading, computations, standardized coefficents, structure matrix, tests of signficance Introduction Discriminant

### A Introduction to Matrix Algebra and Principal Components Analysis

A Introduction to Matrix Algebra and Principal Components Analysis Multivariate Methods in Education ERSH 8350 Lecture #2 August 24, 2011 ERSH 8350: Lecture 2 Today s Class An introduction to matrix algebra

### 1/27/2013. PSY 512: Advanced Statistics for Psychological and Behavioral Research 2

PSY 512: Advanced Statistics for Psychological and Behavioral Research 2 Introduce moderated multiple regression Continuous predictor continuous predictor Continuous predictor categorical predictor Understand

### SOME NOTES ON STATISTICAL INTERPRETATION. Below I provide some basic notes on statistical interpretation for some selected procedures.

1 SOME NOTES ON STATISTICAL INTERPRETATION Below I provide some basic notes on statistical interpretation for some selected procedures. The information provided here is not exhaustive. There is more to

### How to Use a Data Spreadsheet: Excel

How to Use a Data Spreadsheet: Excel One does not necessarily have special statistical software to perform statistical analyses. Microsoft Office Excel can be used to run statistical procedures. Although

### There are six different windows that can be opened when using SPSS. The following will give a description of each of them.

SPSS Basics Tutorial 1: SPSS Windows There are six different windows that can be opened when using SPSS. The following will give a description of each of them. The Data Editor The Data Editor is a spreadsheet

### The operating characteristics of the nonparametric Levene test for equal variances with assessment and evaluation data

A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment, Research & Evaluation. Permission is granted to

### S P S S Statistical Package for the Social Sciences

S P S S Statistical Package for the Social Sciences Data Entry Data Management Basic Descriptive Statistics Jamie Lynn Marincic Leanne Hicks Survey, Statistics, and Psychometrics Core Facility (SSP) July

### Bill Burton Albert Einstein College of Medicine william.burton@einstein.yu.edu April 28, 2014 EERS: Managing the Tension Between Rigor and Resources 1

Bill Burton Albert Einstein College of Medicine william.burton@einstein.yu.edu April 28, 2014 EERS: Managing the Tension Between Rigor and Resources 1 Calculate counts, means, and standard deviations Produce

### Lecture 2: Descriptive Statistics and Exploratory Data Analysis

Lecture 2: Descriptive Statistics and Exploratory Data Analysis Further Thoughts on Experimental Design 16 Individuals (8 each from two populations) with replicates Pop 1 Pop 2 Randomly sample 4 individuals