Quantitative Research in Education

Transcription

1 Quantitative Research in Education Intermediate & Advanced Methods DIMITER M. DIMITROV George Mason University New York

2 Published by Whittier Publications, Inc. Oceanside, NY Tel: TEXT (8398) Cover Design: Zheko Aleksiev Copyright 2008 Dimiter M. Dimitrov All Rights Reserved. ISBN No parts of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of the publisher. Printed in the United States of America

3 PREFACE This book offers a comprehensive presentation of quantitative research design and statistical methods in the context of education and related fields. The text is intended primarily for use by students who take intermediate and advanced quantitative research courses as a part of their graduate degree program, but it can be a useful resource for researchers in education, counseling, rehabilitation, psychology, sociology, social work, and human development as well. The main purpose of this book is to provide the readers with an in-depth conceptual and methodological understanding of intermediate and advanced quantitative research methods, as well as the skills necessary to apply such methods using SPSS and to interpret the results. This is achieved by building layers of context-based understanding of research concepts and methods, their statistical translation, methodological principles, computer-based data analysis, presentation of the results in APA style format, and contextual interpretations. The text allows people who experience difficulties with analytic representations of statistical concepts to capitalize on conceptual understanding and still be able to master the research tools necessary for their work on theses, dissertations, and professional research. While there are many excellent introductory books on research design and statistics in education and the social sciences, most books at the intermediate and advanced levels tend to be either too technical and mathematical or too simplistic. Typically, claiming to have an "applied orientation," such books are dominated by presentations of SPSS dialog boxes and printouts at the expense of theoretical and methodological rigor. To bridge the gap between these extremes, this book attempts to provide a balance between conceptual meaning and its statistical translation by developing understanding and application skills in a spiral exposure to quantitative concepts and methods. For example, the comparison of groups on variables of interest is addressed in a sequence from univariate cases of t-tests, nonparametric methods, and analysis of variance (ANOVA) to scenarios illustrating the use of multivariate analysis of variance (MANOVA) and structural equation modeling (SEM). As another example, the concept of validity is addressed in the framework of measurement, research design, and structural equation modeling. Particular attention is devoted to potential problems associated with violation of assumptions, common misconceptions (e.g., conducting MANOVA versus separate ANOVAs), effect sizes, confidence intervals, and sample size. The book is organized in four parts comprising 24 chapters. Each chapter ends with a summary and study questions. Part I [Measurement in Educational Research] consists of three chapters. Chapter 1 presents variables and measurement scales in the context of education. The focus is on the nature of measurement in education, types of variables, types of scales and their transformations, permissible arithmetic operations with scale values, summation symbols, and basic rules of summation. Chapter 2 introduces the classical model of reliability of scores, types of reliability, and reliability of composite scores. Chapter 3 deals with the concept of validity for measurement instruments (e.g., tests, questionnaires, or inventories) and types of validity (content-related validity, criterion-related validity, and construct-related validity). Part II [Research Design] consists of two chapters. Chapter 4 deals with research problems, hypotheses, and types of quantitative research: nonexperimental research, experimental research, and threats to internal and external validity. Chapter 5 presents pre-experimental and true experimental research designs that involve quantitative methods of data analysis. The focus is primarily on conceptual understanding and methodological principles underlying the application of such designs in educational research. xv

4 xvi Preface Part III [Univariate Statistics in Educational Research] consists of fourteen chapters. The first five of these chapters (6, 7, 8, 9, and 10) cover introductory statistics and prepare the ground for understanding and practical applications of intermediate statistics in educational research. The next six chapters (11 through 16) provide intermediate treatment of correlation, regression, and analysis of variance (ANOVA) including some nonparametric methods. The last three chapters in this section (17, 18, and 19) provide more advanced treatment of multiple regression, analysis of variance, and the relations between them. Part IV [Multivariate Statistics in Educational Research] consists of five chapters. This part covers the topics of logistic regression, multivariate analysis of variance (MANOVA), exploratory factor analysis, confirmatory factor analysis, and elements of structural equation modeling (SEM). The analytic framework of these topics is simplified and tailored to conceptual understanding, computer-aided applications, and interpretations in the context of educational research. Supplements Data sets for computer-based applications in examples using SPSS can be downloaded from the online supplement to this book [ This supplement provides also (a) answers to the study questions for each chapter, (b) addendum to some topics discussed in the book, (c) syntax for confirmatory factor analysis, path analysis, and group comparison on latent variables in the framework of major computer programs LISREL, AMOS, EQS, and Mplus [used for illustrations in Chapters 23 and 24], and (d) additional references (books, articles, and online products) related to the content of this book. Acknowledgments I would like to acknowledge the assistance of my graduate research assistant, Jill Lammert, who provided valuable editorial comments and suggestions throughout the writing of this book. I also appreciate the feedback and encouragement of colleagues from different universities as well as graduate students who used draft chapters of this book in quantitative research courses that they took with me in the Graduate School of Education at George Mason University. Dimiter M. Dimitrov

5 Contents Preface xv PART I MEASUREMENT IN EDUCATION 1 Chapter 1 VARIABLES AND MEASUREMENT SCALES Variables in Educational Research Observable versus Latent Variables Continuous versus Discrete Variables Scales of Measurement What is Measurement? Nominal Scale Ordinal Scale Interval Scale Ratio Scale Scale Transformations and Operations Scaling of Individual Items Symbols and Rules for Summation of Variables Symbolic Notations Summation Operator Summary Study Questions 13 Chapter 2 RELIABILITY What is Reliability? Classical Concept of Reliability True Score Definition of Reliability Standard Error of Measurement Types of Reliability Internal Consistency Reliability Test-retest Reliability Alternate Forms Reliability Criterion-referenced Reliability Interrater Reliability Reliability of Composite Scores Reliability of Sum of Scores Reliability of Difference of Scores 24 v

6 vi Contents 2.5 Reliability Estimation with SPSS Calculation of Cronbach s alpha Calculation of Cohen s kappa Summary Study Questions 28 Chapter 3 VALIDITY What is Validity? Aspects of Construct Validity Content Aspect of Validity Substantive Aspect of Validity Structural Aspect of Validity Generalizability Aspect of Validity External Aspect of Validity Consequential Aspect of Validity Summary Study Questions 35 PART II RESEARCH DESIGN 37 Chapter 4 QUANTITATIVE RESEARCH Research Questions and Hypotheses Types of Quantitative Research Nonexperimental Research Descriptive research Correlational research Ex post facto research Meta-analysis research Experimental Research True experimental research Quasi-experimental research Single-case research Internal and External Validity in Experimental Research Threats to internal validity Threats to external validity Summary Study Questions 51

7 Contents vii Chapter 5 BASIC RESEARCH DESIGNS Pre-experimental Designs One Group Posttest-Only Design One Group Pretest-Posttest Design Nonrandomized Control Group Posttest-Only Design True Experimental Designs Randomized Pretest-Posttest Control Group Design Randomized Solomon Four-Group Design Randomized Control-Group Posttest Only Design Quasi-Experimental Designs Nonrandomized Pretest-Posttest Control Group Design One Group Time-Series Design Control Group Time-Series Design Summary Study Questions 63 PART III UNIVARIATE DATA ANALYSIS 65 Chapter 6 REVIEW OF INTRODUCTORY STATISTICS Organizing and Graphing Data Frequency Table Basic Distribution Graphs Describing Distributions Percentiles Measures of Central Tendency Mode Median Mean Properties of the mode, median, and mean Measures of Variation Variance Standard deviation Pooled variance Some basic rules Standard Scores Scale Transformation Summary Study Questions 82

8 viii Contents Chapter 7 BASIC DISTRIBUTIONS Normal Distribution What is a Normal Distribution? Basic Properties of the Normal Distribution Determining Percentiles and Percentile Ranks Sampling Distribution of the Mean Normal Q-Q Plot Student s t-distribution F-Distribution Chi-square Distribution Summary Study Questions 95 Chapter 8 HYPOTHESIS TESTING What is Hypothesis Testing? When To Reject (or Not) the Null Hypothesis? Testing Hypotheses about the Mean One-sample Case for the Mean Two-sample Case for the Mean: Independent Samples Two-sample Case for the Mean: Dependent Samples Summary Study Questions 114 Chapter 9 HYPOTHESIS TESTING FOR PROPORTIONS One-Sample Case for Proportion Testing H 0 : P 1 = P 2 for Independent Samples Testing H 0 : P 1 = P 2 for Dependent Samples Summary Study Questions 132 Chapter 10 CORRELATION AND SIMPLE REGRESSION Correlation between Two Variables What is Linear Relationship (Correlation) between Two Variables? The Pearson Product-Moment Correlation Coefficient Simple Linear Regression Correlation, Prediction, and Causation The Regression Line 144

9 Contents ix Interpretation of the Slope Conditional Distributions of Y-scores Assumptions with Simple Linear Regression Summary Study Questions 155 Chapter 11 PARTIAL AND PART CORRELATION Partial Correlation Part Correlation Summary Study Questions 166 Chapter 12 NONPARAMETRIC TESTS The Man-Whitney U Test The Wilcoxon Signed-Rank Test for Dependent Samples Chi-Square Goodness-of-fit Test Chi-Square Test for Association Summary Study Questions 183 Chapter 13 MULTIPLE REGRESSION The Concept of Multiple Regression Comparison of Full and Restricted Regression Models Multicollinearity Cross-validation Statistical Power, Effect Size, and Sample Size Outliers and Influential Data Points Categorical Predictors in Multiple Regression Interaction between Predictors in Multiple Regression What is Interaction between Predictors? Testing for Interaction between Predictors Centering Predictors Selection of Predictors in Multiple Regression APA Style for Multiple Regression Results Summary Study Questions 214

10 x Contents Chapter 14 ONE-FACTOR ANALYSIS OF VARIANCE The Concept of One-Factor Analysis of Variance Assumptions in ANOVA Effects in One-factor ANOVA Within-groups and Between-groups Variance Linear Model for One-factor ANOVA Testing the ANOVA Null Hypothesis Multiple Comparisons Post Hoc Comparisons The Tukey method of multiple comparisons Bonferroni method of multiple comparisons Planned Comparisons Contrasts for planned multiple comparisons Dunnett method of multiple comparisons Determining Effect Size Effect Size of Mean Differences Omnibus Effect Size Determining the Sample Size Consequences of Violating the ANOVA Assumptions Interpretation of SPSS Output for One-factor ANOVA Summary Study Questions 239 Chapter 15 TWO- AND THREE-FACTOR ANOVA Two-factor ANOVA Null Hypotheses in Two-factor ANOVA Assumptions in Two-factor ANOVA Effects in Two-factor ANOVA Linear Model for the Data in Two-factor ANOVA Sum of Squares in Two-factor ANOVA Mean Squares in Two-factor ANOVA Testing the Null Hypotheses in Two-factor ANOVA Omnibus Effect Size in Two-factor ANOVA Types of Interaction in Two-factor ANOVA Testing for Simple Main Effects Using SPSS for Two-factor ANOVA Three-factor ANOVA Summary Study Questions 265

11 Contents xi Chapter 16 ANALYSIS OF COVARIANCE The Logic Behind ANCOVA Basic Concepts in ANCOVA Adjusted Group Means in ANCOVA Increased Test Power with ANCOVA Assumptions in ANCOVA Performing ANCOVA and Interpreting the Results ANCOVA versus ANOVA on Gain Score Summary Study Questions 280 Chapter 17 MULTIPLE REGRESSION AND ANOVA One-Factor ANOVA via Multiple Regression Contrast Coding for ANOVA with Two Groups Contrast Coding for One-factor ANOVA with Three Groups Orthogonal Contrasts Two-Factor ANOVA via Multiple Regression Summary Study Questions 293 Chapter 18 ANOVA WITH RANDOM FACTORS ANOVA with One Random Factor Random Effects Assumptions of the Random-factor ANOVA Expected Mean Square in the Random-factor ANOVA The Primary Question in a Random-factor ANOVA Two-factor Mixed-Effects ANOVA Model The Concept of a Mixed-effects Model Assumptions of the Two-factor Mixed ANOVA Model Expected Mean Square in the Two-factor Mixed ANOVA Effect Size of Mean Differences among Levels of the Fixed Factor Generalizations with the Two-factor Mixed ANOVA Summary Study Questions 307 Chapter 19 REPEATED-MEASURES ANOVA A Simple Repeated-Measures ANOVA 309

12 xii Contents Univariate Repeated-measures Analysis Assumptions in Repeated-measures ANOVA The Multivariate Test for Repeated-measures ANOVA Univariate or Multivariate Approach to Repeated-measures ANOVA? SPSS for the Simple Repeated-measures ANOVA Repeated-Measures ANOVA with One Between-Subjects Factor Caution with Repeated-Measures ANOVA for Pretest-Posttest Data Summary Study Questions 326 PART IV MULTIVARIATE DATA ANALYSIS 329 Chapter 20 LOGISTIC REGRESSION The Concept of Logistic Regression Probability, Odds, and Odds Ratio The Logistic Model Logit Form of the Logistic Regression Model Interpretation of the Regression Coefficients Tests and Interpretations of Logistic Regression Results Goodness-of-fit Tests Hosmer-Lemeshow Goodness-of-fit Test Test for Significance of Predictor Variables Effect Size Information with Logistic Regression Classification Table Coding Categorical Predictors Using SPSS for Binary Logistic Regression Comparison of Full and Restricted Models Selection of Predictors in Logistic Regression Assumptions in Logistic Regression Summary Study Questions 351 Chapter 21 MULTIVARIATE ANALYSIS OF VARIANCE The Concept of MANOVA MANOVA versus Separate ANOVAs When to Use Separate ANOVAs? When to Use MANOVA? Assumptions in MANOVA MANOVA with Discriminant Analysis MANOVA with Planned Comparisons Sample Size in MANOVA 366

13 Contents xiii 21.9 Summary Study Questions 368 Chapter 22 EXPLORATORY FACTOR ANALYSIS Correlated Variables and Underlying Factors Basic Concepts in Exploratory Factor Analysis Communalities and Eigenvalues The Principle Factor Method of Extracting Factors Rotation of Factors Determining the Number of Factors Eigenvalues of one or higher Criterion Scree Test Parallel Analysis Using SPSS for Exploratory Factor Analysis Summary Study Questions 384 Chapter 23 CONFIRMATORY FACTOR ANALYSIS Differences between EFA and CFA Models Basic Steps for CFA Specification of the CFA Model Evaluation of the CFA Model Adequacy Summary Study Questions 398 Chapter 24 ELEMENTS OF STRUCTURAL EQUATION MODELING Path Analysis Path Coefficients Exogenous and Endogenous Variables Assumptions Decomposition of Correlation Coefficients Testing the Causal Model for Data Fit Just-identified models Overidentified models Elements of Structural Equation Modeling Upgrading Models for Path Analysis to Typical SEM Models Comparing Groups on Latent Variables (Constructs) SEM versus MANOVA Factorial invariance across groups 414

14 xiv Contents Partial measurement invariance Structured means modeling Group-code (MIMIC) modeling Summary Study Questions 427 REFERENCES 429 Statistical Tables 434 Table A-1 Standard normal distribution: z-scores and upper-tail probabilities 434 Table A-2 Critical values of the Student's t-distribution 435 Table A-3 Critical values of the chi-square distribution 436 Table A-4 Critical values of the F-distribution 437 Table A-5 Critical U-values of the Mann-Whitney distribution 442 Table A-6 Critical T-values of the Wilcoxon matched-pairs signed-ranks test 443 Author Index 444 Subject Index 446