Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences

Transcription

1 Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences Third Edition Jacob Cohen (deceased) New York University Patricia Cohen New York State Psychiatric Institute and Columbia University College of Physicians and Surgeons Stephen G. West Arizona State University Leona S. Aiken Arizona State University leq LAWRENCE ERLBAUM ASSOCIATES, PUBLISHERS 2003 Mahwah, New Jersey London

2 Contents Preface Chapter 1: Introduction 1.1 Multiple Regression/Correlation as a General Data-Analytic System Overview Testing Hypotheses Using Multiple Regression/Correlation: Some Examples Multiple Regression/Correlation in Prediction Models A Comparison of Multiple Regression/Correlation and Analysis of Variance Approaches Historical Background Hypothesis Testing and Effect Sizes Multiple Regression/Correlation and the Complexity of Behavioral Science Multiplicity of Influences Correlation Among Research Factors and Partialing Form of Information Shape of Relationship General and Conditional Relationships Orientation of the Book Nonmathematical Applied Data-Analytic Inference Orientation and Specification Error Computation, the Computer, and Numerical Results Computation 14

3 viii CONTENTS Numerical Results: Reporting and Rounding Signiflcance Tests, Confidence Intervals, and Appendix Tables The Spectrum of Behavioral Science Plan for the Book Content Structure: Numbering of Sections, Tables, andequations Summary 18 Chapter 2: Bivariate Correlation and Regression 2.1 Tabular and Graphic Representations of Relationships The Index of Linear Correlation Between Two Variables: The Pearson Product Moment Correlation Coefficient Standard Scores: Making Units Comparable The Product Moment Correlation as a Function of Differences Between z Scores Alternative Formulas for the Product Moment Correlation Coefficient r as the Average Product of z Scores Raw Score Formulas for r Point Biserial r Phi (<)>) Coefficient Rank Correlation Regression Coefficients: Estimating yfromx Regression Toward the Mean The Standard Error of Estimate and Measures of the Strength of Association Summary of Definitions and Interpretations Statistical Inference With Regression and Correlation Coefficients Assumptions Underlying Statistical Inference With Byx, B 0, F and r XY Estimation With Confidence Intervals Null Hypothesis Signiflcance Tests (NHSTs) Confidence Limits and Null Hypothesis Signiflcance Testing Precision and Power Precision of Estimation Power of Null Hypothesis Signiflcance Tests Factors Affecting the Size of r The Distributions of X and Y The Reliability of the Variables Restriction of Range Part-Whole Correlations Ratio or Index Variables Curvilinear Relationships Summary 62

4 Chapter 3: Multiple Regression/Correlation With Two or More Independent Variables 3.1 Introduction: Regression and Causal Models What Is a Cause? Diagrammatic Representation of Causal Models Regression With Two Independent Variables Measures of Association With Two Independent Variables Multiple R and R Semipartial Correlation Coefficients and Increments to R Partial Correlation Coefficients Patterns of Association Between Y and Two Independent Variables Direct and Indirect Effects Partial Redundancy Suppression in Regression Models Spurious Effects and Entirely Indirect Effects Multiple Regression/Correlation With k Independent Variables Introduction: Components of the Prediction Equation Partial Regression Coefficients R, R 2, and Shrunken R sr and sr pr andpr Example of Interpretation of Partial Coefficients Statistical Inference With k Independent Variables Standard Errors and Confidence Intervals for B andß Confidence Intervals for R Confidence Intervals for Differences Between Independent R 2 s Statistical Tests on Multiple and Partial Coefficients Statistical Precision and Power Analysis Introduction: Research Goals and the Null Hypothesis The Precision and Power ofr Precision and Power Analysis for Partial Coefficients Using Multiple Regression Equations in Prediction Prediction of Y for a New Observation Correlation of Individual Variables With Predicted Values Cross-Validation and Unit Weighting Multicollinearity Summary 99

5 X CONTENTS Chapter 4: Data Visualization, Exploration, and Assumption Checking: Diagnosing and Solving Regression Problems I 4.1 Introduction Some Useful Graphical Displays of the Original Data Univariate Displays Bivariate Displays Correlation and Scatterplot Matrices Assumptions and Ordinary Least Squares Regression Assumptions Underlying Multiple Linear Regression Ordinary Least Squares Estimation Detecting Violations of Assumptions Form of the Relationship Omitted Independent Variables Measurement Error Homoscedasticity of Residuais Nonindependence of Residuais Normality of Residuais Remedies: Alternative Approaches When Problems Are Detected Form of the Relationship Inclusion of All Relevant Independent Variables Measurement Error in the Independent Variables Nonconstant Variance Nonindependence of Residuais Summary 150 Chapter 5: Data-Analytic Strategies Using Multiple Regression/Correlation 5.1 Research Questions Answered by Correlations and Their Squares Net Contribution to Prediction Indicesof Differential Validity Comparisons of Predictive Utility Attribution of a Fraction of the XY Relationship to a Third Variable Which of Two Variables Accounts for More of the XY Relationship? Are the Various Squared Correlations in One Population Different From Those in Another Given the Same Variables? Research Questions Answered by B Or ß Regression Coefficients as Reflections of Causal Effects Alternative Approaches to Making ß ra Substantively Meaningful Are the Effects of a Set of Independent Variables on Two Different Outcomes in a Sample Different? 157

6 5.2.4 What Are the Reciprocal Effects of Two Variables on One Another? Hierarchical Analysis Variables in Multiple Regression/ Correlation Causal Priority and the Removal of Confounding Variables Research Relevance Examination of Alternative Hierarchical Sequences of Independent Variable Sets Stepwise Regression The Analysis of Sets of Independent Variables TypesofSets The Simultaneous and Hierarchical Analyses of Sets Variance Proportions for Sets and the Ballantine Again B and ß Coefficients for Variables Within Sets Significance Testing for Sets Application in Hierarchical Analysis Application in Simultaneous Analysis Using Computer Output to Determine Statistical Significance An Alternative F Test: Using Model 2 Error Estimate From the Final Model Power Analysis for Sets Determining n* for the F Test of sr\ with Model 1 or Model 2 Error Estimating the Population sr 2 Values Setting Power for n* Reconciling Differentn*s Power as a Function of n Tactics of Power Analysis Statistical Inference Strategy in Multiple Regression/ Correlation Controlling and Balancing Type I and Type II Errors in Inference Less Is More LeastlsLast Adaptation of Fisher's Protected t Test Statistical Inference and the Stage of Scientific Investigations Summary 190 CONTENTS Chapter 6: Quantitative Scales, Curvilinear Reiationships, and Transformations Introduction What Do We Mean by Linear Regression? 193

7 CONTENTS Linearity in the Variables and Linear Multiple Regression Four Approaches to Examining Nonlinear Relationships in Multiple Regression 195 Power Polynomials Method An Example: Quadratic Fit Centering Predictors in Polynomial Equations Relationship of Test of Significance of Highest Order Coefficient and Gain in Prediction Interpreting Polynomial Regression Results Another Example: A Cubic Fit Strategy and Limitations More Complex Equations 213 Orthogonal Polynomials The Cubic Example Revisited Unequal n and Unequal Intervals Applications and Discussion 220 Nonlinear Transformations Purposes of Transformation and the Nature of Transformations The Conceptual Basis of Transformations and Model Checking Before and After Transformation Is It Always Ideal to Transform? Logarithms and Exponents; Additive and Proportional Relationships Linearizing Relationships Linearizing Relationships Based on Strong Theoretical Models Linearizing Relationships Based on Weak Theoretical Models Empirically Driven Transformations in the Absence of Strong or Weak Models Empirically Driven Transformation for Linearization: The Ladder of Re-expression and the Bulging Rule Empirically Driven Transformation for Linearization in the Absence of Models: Box-Cox Family of Power Transformations on Y Empirically Driven Transformation for Linearization in the Absence of Models: Box-Tidwell Family of Power Transformations on X Linearization of Relationships With Correlations: Fisher z' Transform of r Transformations That Linearize Relationships for Counts and Proportions Variance Stabilizing Transformations and Alternatives for Treatment of Heteroscedasticity Transformations to Normalize Variables Diagnostics Following Transformation 247

8 CONTENTS XIII Measuring and Comparing Model Fit Second-Order Polynomial Numerical Example Revisited When to Transform and the Choice of Transformation Nonlinear Regression Nonparametric Regression Summary 253 Chapter 7: Interactions Among Continuous Variables Introduction Interactions Versus Additive Effects Conditional First-Order Effects in Equations Containing Interactions Centering Predictors and the Interpretation of Regression Coefficients in Equations Containing Interactions Regression with Centered Predictors Relationship Between Regression Coefficients in the Uncentered and Centered Equations Centered Equations With No Interaction Essential Versus Nonessential Multicollinearity Centered Equations With Interactions The Highest Order Interaction in the Centered Versus Uncentered Equation Do Not Center Y A Recommendation for Centering Simple Regression Equations and Simple Slopes Plotting Interactions Moderator Variables Simple Regression Equations Overall Regression Coefficient and Simple Slope at the Mean Simple Slopes From Uncentered Versus Centered Equations Are Identical Linear by Linear Interactions Interpreting Interactions in Multiple Regression and Analysis of Variance Post Hoc Probing of Interactions Standard Error of Simple Slopes Equation Dependence of Simple Slopes and Their Standard Errors Tests of Significance of Simple Slopes Confidence Intervals Around Simple Slopes A Numerical Example The Uncentered Regression Equation Revisited First-Order Coefficients in Equations Without and With Interactions Interpretation and the Range of Data 282

9 XIV CONTENTS 7.5 Standardized Estimates for Equations Containing Interactions Interactions as Partialed Effects: Building Regression Equations With Interactions Patteras of First-Order and Interactive Effects Three Theoretically Meaningful Patterns of First-Order and Interaction Effects Ordinal Versus Disordinal Interactions Three-Predictor Interactions in Multiple Regression Curvilinear by Linear Interactions Interactions Among Sets of Variables Issues in the Detection of Interactions: Reliability, Predictor Distributions, Model Specification Variable Reliability and Power to Detect Interactions Sampling Designs to Enhance Power to Detect Interactions Optimal Design Difficulty in Distinguishing Interactions Versus Curvilinear Effects Summary 300 Chapter 8: Categorical or Nominal Independent Variables Introduction Categories as a Set of Independent Variables The Representation of Categories or Nominal Scales Dummy-Variable Coding Coding the Groups Pearson Correlations of Dummy Variables With Y Correlations Among Dummy-Coded Variables Multiple Correlation of the Dummy-Variable Set Wim Regression Coefficients for Dummy Variables Partial and Semipartial Correlations for Dummy Variables Dummy-Variable Multiple Regression/Correlation and One-Way Analysis of Variance A Cautionary Note: Dummy-Variable-Like Coding Systems Dummy-Variable Coding When Groups Are Not Mutually Exclusive Unweighted Effects Coding Introduction: Unweighted and Weighted Effects Coding Constructing Unweighted Effects Codes The R 2 and the r K s for Unweighted Effects Codes Regression Coefficients and Other Partial Effects in Unweighted Code Sets 325

10 CONTENTS XV 8.4 Weighted Effects Coding Selection Considerations for Weighted Effects Coding Constructing Weighted Effects The R 2 and R 2 for Weighted Effects Codes Interpretation and Testing of B With Unweighted Codes Contrast Coding Considerations in the Selection of a Contrast Coding Scheine Constructing Contrast Codes The R 2 and R Partial Regression Coefficients Statistical Power and the Choice of Contrast Codes Nonsense Coding Coding Schemes in the Context of Other Independent Variables Combining Nominal and Continuous Independent Variables Calculating Adjusted Means for Nominal Independent Variables Adjusted Means for Combinations of Nominal and Quantitative Independent Variables Adjusted Means for More Than Two Groups and Alternative Coding Methods Multiple Regression/Correlation With Nominal Independent Variables and the Analysis of Covariance Summary 351 Chapter 9: Interactions With Categorical Variables Nominal Scale by Nominal Scale Interactions The 2 by 2 Design Regression Analyses of Multiple Sets of Nominal Variables With More Than Two Categories Interactions Involving More Than Two Nominal Scales An Example of Three Nominal Scales Coded by Alternative Methods Interactions Among Nominal Scales in Which Not All Combinations Are Considered What If the Categories for One or More Nominal "Scales" Are Not Mutually Exclusive? Consideration of pr, ß, and Variance Proportions for Nominal Scale Interaction Variables Summary of Issues and Recommendations for Interactions Among Nominal Scales Nominal Scale by Continuous Variable Interactions A Reminder on Centering 375

11 XVi CONTENTS Interactions of a Continuous Variable With Dummy-Variable Coded Groups Interactions Using Weighted or Unweighted Effects Codes Interactions With a Contrast-Coded Nominal Scale Interactions Coded to Estimate Simple Slopes of Groups Categorical Variable Interactions With Nonlinear Effects of Scaled Independent Variables Interactions of a Scale With Two or More Categorical Variables Summary 388 Chapter 10: Outliers and Multicoilinearity: Diagnosing and Solving Regression Problems II Introduction Outliers: Introduction and Illustration Detecting Outliers: Regression Diagnostics Extremity on the Independent Variables: Leverage Extremity on Y: Discrepancy Influence on the Regression Estimates Location of Outlying Points and Diagnostic Statistics Summary and Suggestions Sources of Outliers and Possible Remedial Actions Sources of Outliers Remedial Actions Multicoilinearity Exact Cpllinearity Multicoilinearity: A Numerical Illustration Measures of the Degree of Multicoilinearity Remedies for Multicoilinearity Model Respecification Collection of Additional Data Ridge Regression Principal Components Regression Summary of Multicoilinearity Considerations Summary 430 Chapter 11: Missing Data Basic Issues in Handling Missing Data Minimize Missing Data Types of Missing Data Traditional Approaches to Missing Data 433

12 CONTENTS xvii 11.2 Missing Data in Nominal Scales Coding Nominal Scale X for Missing Data Missing Data on Two Dichotomies Estimation Using the EM Algorithm Missing Data in Quantitative Scales Available Alternatives Imputation of Values for Missing Cases Modeling Solutions to Missing Data in Scaled Variables An Illustrative Comparison of Alternative Methods RulesofThumb Summary 450 Chapter 12: Multiple Regression/Correlation and Causal Models Introduction Limits on the Current Discussion and the Relationship Between Causal Analysis and Analysis of Covariance Theories and Multiple Regression/Correlation Models That Estimate and Test Them Kinds of Variables in Causal Models Regression Models as Causal Models Models Without Reciprocal Causation Direct and Indirect Effects Path Analysis and Path Coefficients Hierarchical Analysis and Reduced Form Equations Partial Causal Models and the Hierarchical Analysis of Sets Testing Model Elements Models With Reciprocal Causation Identification and Overidentification Just Identified Models Overidentification Underidentification Latent Variable Models An Example of a Latent Variable Model How Latent Variables Are Estimated Fixed and Free Estimates in Latent Variable Models Goodness-of-Fit Tests of Latent Variable Models Latent Variable Models and the Correction for Attenuation Characteristics of Data Sets That Make Latent Variable Analysis the Method of Choice A Review of Causal Model and Statistical Assumptions 475

13 XVÜi CONTENTS Specification Error Identification Error Comparisons of Causal Models Nested Models Longitudinal Data in Causal Models Summary 477 Chapter 13: Alternative Regression Models: Logistic, Poisson Regression, and the Generalized Linear Model Ordinary Least Squares Regression Revisited Three Characteristics of Ordinary Least Squares Regression The Generalized Linear Model Relationship of Dichotomous and Count Dependent Variables Y to a Predictor Dichotomous Outcomes and Logistic Regression Extending Linear Regression: The Linear Probability Model and Discriminant Analysis The Nonlinear Transformation From Predictor to Predicted Scores: Probit and Logistic Transformation The Logistic Regression Equation Numerical Example: Three Forms of the Logistic Regression Equation Understanding the Coefficients for the Predictor in Logistic Regression Multiple Logistic Regression Numerical Example Confidence Intervals on Regression Coefficients and Odds Ratios Estimation of the Regression Model: Maximum Likelihood Deviances: Indices of Overall Fit of the Logistic Regression Model Multiple R 2 Analogs in Logistic Regression Testing Significance of Overall Model Fit: The Likelihood Ratio Test and the Test of Model Deviance x 2 Test for the Significance of a Single Predictor in a Multiple Logistic Regression Equation Hierarchical Logistic Regression: Likelihood Ratio x 2 Test for the Significance of a Set of Predictors Above and Beyond Another Set Akaike's Information Criterion and the Bayesian Information Criterion for Model Comparison Some Treachery in Variable Scaling and Interpretation of the Odds Ratio 509

14 CONTENTS xix Regression Diagnostics in Logistic Regression Sparseness of Data Classification of Cases Extensions of Logistic Regression to Multiple Response Categories: Polytomous Logistic Regression and Ordinal Logistic Regression Polytomous Logistic Regression Nested Dichotomies Ordinal Logistic Regression Models for Count Data: Poisson Regression and Alternatives Linear Regression Applied to Count Data Poisson Probability Distribution Poisson Regression Analysis Overdispersion and Alternative Models Independence of Observations Sources on Poisson Regression Füll Circle: Parallels Between Logistic and Poisson Regression, and the Generalized Linear Model Parallels Between Poisson and Logistic Regression The Generalized Linear Model Revisited Summary 535 Chapter 14: Random Coefficient Regression and Multilevel Models Clustering Within Data Sets Clustering, Alpha Inflation, and the Intraclass Correlation Estimating the Intraclass Correlation Analysis of Clustered Data With Ordinary Least Squares Approaches Numerical Example, Analysis of Clustered Data With Ordinary Least Squares Regression The Random Coefficient Regression Model Random Coefficient Regression Model and Multilevel Data Structure Ordinary Least Squares (Fixed Effects) Regression Revisited Fixed and Random Variables Clustering and Hierarchically Structured Data Structure of the Random Coefficient Regression Model Level 1 Equations Level 2 Equations Mixed Model Equation for Random Coefficient Regression Variance Components New Parameters in the Multilevel Model Variance Components and Random Coefficient Versus Ordinary Least Squares (Fixed Effects) Regression 549

15 XX CONTENTS Parameters of the Random Coefficient Regression Model: Fixed and Random Effects Numerical Example: Analysis of Clustered Data With Random Coefficient Regression Unconditional Cell Means Model and the Intraclass Correlation Testing the Fixed and Random Parts of the Random Coefficient Regression Model Clustering as a Meaningful Aspect of the Data Multilevel Modeling With a Predictor at Level Level 1 Equations Revised Level 2 Equations Mixed Model Equation With Level 1 Predictor and Level 2 Predictor of Intercept and Slope and the Cross-Level Interaction An Experimental Design as a Multilevel Data Structure: Combining Experimental Manipulation With Individual Differences Numerical Example: Multilevel Analysis Estimation of the Multilevel Model Parameters: Fixed Effects, Variance Components, and Level 1 Equations Fixed Effects and Variance Components An Equation for Each G/oup: Empirical Bayes Estimates of Level 1 Coefficients Statistical Tests in Multilevel Models Fixed Effects Variance Components Some Model Specification Issues The Same Variable at Two Levels Centering in Multilevel Models Statistical Power of Multilevel Models Choosing Between the Fixed Effects Model and the Random Coefficient Model Sources on Multilevel Modeling Multilevel Models Applied to Repeated Measures Data Summary 567 Chapter 15: Longitudinal Regression Methods 15.1 Introduction Chapter Goals Purposes of Gathering Data on Multiple Occasions Analyses of Two-Time-Point Data Change or Regressed Change? Alternative Regression Models for Effects Over a Single Unit of Time Three- or Four-Time-Point Data Repeated Measure Analysis of Variance 573

16 Multiple Error Terms in Repeated Measure Analysis of Variance Trend Analysis in Analysis of Variance Repeated Measure Analysis of Variance in Which Time Is Not the Issue Multilevel Regression of Individual Changes Over Time Patterns of Individual Change Over Time Adding Other Fixed Predictors to the Model Individual Differences in Variation Around Individual Slopes Alternative Developmental Models and Error Structures Alternative Link Functions for Predicting Y From Time Unbalanced Data: Variable Timing and Missing Data Latent Growth Models: Structural Equation Model Representation of Multilevel Data Estimation of Changes in True Scores Representation of Latent Growth Models in Structural Equation Model Diagrams Comparison of Multilevel Regression and Structural Equation Model Analysis of Change Time Varying Independent Variables Survival Analysis Regression Analysis of Time Until Outcome and the Problem of Censoring Extension to Time-Varying Independent Variables Extension to Multiple Episode Data Extension to a Categorical Outcome: Event-History Analysis Time Series Analysis 600 / Units of Observation in Time Series Analyses Time Series Analyses Applications Time Effects in Time Series Extension of Time Series Analyses to Multiple Units or Subjects Dynamic System Analysis Statistical Inference and Power Analysis in Longitudinal Analyses Summary 605 CONTENTS Chapter 16: Multiple Dependent Variables: Set Correlation Introduction to Ordinary Least Squares Treatment of Multiple Dependent Variables Set Correlation Analysis 608

17 XXII CONTENTS Canonical Analysis Elements of Set Correlation Measures of Multivariate Association R Y,x> me Proportion of Generalized Variance Ty x and P Y,x> Proportions of Additive Variance Partialing in Set Correlation Frequent Reasons for Partialing Variable Sets From the Basic Sets The Five Types of Association Between Basic Y and X Sets Tests of Statistical Significance and Statistical Power Testing the Null Hypothesis Estimators of the Population R YJC, T\j, and P 2 YJi Guarding Against Type I Error Inflation Statistical Power Analysis in Set Correlation Comparison of Set Correlation With Multiple Analysis of Variance New Analytic Possibilities With Set Correlation Illustrative Examples A Simple Whole Association A Multivariate Analysis of Partial Variance A Hierarchical Analysis of a Quantitative Set and Its Unique Components Bipartial Association Among Three Sets Summary 627 APPENDICES Appendix 1: The Mathematical Basis for Multiple Regression/Correlation and Identification of the Inverse Matrix Elements 631 ALI Alternative Matrix Methods 634 AI.2 Determinants 634 Appendix 2: Determination of the Inverse Matrix and Applications Thereof 636 A2.1 Hand Calculation of the Multiple Regression/Correlation Problem 636 A2.2 Testing the Difference Between Partial ßs and Bs From the Same Sample 640 A2.3 Testing the Difference Between ßs for Different Dependent Variables From a Single Sample 642

18 CONTENTS XXÜi Appendix Tables 643 Table A t Values for a =.01,.05 (Two Tailed) 643 Table B z! Transformation of r 644 Table C Normal Distribution 645 Table D F Values for a =.01, Table E L Values for a =.01, Table F Power of Significance Test of r at a =.01,.05 (Two Tailed) 652 Table G n* to Detect r by t Test at a =.01,.05 (Two Tailed) 654 References Glossary Statistical Symbols and Abbreviations Author Index Subject Index i