Neil H. Timm Applied Multivariate Analysis With 42 Figures Springer
Contents Preface Acknowledgments List of Tables List of Figures vii ix xix xxiii 1 Introduction 1 1.1 Overview 1 1.2 Multivariate Models and Methods 1 1.3 Scope of the Book 3 2 Vectors and Matrices 7 2.1 Introduction 7 2.2 Vectors, Vector Spaces, and Vector Subspaces 7 a. Vectors 7 b. Vector Spaces 8 c. Vector Subspaces 9 2.3 Bases, Vector Norms, and the Algebra of Vector Spaces 12 a. Bases 13 b. Lengths, Distances, and Angles 13 c. Gram-Schmidt Orthogonalization Process 15 d. Orthogonal Spaces 17 e. Vector Inequalities, Vector Norms, and Statistical Distance 21
xii Contents 2.4 Basic Matrix Operations 25 a. Equality, Addition, and Multiplication of Matrices 26 b. Matrix Transposition 28 c. Some Special Matrices 29 d. Trace and the Euclidean Matrix Norm 30 e. Kronecker and Hadamard Products 32 f. Direct Sums 35 g. The Vec(-) and Vech(-) Operators 35 2.5 Rank, Inverse, and Determinant 41 a. Rank and Inverse 41 b. Generalized Inverses 47 c. Determinants 50 2.6 Systems of Equations, Transformations, and Quadratic Forms 55 a. Systems of Equations 55 b. Linear Transformations 61 c. Projection Transformations 63 d. Eigenvalues and Eigenvectors 67 e. Matrix Norms 71 f. Quadratic Forms and Extrema 72 g. Generalized Projectors 73 2.7 Limits and Asymptotics 76 3 Multivariate Distributions and the Linear Model 79 3.1 Introduction 79 3.2 Random Vectors and Matrices 79 3.3 The Multivariate Normal (MVN) Distribution 84 a. Properties of the Multivariate Normal Distribution 86 b. Estimating /x and E 88 c. The Matrix Normal Distribution 90 3.4 The Chi-Square and Wishart Distributions 93 a. Chi-Square Distribution 93 b. The Wishart Distribution 96 3.5 Other Multivariate Distributions 99 a. The Univariate t and F Distributions 99 b. Hotelling's T 2 Distribution 99 c. The Beta Distribution 101 d. Multivariate t, F, and / 2 Distributions 104 3.6 The General Linear Model 106 a. Regression, ANOVA, and ANCOVA Models 107 b. Multivariate Regression, MANOVA, and MANCOVA Models... 110 c. The Seemingly Unrelated Regression (SUR) Model 114 d. The General MANOVA Model (GMANOVA) 115 3.7 Evaluating Normality 118 3.8 Tests of Covariance Matrices 133 a. Tests of Covariance Matrices 133
Contents xiii b. Equality of Covariance Matrices 133 c. Testing for a Specific Covariance Matrix 137 d. Testing for Compound Symmetry 138 e. Tests of Sphericity 139 f. Tests of Independence 143 g. Tests for Linear Structure 145 3.9 Tests of Location 149 a. Two-Sample Case, Ei = E 2 = 149 b. Two-Sample Case, Ei ^ E 2 156 c. Two-Sample Case, Nonnormality 160 d. Profile Analysis, One Group 160 e. Profile Analysis, Two Groups 165 f. Profile Analysis, Ei ^ E 2 175 3.10 Univariate Profile Analysis 181 a. Univariate One-Group Profile Analysis 182 b. Univariate Two-Group Profile Analysis 182 3.11 Power Calculations 182 Multivariate Regression Models 185 4.1 Introduction 185 4.2 Multivariate Regression 186 a. Multiple Linear Regression 186 b. Multivariate Regression Estimation and Testing Hypotheses 187 c. Multivariate Influence Measures 193 d. Measures of Association, Variable Selection and Lack-of-Fit Tests.. 197 e. Simultaneous Confidence Sets for a New Observation y new and the Elements of B 204 f. Random X Matrix and Model Validation: Mean Squared Error of Prediction in Multivariate Regression 206 g. Exogeniety in Regression 211 4.3 Multivariate Regression Example 212 4.4 One-Way MANOVA and MANCOVA 218 a. One-Way MANOVA 218 b. One-Way MANCOVA 225 c. Simultaneous Test Procedures (STP) for One-Way MANOVA /MANCOVA 230 4.5 One-Way MANOVA/MANCOVA Examples 234 a. MANOVA (Example 4.5.1) 234 b. MANCOVA (Example 4.5.2) 239 4.6 MANOVA/MANCOVA with Unequal E, or Nonnormal Data 245 4.7 One-Way MANOVA with Unequal E, Example 246 4.8 Two-Way MANOVA/MANCOVA 246 a. Two-Way MANOVA with Interaction 246 b. Additive Two-Way MANOVA 252 c. Two-Way MANCOVA 256
xiv Contents d. Tests of Nonadditivity 256 4.9 Two-Way MANOVA/MANCOVA Example 257 a. Two-Way MANOVA (Example 4.9.1) 257 b. Two-Way MANCOVA (Example 4.9.2) 261 4.10 Nonorthogonal Two-Way MANOVA Designs 264 a. Nonorthogonal Two-Way MANOVA Designs with and Without Empty Cells, and Interaction 265 b. Additive Two-Way MANOVA Designs With Empty Cells 268 4.11 Unbalance, Nonorthogonal Designs Example 270 4.12 Higher Ordered Fixed Effect, Nested and Other Designs 273 4.13 Complex Design Examples 276 a. Nested Design (Example 4.13.1) 276 b. Latin Square Design (Example 4.13.2) 279 4.14 Repeated Measurement Designs 282 a. One-Way Repeated Measures Design 282 b. Extended Linear Hypotheses 286 4.15 Repeated Measurements and Extended Linear Hypotheses Example...294 a. Repeated Measures (Example 4.15.1) 294 b. Extended Linear Hypotheses (Example 4.15.2) 298 4.16 Robustness and Power Analysis for MR Models 301 4.17 Power Calculations Power.sas 304 4.18 Testing for Mean Differences with Unequal Covariance Matrices 307 5 Seemingly Unrelated Regression Models 311 5.1 Introduction 311 5.2 The SUR Model 312 a. Estimation and Hypothesis Testing 312 b. Prediction 314 5.3 Seeming Unrelated Regression Example 316 5.4 The CGMANOVA Model 318 5.5 CGMANOVA Example 319 5.6 The GMANOVA Model 320 a. Overview 320 b. Estimation and Hypothesis Testing 321 c. Test of Fit 324 d. Subsets of Covariates 324 e. GMANOVA vs SUR 326 f. Missing Data 326 5.7 GMANOVA Example 327 a. One Group Design (Example 5.7.1) 328 b. Two Group Design (Example 5.7.2) 330 5.8 Tests of Nonadditivity 333 5.9 Testing for Nonadditivity Example 335 5.10 Lack of Fit Test 337 5.11 Sum of Profile Designs 338
Contents xv 5.12 The Multivariate SUR (MSUR) Model 339 5.13 Sum of Profile Example 341 5.14 Testing Model Specification in SUR Models 344 5.15 Miscellanea 348 Multivariate Random and Mixed Models 351 6.1 Introduction 351 6.2 Random Coefficient Regression Models 352 a. Model Specification 352 b. Estimating the Parameters 353 c. Hypothesis Testing 355 6.3 Univariate General Linear Mixed Models 357 a. Model Specification 357 b. Covariance Structures and Model Fit 359 c. Model Checking 361 d. Balanced Variance Component Experimental Design Models 366 e. Multilevel Hierarchical Models 367 f. Prediction 368 6.4 Mixed Model Examples 369 a. Random Coefficient Regression (Example 6.4.1) 371 b. Generalized Randomized Block Design (Example 6.4.2) 376 c. Repeated Measurements (Example 6.4.3) 380 d. HLM Model (Example 6.4.4) 381 6.5 Mixed Multivariate Models 385 a. Model Specification 386 b. Hypothesis Testing 388 c. Evaluating Expected Mean Square 391 d. Estimating the Mean 392 e. Repeated Measurements Model 392 6.6 Balanced Mixed Multivariate Models Examples 394 a. Two-way Mixed MANOVA 395 b. Multivariate Split-Plot Design 395 6.7 Double Multivariate Model (DMM) 400 6.8 Double Multivariate Model Examples 403 a. Double Multivariate MANOVA (Example 6.8.1) 404 b. Split-Plot Design (Example 6.8.2) 407 6.9 Multivariate Hierarchical Linear Models 415 6.10 Tests of Means with Unequal Covariance Matrices 417 Discriminant and Classification Analysis 419 7.1 Introduction 419 7.2 Two Group Discrimination and Classification 420 a. Fisher's Linear Discriminant Function 421 b. Testing Discriminant Function Coefficients 422 c. Classification Rules 424
xvi Contents d. Evaluating Classification Rules 427 7.3 Two Group Discriminant Analysis Example 429 a. Egyptian Skull Data (Example 7.3.1) 429 b. Brain Size (Example 7.3.2) 432 7.4 Multiple Group Discrimination and Classification 434 a. Fisher's Linear Discriminant Function 434 b. Testing Discriminant Functions for Significance 435 c. Variable Selection 437 d. Classification Rules 438 e. Logistic Discrimination and Other Topics 439 7.5 Multiple Group Discriminant Analysis Example 440 8 Principal Component, Canonical Correlation, and Exploratory Factor Analysis 445 8.1 Introduction 445 8.2 Principal Component Analysis 445 a. Population Model for PCA 446 b. Number of Components and Component Structure 449 c. Principal Components with Covariates 453 d. Sample PCA 455 e. Plotting Components 458 f. Additional Comments 458 g. Outlier Detection 458 8.3 Principal Component Analysis Examples 460 a. Test Battery (Example 8.3.1) 460 b. Semantic Differential Ratings (Example 8.3.2) 461 c. Performance Assessment Program (Example 8.3.3) 465 8.4 Statistical Tests in Principal Component Analysis 468 a. Tests Using the Covariance Matrix 468 b. Tests Using a Correlation Matrix 472 8.5 Regression on Principal Components 474 a. GMANOVA Model 475 b. The PCA Model 475 8.6 Multivariate Regression on Principal Components Example 476 8.7 Canonical Correlation Analysis 477 a. Population Model for CCA 477 b. Sample CCA 482 c. Tests of Significance 483 d. Association and Redundancy 485 e. Partial, Part and Bipartial Canonical Correlation 487 f. Predictive Validity in Multivariate Regression using CCA 490 g. Variable Selection and Generalized Constrained CCA 491 8.8 Canonical Correlation Analysis Examples 492 a. Rohwer CCA (Example 8.8.1) 492 b. Partial and Part CCA (Example 8.8.2) 494
Contents xvii 8.9 Exploratory Factor Analysis 496 a. Population Model for EFA 497 b. Estimating Model Parameters 502 c. Determining Model Fit 506 d. Factor Rotation 507 e. Estimating Factor Scores 509 f. Additional Comments 510 8.10 Exploratory Factor Analysis Examples 511 a. Performance Assessment Program (PAP Example 8.10.1) 511 b. Di Vesta and Walls (Example 8.10.2) 512 c. Shin (Example 8.10.3) 512 9 Cluster Analysis and Multidimensional Scaling 515 9.1 Introduction 515 9.2 Proximity Measures 516 a. Dissimilarity Measures 516 b. Similarity Measures 519 c. Clustering Variables 522 9.3 Cluster Analysis 522 a. Agglomerative Hierarchical Clustering Methods 523 b. Nonhierarchical Clustering Methods 530 c. Number of Clusters 531 d. Additional Comments 533 9.4 Cluster Analysis Examples 533 a. Protein Consumption (Example 9.4.1) 534 b. Nonhierarchical Method (Example 9.4.2) 536 c. Teacher Perception (Example 9.4.3) 538 d. Cedar Project (Example 9.4.4) 541 9.5 Multidimensional Scaling 541 a. Classical Metric Scaling 542 b. Nonmetric Scaling 544 c. Additional Comments 547 9.6 Multidimensional Scaling Examples 548 a. Classical Metric Scaling (Example 9.6.1) 549 b. Teacher Perception (Example 9.6.2) 550 c. Nation (Example 9.6.3) 553 10 Structural Equation Models 557 10.1 Introduction 557 10.2 Path Diagrams, Basic Notation, and the General Approach 558 10.3 Confirmatory Factor Analysis 567 10.4 Confirmatory Factor Analysis Examples 575 a. Performance Assessment 3 - Factor Model (Example 10.4.1) 575 b. Performance Assessment 5-Factor Model (Example 10.4.2) 578 10.5 Path Analysis 580
xviii Contents 10.6 Path Analysis Examples 586 a. Community Structure and Industrial Conflict (Example 10.6.1)....586 b. Nonrecursive Model (Example 10.6.2) 590 10.7 Structural Equations with Manifest and Latent Variables 594 10.8 Structural Equations with Manifest and Latent Variables Example... 595 10.9 Longitudinal Analysis with Latent Variables 600 10.10 Exogeniety in Structural Equation Models 604 Appendix 609 References 625 Author Index 667 Subject Index 675