The president of a Fortune 500 firm wants to measure the firm s image.

Transcription

1 4. Factor Analysis A related method to the PCA is the Factor Analysis (FA) with the crucial difference that in FA a statistical model is constructed to explain the interrelations (correlations) between observed variables. 1

2 Example 4.1: The marketing manager of an apparel firm wants to determine whether or not a relationship exist between patriotism and consumers attitudes about domestic and foreign products. The president of a Fortune 500 firm wants to measure the firm s image. In each of these cases the concepts are evidently so large that they hardly can be measured satisfactorily with a single variable. Instead several indicators must be constructed to get an idea, for example, of the image of a firm. Hence, the manager has a thought (model) that the image is the background determinant of the observed indicators (or variables). 2

3 Variables that cannot be measured (observed) directly are usually called latent variables or factors, and those that are observed directly as manifest variables. 3

4 The Model Observed variables: x 1,..., x p Latent, common factors: ξ 1,..., ξ m, with m << p. such that (1) x i = µ i + λ i1 ξ λ im ξ m + δ i, where λ ij : the loading of variable variable x i on factor ξ j, δ i : is the error term, sometimes called a unique factor of variable x i, µ i : the expected value of x i (in the following we assume µ i = 0). 4

5 Remark 4.1: The factors ξ j are unobserved. Thus model (1) cannot be estimated like the usual regression model. Remark 4.2: Often factors are denoted by f j, but here we use the same notations that are custom in the latent variable structural equation models, dealt with later. A basic assumption is that Cov[ɛ i, ɛ k ] = 0, i.e., the error terms between variables x i and x k are uncorrelated (i k). Thus, all the correlation between the observed variables are explained by the common factors. 5

6 Example 4.2: Two factors (idelized) model for six observed variables. λ 11 ξ λ 21 1 x 1 x 2 δ1 δ2 7 λ 31 x 3 δ3 φ 12 λ 42 λ ξ 52 2 x 4 x 5 δ4 δ5 7 λ 62 x 6 δ6 In this idealized model all correlations are determined through the correlation, φ 12, between the factors and the appropriate factor loadings λ ij. Thus, due to the structuring the covariances these models are usally called covariance structure models 6

7 The solution for the factor problem is obtained by solving the unknown parameters in the implied covariance matrix of the x- variables (the fundamental equation of factor analysis) (2) Σ = ΛΦΛ + Θ δ, where Σ is the covariance matrix of the x- variables, Λ is the matrix of the loadings λ ij, Φ is the correlation matrix of the factors, and Θ δ is a diagonal matrix with error variances, Var[δ i ] on the diagonal. Remark 4.3: In exploratory factor analysis, due to the non-uniqueness of the solution, we can select initially Φ = I, the identity matrix, which implies that the factors are assumed to be uncorrelated. Then (2) reduces to (3) Σ = ΛΛ + Θ δ. 7

8 Example 4.3: Holzinger and Swineford (1939) data. The following psychological test were measured from 145 student in two Chicago school. The three first are aimed to measure spatial or visual perception and the last three verbal ability. Variables: visperc: Visual perception scores cubes: Test of spatial visualization lozenges: Test of spatial orientation paragraph: Paragraph comprehension score sentence: Sentence completion score wordmean: Word meaning test score 8

9 Spatial perception and verbal ability are considered as underlying abilities that are not directly measurable. The tests perform indicators of these abilities. The correlation matrix of the variables: ================================================================ visperc cubes lozenges paragraph sentence wordmean visperc 1 cubes lozenges paragraph sentence wordmean ================================================================ 9

10 Using AMOS, we can first visually structure the model and then estimate. Below are reported standardized estimates and a goodness of fit statistic. 10

11 In Amos an alternative to graphic specification is to specify and fit a model the model by VB.NE or C# classes. Below is C#. #region "Header" using System; using System.Diagnostics; using AmosEngineLib; using AmosGraphics; using PBayes; using tm = AmosEngineLib.AmosEngine.TMatrixID; #endregion 11

12 class MainModule { static void Main() { // Your code goes here. AmosEngine holz = new AmosEngine(); try { holz.textoutput(); holz.smc(); holz.standardized(); holz.begingroup("grant.sav"); holz.astructure("visperc <-- e_v (1)"); holz.astructure("visperc <-- spatial"); holz.astructure("cubes <-- e_c (1)"); holz.astructure("cubes <-- spatial"); holz.astructure("lozenges <-- e_l (1)"); holz.astructure("lozenges <-- spatial"); } } holz.astructure("paragrap <-- e_p (1)"); holz.astructure("paragrap <-- visual"); holz.astructure("sentence <-- e_s (1)"); holz.astructure("sentence <-- visual"); holz.astructure("wordmean <-- e_w (1)"); holz.astructure("wordmean <-- visual"); holz.astructure("spatial <--> visual"); holz.astructure("spatial (1)"); holz.astructure("visual (1)"); holz.fitmodel(); } finally { holz.dispose(); } 12

13 This is an example of confirmatory factor analysis (CFA). In practice, however, there is usually no clear idea about the latent variables. In such a case the task is to try to find whether there exist common latent factors and how many. This is the traditional exploratory factor analysis (EFA) set up. 13

14 Objectives of Exploratory Factor Analysis 1. Determine the smallest number of common factors that best account for the correlations among the indicators,i.e., determine m in model 1). 2. Identify the most plausible factors via rotation. 3. Estimate the pattern coefficients, structure loadings, communalities, and unique variances of the variables. 4. Provide interpretation for the common factors. 5. If necessary, estimate the factor scores. 14

15 Factors are usually extracted from a correlation matrix (standardized solution). Qualitatively it should not make difference whether the factors are extracted from a correlation or a covariance matrix, i.e., the interpretation of the end result should be the same. 15

16 The following data is used to illustrate the above steps. data. Example 4.4: The product manager of a consumer package goods firm is interested in identifying the major underlying dimensions that consumers use to evaluate detergents in the market place. A sample of 143 respondents rated three brands of detergents on 12 product attributes using a five-point semantic differential scale. The attributes were: V 1 : V 2 : V 3 : V 4 : V 5 : V 6 : V 7 : V 8 : V 9 : V 10 : V 11 : V 12 : Gentle to natural fabrics Won t harm colors Won t harm synthetics Safe for lingerie Strong, powerful Gets dirt out Makes color bright Removes grease stains Good for greasy oil Pleasant fragrance Removes collar soil Removes stubborn stains 16

17 The sample correlation matrix is V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V

18 The correlations between the variables are ranging from 0.17 to From factor analysis point of view this means that there are probably two or more factors behind the correlations of the variables. 18

19 Data Appropriate for Factor Analysis? As stated above, the main goal of the factor analysis is to explain the intercorrelations between observed variables with a small number of common factors. High correlations among the variables indicate that the variables can be grouped to homogeneous sets of variables such that each set measures the same underlying constructs or dimensions. Low correlations indicate that variables do not have much in common. 19

20 Kaiser s measure of overall sampling adequacy (MSA) provides a measure to assess to what extend the observed variables belong together (are homogeneous): MSA Recommendation 0.90 Marvelous Meritorious Middling Mediocre Miserable < 0.50 Unacceptable The overall MSA should be over 0.80, but a measure of above 0.60 is tolerable. In the above example MSA = Thus, the correlation matrix should be appropriate for factoring. 20

21 Number of common factor (i) General (heuristic) rules: a) As many factors as there are eigenvalues greater than one in the correlation matrix. b) Scree plot rule. c) As many factors as can be interpreted. (ii) Statistical rules: d) Likelihood ratio test: H 0 : m factors are sufficient H 1 : more factors are needed. e) Criterion functions (AIC, SC). Select that number of factors where the used criterion assumes the minimum. 21

22 The statistical rules are based on comparison how well the sample covariance (or correlation) matrix S can be reproduced by the fitted covariance matrix (4) ˆΣ = ˆΛˆΛ + ˆΘ δ. If the fit is perfect, S ˆΣ = 0. 22

23 Example 4.5 Eigenvalues and scree-plot for the detergent correlation matrix. When using a correlation matrix as data in SPSS, a little extra work is needed. The data is organized in Excel as as follows. rowtype_ varname_ v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 n corr v1 1 corr v corr v corr v corr v corr v corr v corr v corr v corr v corr v corr v

24 The factor analysis must be run using the syntax, where it can be indicated to the program that we analyze a correlation matrix. FACTOR /matrix in(cor=*) /ANALYSIS V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 /PRINT INITIAL CORRELATION DET EXTRACTION /CRITERIA factors(2) ITERATE(25) /EXTRACTION ml /ROTATION varimax /plot = eigen rotation /METHOD=CORRELATION. 24

25 Total Variance Explained Component Initial Eigenvalues Total % of Variance Cumulative % Total % of Variance Cumulative % Extraction Method: Principal Component Analysis. Extraction Sums of Squared Loadings Scree Plot Eigenvalue Component Number

26 Both the scree plot and the eigenvalue rule suggest two factors. ================================================= m AIC BIC ChiSq df p-val(chisq) * * ================================================= * = minimum The chi-square test indicates that more than two factors are needed. AIC suggests three and or BIC two factors. Combining the results of different criteria, the two factor solution seems appropriate. 26

27 The Factor Extraction Earlier the most popular method was the (iterated) Principal Axis procedure, which is technically analogous to the principal component solution, which is the default extraction method in SAS and SPSS. Another popular method now is the method of Maximum Likelihood for its several statistically attractive properties. 27

28 Example 4.6: In the detergent data the factors are extracted using the ML-method. Factor Matrix(a) ================================================= Factor v1: Gentle to natural fabrics, v2: Won t harm colors, v3: Won t harm synthetics, v4: Safe for lingerie, v5: Strong, powerful, v6: Gets dirt out, v7: Makes color bright, v8: Removes grease stains, v9: Good for greasy oil, v10: Pleasant fragrance, v11: Removes collar soil, v12: Removes stubborn stains ================================================= Extraction Method: Maximum Likelihood. a 2 factors extracted. 4 iterations required. 28

29 The variables are clustered in the factors space as follows Factor Plot V2 V1 V3 V4 Factor V6 V7 V8-0.3 V5 V Factor

30 Communalities The factor model (1) x i = λ i1 ξ λ im ξ m + ɛ i divides the variability of x in to two parts: the systematic part explained by the factors and the error part. The variance of x decomposes accordingly as varian (5) Var[x i ] = h 2 i + Var[ɛ i], where (6) h 2 i = Var[λ i1ξ λ im ξ m ] is called the communality of variable x i (variance due to the common factors). 30

31 Communality estimates of the detergent data are the following Communalities ============================================ Initial Extraction Gentle to natural fabrics, Won t harm colors, Won t harm synthetics, Safe for lingerie, Strong, powerful, Gets dirt out, Makes color bright, Removes grease stains, Good for greasy oil, Pleasant fragrance, Removes collar soil, Removes stubborn stains ============================================ Extraction Method: Maximum Likelihood. The initial column indicates the starting value for the solution algorithm of the factor solution. The final communalities indicate that the factor structure explain from 43.4 to 77.1 percent of the variances of the x-variables. 31

32 Factor Rotation A high loading of a variable on a factor indicates that the variable is strongly related to the factor. As a consequence the interpretation (naming) of a specific factor is based on those variables that have high (absolute) loadings on that factor. However, because there is no hierarchy between the factors as was the case of PCA, the first factor extraction is arbitrary in the sense that statistically there are indefinitely many equivalent solutions. The the factor axes can be rotated to clarify the structure. 32

33 This procedure can be utilized to reach a simple structure whose prototype is as follows (p = 10, m = 3) f 1 f 2 f 3 x x x x x x x x x x Hence, the most simple structure is if each variable loads only on a single factor. 33

34 The rotation methods are divided into roughly into two classes: Orthogonal and oblique methods. The most popular orthogonal rotation methods are VARIMAX and QUARTIMAX. The former aims especially at the above described simple structure, and the latter to a solution where there is expected to be a general factor and some special factors. 34

35 Example 4.7: VARIMAX rotated solution for the detergent two-factor solution is the following Factor Plot in Rotated Factor Space 0.9 V1 V3 V4 0.6 V2 Factor V11 V10 V7 V8 V9 V6 V12 V Factor Varimax Rotated Factor Matrix ============================================== Factor v1: Gentle to natural fabrics, v2: Won t harm colors, v3: Won t harm synthetics, v4: Safe for lingerie, v5: Strong, powerful, v6: Gets dirt out, v7: Makes color bright, v8: Removes grease stains, v9: Good for greasy oil, v10: Pleasant fragrance, v11: Removes collar soil, v12: Removes stubborn stains ============================================== Extraction Method: Maximum Likelihood. Rotation Method: Varimax with Kaiser Normalization. 35

36 The rotated solution indicates that variables V 1 V 4 are strongly associated to the second factor, and the rest to first. All the four first variables are measuring some mildness properties and the rest eight variables the efficacy of cleaning clothes. As a result the first factor could be named as mildness or gentleness and the second efficacy. Note: Usually the solution is not this clear cut, but variables are loading on several factors at the same time. 36

37 Remark 4.4: The factor loadings in the orthogonal (standardized) solution are at the same time correlations of the original variables with the factors. In the oblique rotation the correlations must be calculated separately. The correlation matrix of the original variables with the factors is called the factor structure matrix, while the rotated loading matrix is called the factor pattern matrix. 37

38 Oblimin rotated solution Pattern Matrix ============================================== Factor v1: Gentle to natural fabrics, v2: Won t harm colors, v3: Won t harm synthetics, v4: Safe for lingerie, v5: Strong, powerful, v6: Gets dirt out, v7: Makes color bright, v8: Removes grease stains, v9: Good for greasy oil, v10: Pleasant fragrance, v11: Removes collar soil, v12: Removes stubborn stains ============================================== Extraction Method: Maximum Likelihood. Rotation Method: Oblimin with Kaiser Normalization. 38

39 Factor structure of the Oblimin rotated solution Structure Matrix ================================================ Factor Gentle to natural fabrics, Won t harm colors, Won t harm synthetics, Safe for lingerie, Strong, powerful, Gets dirt out, Makes color bright, Removes grease stains, Good for greasy oil, Pleasant fragrance, Removes collar soil, Removes stubborn stains ================================================ Extraction Method: Maximum Likelihood. Rotation Method: Oblimin with Kaiser Normalization. Factor Correlation Matrix ====================== Factor ====================== Extraction Method: Maximum Likelihood. Rotation Method: Oblimin with Kaiser Normalization. 39

40 Goodness of a Factor Solution The goal of the FA is to explain the correlations between the observed variables. The more closely the factor solution can reproduce the correlations the better it is. 40

41 An overall measure for this is the RMSP (Root Mean Square off-diagonal Residual), calculated from the formula (7) RMSR = 1 p(p 1)/2 p 1 i 1 i=1 j=1 (r ij ˆρ ij ) 2, where ˆρ ij is the correlation of x i and x j predicted by the factor model. The overall measure for this is the RMSP (Root Mean Square off-diagonal Partials) calculated from the residual matrix. 41

42 In the detergent example RMSR = and RMSP = Yet other approaches are to investigate the standardized residual matrix and partial correlations where factors are partialed out. An absolute value of a standardized residual correlation larger than two indicates that the corresponding correlation may not explained satisfactorily by the factor solution. Large partial correlations indicate that the common factors do not explain the corresponding correlation. 42

43 An approximate standard error for a (partial) correlation is 1/ n 1 if the population partial correlation is zero (factor model captures the correlation). Thus, as a common rule, (residual or partial) correlations on absolute value larger than 2/ n 1 deserve attention. 43

44 Example 4.8: (Continues) SAS produced residual correlations are as follows Residual Correlations With Uniqueness on the Diagonal V1 V2 V3 V4 V V V V V V V V V V V V Residual Correlations With Uniqueness on the Diagonal V5 V6 V7 V8 V V V V V V V V V V V V

45 Residual Correlations With Uniqueness on the Diagonal V9 V10 V11 V12 V V V V V V V V V V V V Root Mean Square Off-Diagonal Residuals: Overall = V1 V2 V3 V4 V5 V Root Mean Square Off-Diagonal Residuals: Overall = V7 V8 V9 V10 V11 V

46 Partial Correlations Controlling Factors V1 V2 V3 V4 V V V V V V V V V V V V Partial Correlations Controlling Factors V5 V6 V7 V8 V V V V V V V V V V V V

47 Partial Correlations Controlling Factors V9 V10 V11 V12 V V V V V V V V V V V V Root Mean Square Off-Diagonal Partials: Overall = V1 V2 V3 V4 V5 V Root Mean Square Off-Diagonal Partials: Overall = V7 V8 V9 V10 V11 V

48 Factor Scores Factor analysis is often used as a preliminary data reduction step as a part of a larger analysis. In a later analysis factors themselves are used in place of the original variables from which they were derived. Therefore, the factor values, called factor scores are needed. 48

49 Factor scores are calculated as (8) ˆξ j = a j1 x 1 + a j2 x a jp x p, (j = 1,..., m) where the weights depend on factor loadings and residual variances. Remark 4.5: Factor scores cannot be calculated if the correlation matrix is available only. 49