DATA REDUCTION TECHNIQUES

Transcription

1 DATA REDUCTION TECHNIQUES RANJANA AGARWAL AND A.R. RAO Indian Agricultural Statistics Research Institute Library Avenue, New Delhi Introduction Data reduction techniques are applied where the goal is to aggregate or amalgamate the information contained in large data sets into manageable (smaller) information nuggets. Data reduction techniques can include simple tabulation, aggregation (computing descriptive statistics) or more sophisticated techniques like principal components analysis, factor analysis. Here, mainly principal component analysis (PCA) and factor analysis are covered along with examples and software demonstration. Different commands used in SPSS and SAS packages for PCA and factor analyses are also given.. Principal Component Analysis Often, the variables under study are highly correlated and as such they are effectively saying the same thing. It may be useful to transform the original set of variables to a new set of uncorrelated variables called principal components. These new variables are linear combinations of original variables and are derived in decreasing order of importance so that the first principal component accounts for as much as possible of the variation in the original data. Also, PCA is a linear dimensionality reduction technique, which identifies orthogonal directions of maximum variance in the original data, and projects the data into a lower-dimensionality space formed of a sub set of the highest variance components. Let x 1, x, x 3,...,x p are variables under study, then first principal component may be defined as z 1 = a 11 x 1 + a 1 x a 1p x p such that variance of z 1 is as large as possible subject to the condition that a11 + a a1p = 1 This constraint is introduced because if this is not done, then Var(z 1 ) can be increased simply by multiplying any a 1j s by a constant factor. The second principal component is defined as z = a 1 x 1 + a x a p x p such that Var(z ) is as large as possible next to Var( z 1 ) subject to the constraint that a 1 p = + a a 1 and cov(z 1, z ) = 0 and so on. It is quite likely that first few principal components account for most of the variability in the original data. If so, these few principal components can then replace the initial p variables in subsequent analysis, thus, reducing the effective dimensionality of the problem. An analysis of principal components often reveals relationships that were not

2 previously suspected and thereby allows interpretation that would not ordinarily result. However, Principal Component Analysis is more of a means to an end rather than an end in itself because this frequently serves as intermediate steps in much larger investigations by reducing the dimensionality of the problem and providing easier interpretation. It is a mathematical technique, which does not require user to specify the statistical model or assumption about distribution of original variates. It may also be mentioned that principal components are artificial variables and often it is not possible to assign physical meaning to them. Further, since Principal Component Analysis transforms original set of variables to new set of uncorrelated variables, it is worth stressing that if original variables are uncorrelated, then there is no point in carrying out principal component analysis. Computation of Principal Components and Scores Consider the following data on average minimum temperature (x 1 ), average relative humidity at 8 hrs. (x ), average relative humidity at 14 hrs. (x 3 ) and total rainfall in cm. (x 4 ) pertaining to Raipur district from 1970 to 1986 for kharif season from 1st May to 7th Oct. S.No. x 1 x x 3 x Mean S.D The variance-covariance matrix is as follows: = The eigen values (λ i 's) and eigen vectors (a i ) of the above matrix is obtained and is given as follows in decreasing order alongwith the corresponding eigen vectors:

3 λ 1 = a 1 = (0.006, 0.061, 0.103, 0.993) λ = a = (0.955, -0.96, 0.011, 0.01) λ 3 = 7.87 a 3 = (0.141, 0.485, 0.855, ) λ 4 = a 4 = (0.60, 0.80, , 0.001) The principal components for this data will be z 1 = x x x x 4 z = x x x x 4 z 3 = x x x x 4 z 4 = 0.6 x x x x 4 The variance of principal components will be eigen values i.e. Var(z 1 ) = , Var(z ) =18.375, Var (z 3 ) = 7.87, Var(z 4 ) = The total variation explained by original variables is Var(x 1 ) + Var(x ) + Var(x 3 ) + Var(x 4 ) = = The total variation explained by principal components is λ 1 + λ + λ 3 + λ 4 = = As such, it can be seen that the total variation explained by principal components is same as that explained by original variables. It could also be proved mathematically as well as empirically that the principal components are uncorrelated. The proportion of total variation accounted for by the first principal component is λ 1 λ1 3 4 = = 0.97 Continuing, the first two components account for a proportion λ 1 λ 1 of the total variance. 3 4 = = 0.99 Hence, in further analysis, the first or first two principal components z 1 and z could replace four variables by sacrificing negligible information about the total variation in the system. The scores of principal components can be obtained by substituting the values of x i s in equations of z i s. For above data, the first two principal components for first observation i.e. for year 1970 can be worked out as 3

4 z 1 = x x x x = z = x x x x = Similarly for the year 1971 z 1 = x x x x = z = x x x x = Thus the whole data with four variables can be converted to a new data set with two principal components. Note: The principal components depend on the unit of measurement, for example, if in the above example x 1 is measured in 0 F instead of 0 C and x 4 in mm in place of cm, the data gives different principal components when transformed to original x s. In very specific situations the results are same. The conventional way of getting around this problem is to use standardized variables with unit variances, i.e., correlation matrix in place of dispersion matrix. But the principal components obtained from original variables as such and from correlation matrix will not be same and they may not explain the same proportion of variance in the system. Further more, one set of principal components is not simple function of the other. When the variables are standardized, the resulting variables contribute almost equally to the principal components determined from correlation matrix. Variables should probably be standardized if they are measured in units with widely differing ranges or if measured units are not commensurate. Often population dispersion matrix or correlation matrix is not available. In such situations, sample dispersion matrix or correlation matrix can be used. Applications of Principal Components The most important use of principal component analysis is reduction of data. It provides the effective dimensionality of the data. If first few components account for most of the variation in the original data, then first few components scores can be utilized in subsequent analysis in place of original variables. Plotting of data becomes difficult with more than three variables. Through principal component analysis, it is often possible to account for most of the variability in the data by first two components, and it is possible to plot the values of first two components scores for each individual. Thus, principal component analysis enables us to plot the data in two dimensions. Particularly detection of outliers or clustering of individuals will be easier through this technique. Often, use of principal component analysis reveals grouping of variables which would not be found by other means. Reduction in dimensionality can also help in analysis where number of variables is more than the number of observations, for example, in discriminant analysis and regression analysis. In such cases, principal component analysis is helpful by reducing the dimensionality of data. Multiple regression can be dangerous if independent variables are highly correlated. Principal component analysis is the most practical technique to solve the problem. Regression analysis can be carried out using principal components as regressors in place of original variables. This is known as principal component regression. 4

5 SPSS Syntax for Principal Component Analysis matrix. read x /file='c:\princomp.dat' /field=1 to 80 /size={17,4}. compute x1=sscp(x). compute x=csum(x). compute x3=nrow(x). compute x4=x/x3. compute x5=t(x4)*x4. compute x6=(x1-(x3*x5))/16. call eigen(x6,a, λ ). compute P=x*a. print x. print x1. print x. print x3. print x4. print x5. print x6. print a. print λ. print P. end matrix. OUTPUT from SPSS X =(data file )

6 X6 (Variance covariance matrix) a (Eigen vectors) λ (Eigen values) P (PC Variables) SAS Commands for Principal Components Analysis data test; input x1 x x3 x4; cards; ; proc princomp cov; var x1 x x3 x4; run; 6

7 . FACTOR ANALYSIS The essential purpose of factor analysis is to describe, if possible, the covariance relationships among many variables in terms of a few underlying, but unobservable, random quantities called factors. Under the factor model assuming linearity, each response variate is represented as a linear function of a small number of unobservable common factors and a single latent specific factor. The common factors generate the covariances among the observable responses while the specific terms contribute only to the variances of their particular response. Basically the factor model is motivated by the following argument - Suppose variables can be grouped by their correlations, i.e., all variables within a particular group are highly correlated among themselves but have relatively small correlations with variables in a different group. It is conceivable that each group of variables represents a single underlying construct, or factor, that is responsible for the observed correlations. For example, for an individual, marks in different subjects may be governed by aptitudes (common factors) and individual variations (specific factors) and interest may lie in obtaining scores on unobservable aptitudes (common factors) from observable data on marks in different subjects. The Factor Model Suppose observations are made on p variables, x 1, x,,x p. The factor analysis model assumes that there are m underlying factors (m<p) f 1, f,,f m and each observed variables is a linear function of these factors together with a residual variate, so that x j = a j1 f 1 + a j f + + a jm f m + e j, j = 1,,,p where a j1, a j,,a jm are called factor loadings i.e. a jk is loading of j th variable on k th factor. e j 's are called specific factors. The proportion of the variance of the j th variable contributed by the m common factors is called the j th communality and the proportion due to the specific factors is called the uniqueness, or specific variance. Factor analysis involves: Deciding number of common factors (m) Estimating factor loadings (a ji ) Calculating factor scores (f i ) Methods of Estimation Factor analysis is done in two parts, first solution is obtained by placing some restrictions and then final solution is obtained by rotating this solution. There are two most popular methods available in literature for parameter estimation, the principal component (and the related principal factor) method and the maximum likelihood method. The solution from either method can be rotated in order to simplify the interpretation of factors i.e. either factor loadings are close to unity or close to zero. The most popular method for orthogonal rotation is Varimax Rotation Method. In some specific situations, oblique rotations are also used. It is always prudent to try more than one method of solution. If the factor model is appropriate for the problem at hand, the solutions should be consistent with one another. The estimation and rotation methods require iterative calculations that must be done on a computer. If variables are uncorrelated factor analysis will not be useful. In these 7

8 circumstances, the specific factors play the dominant role, whereas the major aim of the factor analysis is to determine a few important common factors. Number of factors is theoretically given by rank of population variance-covariance matrix. However, in practice, number of common factors retained in the model is increased until a suitable proportion of total variance is explained. Another convention, frequently encountered in computer packages is to set m equal to the number of eigen values greater than one. As in principal component analysis, principal factor method for factor analysis depends upon unit of measurement. If units are changed, the solution will change. However, in this approach estimated factor loadings for a given factor do not change as the number of factors is increased. In contrast to this, in maximum likelihood method, the solution does not change if units of measurements are changed. However, in this method the solution changes if number of common factors is changed. Example.1: In a consumer - preference study, a random sample of customers were asked to rate several attributes of a new product. The response on a 5-point semantic differential scale were tabulated and the attribute correlation matrix constructed is as given below: Attribute Correlation matrix Taste Good buy for money Flavor Suitable for snack Provides energy It is clear from the correlation matrix that variables 1 and 3 and variables and 5 form groups. Variable 4 is closer to the (,5) group than (1,3) group. Observing the results, one can expect that the apparent linear relationships between the variables can be explained in terms of, at most, two or three common factors. Solution from SAS package. Initial Factor Method: Principal Components Prior Communality Estimates: ONE Eigenvalues of the Correlation Matrix: Total = 5 Average = 1 8

9 Eigenvalue Difference Proportion Cumulative FACTOR1 FACTOR TASTE MONEY FLAVOR SNACK ENERGY Variance explained by each factor FACTOR1 FACTOR Final Communality Estimates: Total = TASTE MONEY FLAVOR SNACK ENERGY Residual Correlations with Uniqueness on the Diagonal TASTE MONEY FLAVOR SNACK ENERGY TASTE MONEY FLAVOR SNACK ENERGY Rotation Method: Varimax Rotated Factor Pattern FACTOR1 FACTOR TASTE MONEY FLAVOR SNACK ENERGY Variance explained by each factor FACTOR1 FACTOR Final Communality Estimates: Total = TASTE MONEY FLAVOR SNACK ENERGY

10 Initial Factor Method: Maximum Likelihood Eigenvalues of the Weighted Reduced Correlation Matrix: Total = Average = Eigenvalue Difference Proportion Cumulative Variance explained by each factor Factor Pattern FACTOR1 FACTOR TASTE MONEY FLAVOR SNACK ENERGY FACTOR1 FACTOR Weighted Unweighted TASTE MONEY FLAVOR SNACK ENERGY Communality Rotation Method: Varimax Rotated Factor Pattern FACTOR1 FACTOR TASTE MONEY FLAVOR SNACK ENERGY Variance explained by each factor FACTOR1 FACTOR Weighted Unweighted TASTE MONEY FLAVOR SNACK ENERGY Communality It can be seen that two factor model with factor loadings shown above is providing a good fit to the data as the first two factors explains 93.% of the total standardized sample 10

11 λ1+ λ variance, i.e., x100, where p is the number of variables. It can also be seen from p the results that there is no clear-cut distinction between factor loadings for the two factors before rotation. SAS Commands for Factor Analysis title Factor Analysis ; data consumer (type=corr); _type_= CORR ; Input name$ taste money flavour snack energy; Cards; taste money flavour snack energy ; proc factor res data=consumer method = prin nfact= rotate = varimax preplot plot; var taste money flavour snack energy; run; SPSS Steps for Factor Analysis File New Data Define Variables Statistics Data reduction Factor Variables (mark variables and click 4 to enter variables in the box) Descriptives Statistics ( a univariate descriptives, a initial solution) Corrlation matrix ( a coefficients) Extraction Method (principal components or maximum likelihood) Extract (Number of factors as ) Display (Unrotated factor solution) Continue Rotation Method (varimax) Display (loading plots) Continue Scores Display factor score coefficient matrix Continue OK. References and Suggested Reading Chatfield, C. and Collins, A.J. (1990). Introduction to multivariate analysis. Chapman and Hall publications. Johnson, R.A. and Wichern, D.W. (1996). Applied multivariate statistical analysis. Prentice-Hall of India Private Limited. 11