Multidimensional data and factorial methods

Transcription

1 Multidimensional data and factorial methods Bidimensional data x X 3 6 X x Cartesian plane Multidimensional data n X x x x n X x x x n X m x m x m x nm Factorial plane

2 Interpretation of points position Distance between points: Similarity between individuals Connection between variables

3 n X x x x n X x x x n X m x m x m x nm Individuals space Variables space Clusters of individuals Associations among variables

4 Dimensions reduction n points (units) in R p n points on a plane (R ) p points (variables) in R n p points on a plane (R ) In each space we look for two axes which identify the best plane where to represent (= to project) the n or p points Two separate analyses: reduction of R p searching the two axes which are closer (= mostly correlated) to all variables (syntheses of variables) reduction of R n searching the two axes which are closer to all units

5 tr = trace of a square matrix Sum of diagonal elements: tr ( X) n = i = x ii Some trace properties: ( ) A Β = ( B A) ( A) = ( A' ) ( A' Α) = ij n m m n n m tr tr tr tr tr a r = rank of a matrix Maximum number of linearly independent rows or columns i = j = X n p se r( X) = min( np, ) X has full rank A - = inverse of matrix A Theorem: if A is a square matrix (with det(a) 0), then there is only one matrix B such that: AB=BA=I B is called inverse of A and is denoted by A -

6 Geometrical representation of vecotrs In a given m-dimensional space a point P with coordinates (x, x m ) is a vector v from the space origin to the point P m = : the space is a plane and P has coordinates (x, x ) A vector v is uniquely identified by two elements: x v P norm (or length, or size) and versus x Both norm and direction are determined based on x and x

7 Euclidean norm 3 v = 3 From Pitagorean theorem: v norm: v = v i i v = + 3 = = 3 = 3,6 The norm can then be expressed as follows: v = v' v

8 Geometrical meaning of the operations between vectors The scalar product between vectors is proportional to the orthogonal projection of the first on the second x x θ x x The product is proportional to the cosine of the angle between them: ( ) cos θ = x x ' x x ( ) x ' x = cos θ x x cos(90) = 0 The product of orthogonal vectors is equal to 0

9 Orthogonal projection of a vector x v has norm : the coordinate on U for all points is a multiple of v v ˆx U ˆx is the projection of x on the axis U spanned by v ie: ˆx is a linear combination of v: ˆx = x' v ˆx is the coordinate of x on U Constraints: the difference vector must be ( x xˆ )' v = 0 orthogonal to v: v norm is = v' v =

10 Synthesis of a matrix A synthesis of relations among individuals and variables in a multidimensional space can be given by projecting all points on an axis or on a plan and studying distances among them A C B D Such projections can only approximate existing relations since originary distances are distorted

11 Orthogonal projection of x i on u x U i O u M i H i Orthogonal projection of OM i on U OH p = xu = x u i i ij j j= Orthogonal projection of X on u x xj xp u p Xu = xi xij x ip u i = xiju j j= x u n xnj x np p Coordinate of i-th row of X on u

12 Fitting a subspace to an hyperspace Least squares method: (cloud of points in a p-dimensional space R p ) n = i = ( ) minq MH i i We look for the vector u identifying the axis U that passes as close as possible to all point in the cloud, ie that minimizes the square distances between U and points in the original space

13 Least squares method: x U i O u M i H i Being M i H i orthogonal to U (for all i =,,n) from Pythagorean theorem for each triangle M i H i O we have: n n n ( MH i i ) = ( OMi ) ( OHi ) i = i = i = Being n ( OM ) i a constant: min ( MH i i ) max ( OHi ) i = n n i= i=

14 Determination of u min n ( ) n MH max ( ) i i OHi i = i = n ( OHi ) = ( Xu) Xu = u X Xu i = Constraint: u has norm one (the squares of elements in u add up to ) The maximization function is then: max ( ) u X Xu u u u =

15 Optimal subspace for units In R p max ( ) u X Xu u u u = Maximum problem Unitary norm constraint Lagrange multiplier For optimization (maximization) of functions under constraints Objective function: f = u X Xu ' = max! λ ( u u ) Solution: any vector u such that: = X Xu λu Singular Values Decomposition (SVD) (Eckart Young theorem, 936) λ= eigenvalues of X X u = eigenvectors of X X

16 Eigenvalues properties Each eigenvalue λ has infinite eigenvectors (all collinear) A n,n and A n,n have the same eigenvalues but different eigenvectors tr(a) = sum of eigenvalues of A Symmetrical matrices with real elements: Eigenvalues belong to real numbers Rank = number of not null eigenvalues

17 Global information = variance correlation/association independent vectors are orthogonal: their scalar product is equal to 0 Tr(X X) = total variance = sum of eigenvalues = global information Eigenvalues are determined as independent portion of global information, because Eigenvectors are orthogonal (by constraints): they can span a reference system ( eigenvectors span a plane)

18 Orthogonal subspace of X X When we decompose X X (covariance matrix): Characteristic equation: X Xu = λu X' X σ σ p = σp σp evalues λ λ λ 3 evectors Independent portions of total variance The information summarized by the eigenvalue is represented geometrically by the corresponding eigenvector How much of the information in X (in data) is represented by the axes? Total of information: p tr ( X' X) p = λ = σ i i= i= i Each eigenvalue explains a fraction equal to: λ p i= λ i = tr λ ( X' X)

19 Geometrical representation of a matrix X n statistical units and p variables (continuous or categorical) Data matrix j p X = i x ij n p R Units space N(n) = cluod of n units-points in R p Square matrix to reduce: X X p p n R N(p) = cloud of p variables-points in R n Square matrix to reduce: Variables space XX n n

20 Optimal subspace for units In R p max ( ) u X Xu u u u = Maximum problem Unitary norm constraint Lagrange multiplier For optimization (maximization) of functions under constraints Objective function: f = u X Xu ' = max! λ ( u u ) Solution: any vector u such that: = X Xu λu ie eigenvectors of X X

21 Optimal subspace for units Aim: to build a plane where projecting units, ie to find orthogonal axes The maximum problem has p solutions, ie eigenvecotrs associated to the p eigenvalues of X X Each eigenvector is a possible axis, its eigenvalue measures its representative power

22 Determination of the first axis max( ) u u X Xu u u = Maximum problem Unitary norm constraint The first axis lays along the eigenvector u with norm associated to the highest eigenvalue λ u explains (= represents) a percentage of global information equal to: λ p λ i= i = tr λ ( X' X) First axis information Total variance

23 Determination of the second axis (and the others) max( ) u u X Xu u u = u u = 0 Maximum problem Unitary norm constraint Orthogonality constraint The second eigenvalue λ (the highest after λ ) defines the eigenvector u, orthogonal to u, that the best dimensions subspace The plane defined by the firste eigenvectors u and u explains a percentage of global information equal to: λ + λ λ i= λ + λ = p i tr ( X' X) First axes information Total variance

24 Coordinate of i-th unit-point x U i O u M i H i Orthogonal projection of OM i on u OH = = p xu x u i i ij j j= The coordinate of the unit i on axis α is: α ( ) c i = xu i α The coordinates vector of all n points on axis α is: c α = Xu α

25 Optimal subspace for variables In R n R n = variables space N(p) = cloud of p variables-points max ( ν) ν' XX' ν ν ν = ( v v ) v' XX' v µ ' = max! Characteristic equation: XX ν = µ ν α α α α =,, n µ = α-th eigenvector ν α α = α-th eigenvalue * ( ) Coordinate of the j-th variable (j=,,p) on axis α: c j = x ν α j α Coordinate vector of all p variables on axis α: c α = X ν α

26 Duality between R n ed R p Relation between eigenvalues of X X e XX λ α = µ α for all α =,,p Moreover: X X has p eigenvalues λ α, α =,, p XX has n eigenvalues µ α = λ α, α =,, n, but only p of them are 0 (n - p are always = 0) Relation between eigenvectors of X X e XX Transition formulas: ν = Xu u = X ν u α e v α are proportional α α α α λα λα

27 Reconstruction of matrix X X p = λ α= ν u α α α Or: n α= X = µ u v α α α, p u X = λ + λ, p, p, p u + + λp u p n n n n v v v p X = X + X + + Xp

28 Factorial analysis steps Caractéristiques de 4 modèles de voiture (Source : L argus de l automobile, 004) Cylindrée Puissance Vitesse Poids Largeur Longueur Modèle (cm 3 ) (ch) (km/h) (kg) (mm) (mm) Citroën C Base Smart Fortwo Coupé Mini Nissan Micra Renault Clio 30 V Audi A3 9 TDI Peugeot HDI Peugeot V6 BVA Mercedes Classe C 70 CDI BMW 530d Jaguar S-Type 7 V6 Bi-Turbo BMW 745i Mercedes Classe S 400 CDI Citroën C3 Pluriel 6i BMW Z4 5i Audi TT 8T Aston Martin Vanquish Bentley Continental GT Ferrari Enzo Renault Scenic 9 dci Volkswagen Touran 9 TDI Land Rover Defender Td Land Rover Discovery Td Nissan X-Trail dci M i x u i u H O i 3 Land Rover Discovery Nissan X-Trail d Volkswagen Touran Jaguar S-Type 7 V6 Land Rover Defender Peugeot V6 Mercedes Classe S Renault Scenic 9 d Mercedes Classe C BMW 745i Peugeot HDI BMW 530d Bentley Continental 0 Citroën C3 Pluriel Nissan Micra Audi A3 9 TDI Audi TT 8T 80 Aston Martin Vanquish Citroën C BMW Z4 5i - - Mini 6 70 Renault Clio 30 V6 Smart Fortwo Coupé Ferrari Enzo Data type Criterion to maximize Matrix to decompose Coordinates of points Eigenvectors Eigenvalues Data type Meaning of: Factorial axes, Factorial plans, Points position