Multivariate normal distribution and testing for means (see MKB Ch 3)

Transcription

1 Multivariate normal distribution and testing for means (see MKB Ch 3) Where are we going? 2 One-sample t-test (univariate) Two-sample t-test (univariate) One sample T 2 -test (multivariate) Two sample T 2 -test (multivariate) Multivariate normal distribution 7 Definition Why is it important in multivariate statistics? Why is it important in multivariate statistics? Characterization Properties Properties Data matrices Wishart distribution 15 Quadratic forms of normal data matrices Definition Properties Properties Properties Hotelling s T 2 distribution 21 Definition One-sample T 2 test Relationship with F distribution Mahalonobis distance Two-sample T 2 test Final remarks 27 Assumptions for one-sample T 2 test Assumptions for two-sample T 2 test Multivariate test versus univariate tests

2 Where are we going? 2 / 30 One-sample t-test (univariate) Suppose that x 1,...,x n are i.i.d. N(µ,σ 2 ). Then x = 1 n n i=1 x i N(µ,σ 2 /n) ns 2 = n i=1 (x i x) 2 σ 2 χ 2 n 1 x and s 2 are independent Suppose that the mean µ is unknown. Then we can do a one-sample t-test: H 0 : µ = µ 0, H a : µ µ 0. Test statistic: t = x µ 0 s u/ n, where s u is sample standard deviation with n 1 in the denominator. Under H 0, t t n 1, i.e., it has a student t-distribution with n 1 degrees of freedom. Compute p-value. If p-value < 0.05, we reject the null hypothesis. 3 / 30 Two-sample t-test (univariate) Suppose that x 1,...,x n are i.i.d. N(µ X,σX 2 ), and let y 1,...,y m be i.i.d. N(µ Y,σY 2 ) (independent of x 1,...,x n ). Suppose we want to test whether µ X = µ Y, under the assumption that σ X = σ Y. Then we can do a two-sample t-test: H 0 : µ X = µ Y, H a : µ X µ Y. x ȳ Test statistic: t = where 1n 1m ( + ), s 2 p s 2 p = 1 ( ns 2 n + m 2 X + ms 2 ) Y. Under H 0, t t n+m 2. 4 / 30 One sample T 2 -test (multivariate) Suppose that x 1,...,x n are i.i.d. N p (µ,σ). Then x = 1 n n i=1 x i N p (µ,σ/n). ns W p (Σ,n 1), where S is the sample covariance matrix and W p is a Wishart distribution. x and S are independent. Suppose that the mean µ is unknown. Then we can do a one-sample t-test: H 0 : µ = µ 0, H a : µ µ 0. Test statistic: T = n( x µ 0 ) S 1 u ( x µ 0), where S u is the sample covariance matrix with n 1 in the denominator. Under H 0, T T 2 (p,n 1), i.e., it has Hotelling s T 2 distribution with parameters p and n 1. 5 / 30 2

3 Two sample T 2 -test (multivariate) Suppose that x 1,...,x n are i.i.d. N p (µ X,Σ X ), and let y 1,...,y m be i.i.d. N p (µ Y,Σ Y ) (independent of x 1,...,x n ). Suppose we want to test whether µ X = µ Y, under the assumption that Σ X = Σ Y. Then we can do a two-sample t-test: H 0 : µ X = µ Y, H a : µ X µ Y. Test statistic: T = (nm/(n + m))( x ȳ) S 1 u ( x ȳ), where S u = 1 n + m 2 (ns 1 + ms 2 ). Under H 0, T T 2 (p,n + m 2). 6 / 30 Multivariate normal distribution 7 / 30 Definition A random variable x R has a univariate normal distribution with mean µ and variance σ 2 (we write x N(µ,σ 2 )) iff its density can be written as f(x) = 1 ( exp 1 (x µ) 2 ) 2πσ 2 σ ( 2 = {2πσ 2 } 1/2 exp 1 ) 2 (x µ){σ2 } 1 (x µ). A random vector x R p has a p-variate normal distribution with mean vector µ and covariance matrix Σ (we write x N p (µ,σ)) iff its density can be written as ( f(x) = 2πΣ 1/2 exp 1 ) 2 (x µ) Σ 1 (x µ). 8 / 30 Why is it important in multivariate statistics? It is an easy generalization of the univariate normal distribution. Such generalizations are not obvious for all univariate distributions; sometimes there are several plausible ways to generalize them. The multivariate normal distribution is entirely defined by its first two moments. Hence, it has a sparse parametrization using only p(p + 3)/2 parameters (see board). In the case of normal variables, zero correlation implies independence, and pairwise independence implies mutual independence. These properties do not hold for many other distributions. 9 / 30 3

4 Why is it important in multivariate statistics? Linear functions of a multivariate normal are univariate normal. This yields simple derivations. Even when the original data is not multivariate normal, certain functions like the sample mean will be approximately multivariate normal due to the central limit theorem. The multivariate normal distribution has a simple geometry. Its equiprobability contours are ellipsoids (see picture on slide). 10 / 30 Characterization x is p-variate normal iff a x is univariate normal for all fixed vectors a R p (To allow for a = 0, we regard constants as degenerate forms of the normal distribution.) Geometric interpretation: x is p-variate normal iff its projection on any univariate subspace is normal. This characterization will allow us to derive many properties of the multivariate normal without writing down densities. 11 / 30 Properties (Th of MKB) If x is p-variate normal, and if y = Ax + c where A is any q p matrix and c is any q-vector, then y is q-variate normal (see proof on board). (Cor of MKB) Any subset of elements of a multivariate normal vector are multivariate normal. In particular, all individual elements are univariate normal (see proof on board). 12 / 30 Properties (Th of MKB) If x N p (µ,σ) and y = Ax + c, then y N q (Aµ + c,aσa ) (see proof on board). (Cor of MKB) If x N p (µ,σ) with Σ > 0, then y = Σ 1/2 (x µ) N p (0,I) and (x µ) Σ 1 (x µ) = p i=1 y2 i χ2 p (see proof on board). 13 / 30 4

5 Data matrices Let x 1,...,x n be a random sample from N(µ,Σ). Then we call X = (x 1,...,x n ) a data matrix from N(µ,Σ) or a normal data matrix. (Th of MKB, without proof) If X(n p) is a normal data matrix from N p (µ,σ) and if Y (m q) satisfies Y = AXB, then Y is a normal matrix iff the following two properties hold: A1 = α1 for some scalar α, or B µ = 0 AA = βi for some scalar β, or B ΣB = 0 When both these conditions are satisfied then Y is a normal data matrix from N q (αb µ,βb ΣB). Note: Pre-multiplication with A means that we take linear combinations of the rows. The conditions on A ensure that the new rows are independent. Post-multiplication with B means that we take linear combinations of the columns (variables). 14 / 30 Wishart distribution 15 / 30 Quadratic forms of normal data matrices We now consider quadratic forms of normal data matrices, i.e., functions of the form X CX for some symmetric matrix C. A special case of such a quadratic form is the covariance matrix, which we obtain when C = n 1 (I n 1 11 ) (see proof on board; H = I n 1 11 is called the centering matrix). 16 / 30 Definition If M(p p) can be written as X X where X(m p) is a data matrix from N p (0,Σ), then M is said to have a p-variate Wishart distribution with scale matrix Σ and m degrees of freedom. We write M W p (Σ,m). When Σ = I p, then the distribution is said to be in standard form. Note: The Wishart distribution is a multivariate generalization of the χ 2 distribution: when p = 1, the W 1 (σ 2,m) distribution is given by x x, where x R m contains i.i.d. N 1 (0,σ 2 ) variables. Hence, W 1 (σ 2,m) = σ 2 χ 2 m. 17 / 30 5

6 Properties (Th of MKB) If M W p (Σ,m) and B is a p q matrix, then B MB W q (B ΣB,m) (see proof on board). (Cor of MKB) Diagonal submatrices of M (square submatrices of M whose diagonal corresponds to the diagonal of M) have a Wishart distribution. (Cor of MKB) Σ 1/2 MΣ 1/2 W p (I,m) (Cor of MKB) If M W p (I,m) and B(p q) satisfies B B = I q, then B MB W q (I,m) (Cor of MKB) The ith diagonal element of M, m ii, has a σ 2 i χ2 m distribution (where σ2 i is the ith diagonal element of Σ). All these corollaries follow by choosing particular values of B in Th / 30 Properties (Th of MKB) If M 1 W p (Σ,m 1 ) and M 2 W p (Σ,m 2 ), and if M 1 and M 2 are independent, then M 1 + M 2 W p (Σ,m 1 + m 2 ) (see proof on board). 19 / 30 Properties (Th of MKB) If X(n p) is a data matrix from N p (0,Σ) and C(n n) is a symmetric matrix, then X CX has the same distribution as a weighted sum of independent W p (Σ,1) matrices, where the weights are eigenvalues of C; X CX has a Wishart distribution if C is idempotent. In this case X CX W p (Σ,r) where r = tr(c) = rank(c); If S = n 1 X HX is the sample covariance matrix, then ns W p (Σ,n 1). (See proof on board) 20 / 30 Hotelling s T 2 distribution 21 / 30 Definition If α can be written as md M 1 d where d and M are independently distributed as N p (0,I) and W p (I,m), then we say that α has the Hotelling T 2 distribution with parameters p and m. We write α T 2 (p,m). Hotelling s T 2 distribution is a generalization of the student t-distribution. If x t m, then x 2 T 2 (1,m). 22 / 30 6

7 One-sample T 2 test (Th of MKB) If x and M are independently distributed as N p (µ,σ) and W p (Σ,m) then m(x µ) M 1 (x µ) T 2 (p,m) (see proof on board). (Cor of MKB) if x and S are the mean vector and covariance matrix of a sample of size n from N p (µ,σ), and S u = (n/(n 1))S, then T1 2 = (n 1)( x µ) S 1 ( x µ) = n( x µ)su 1 ( x µ) has a T 2 (p,n 1) distribution (see proof on board). 23 / 30 Relationship with F distribution The T 2 distribution is not readily available in R. But the T 2 distribution is closely related to the F-distribution: (Th of MKB, without proof) T 2 (p,m) = {mp/(m p + 1)}F p,m p+1. (Cor of MKB) If x and S are the mean and covariance of a sample of size n from N p (µ,σ), then n p (n 1)p T 2 1 has a F p,n p distribution. 24 / 30 Mahalonobis distance The so-called Mahalonobis distance between two populations with means µ 1 and µ 2 and common covariance matrix Σ is given by, where 2 = (µ 1 µ 2 ) Σ 1 (µ 1 µ 2 ) In other words, D is the euclidian distance between the re-scaled vectors Σ 1/2 µ 1 and Σ 1/2 µ 2. The sample version of the Mahalonobis distance, D, is defined by D 2 = ( x 1 x 2 ) S 1 u ( x 1 x 2 ), where S u = (n 1 S 1 + n 2 S 2 )/(n 2), x i is the sample mean of sample i, n i is the sample size of sample i, and n = n 1 + n / 30 7

8 Two-sample T 2 test (Th of MKB, without proof) If X 1 and X 2 are independent data matrices, and if the n i rows of X i are i.i.d. N p (µ i,σ i ), i = 1,2, then when µ 1 = µ 2 and Σ 1 = Σ 2, has a T 2 (p,n 2) distribution. T 2 2 = n 1n 2 n D2 Corollary: n 1 p (n 2)p T 2 2 has a F p,n 1 p distribution. 26 / 30 Final remarks 27 / 30 Assumptions for one-sample T 2 test We have a simple random sample from the population The population has a multivariate normal distribution 28 / 30 Assumptions for two-sample T 2 test We have a simple random sample from each population In each population the variables have a multivariate normal distribution The two populations have the same covariance matrix 29 / 30 Multivariate test versus univariate tests See board 30 / 30 8