Chapter 4 Multivariate distributios k Multivariate Distributios All the results derived for the bivariate case ca be geeralized to RV. The joit CDF of,,, k will have the form: P(x, x,, x k ) whe the RVs are discrete F(x, x,, x k ) whe the RVs are cotiuous
Joit Probability Fuctio Defiitio: Joit Probability Fuctio Let,,, k deote k discrete radom variables, the p(x, x,, x k ) is joit probability fuctio of,,, k if px x p x x.,,.,, x x 3. P,, A p x,, x x,, x A Joit Desity Fuctio Defiitio: Joit desity fuctio Let,,, k deote k cotiuous radom variables, the f(x, x,, x k ) = δ /δx,δx,,δx k F(x, x,, x k ) is the joit desity fuctio of,,, k if. f x,, x dx,, dx. f x,, x A 3. P,, A f x,, x dx,, dx
Example: The Multiomial distributio Suppose that we observe a experimet that has k possible outcomes {O, O,, O k } idepedetly times. Let p, p,, p k deote probabilities of O, O,, O k respectively. Let i deote the umber of times that outcome O i occurs i the repetitios of the experimet. The the joit probability fuctio of the radom variables,,, k is! p x x p p p x x,, x! x! xk! x k k Example: The Multiomial distributio x x x Note: p p p k k is the probability of a sequece of legth cotaiig x outcomes O x outcomes O x k outcomes O k! x x x x x!! xk! k is the umber of ways of choosig the positios for the x outcomes O, x outcomes O,, x k outcomes O k 3
Example: The Multiomial distributio x x x x k x x x3 xk! x! x x!! x! x! x! x! x! x! xx! x3! xx x3! k x x,, x! x! xk!! p x x p p p x x x This is called the Multiomial distributio k x k k p p x x p x k k Example: The Multiomial distributio Suppose that a earigs aoucemets has three possible outcomes: O Positive stock price reactio (3% chace) O No stock price reactio (5% chace) O 3 - Negative stock price reactio (% chace) Hece p =.3, p =.5, p 3 =.. Suppose today 4 firms released earigs aoucemets ( = 4). Let = the umber that result i a positive stock price reactio, = the umber that result i o reactio, ad Z = the umber that result i a egative reactio. Fid the distributio of, ad Z. Compute P[ + Z] 4! x y z px, y, z.3.5. x yz 4 x! y! z! 4
Table: p(x,y,z) z x y 3 4.6.6.6 3. 4.65.96.7.8 3.5 4.6.8.35 3 4 3.6 3.54 3 3 3 3 4 4.8 4 4 4 3 4 4 P [ + Z] =.978 z x y 3 4.6.6.6 3. 4.65.96.7.8 3.5 4.6.8.35 3 4 3.6 3.54 3 3 3 3 4 4.8 4 4 4 3 4 4 5
Example: The Multivariate Normal distributio Recall the uivariate ormal distributio f x e the bivariate ormal distributio f x, y e x y x xx x x y x y x x x y y Example: The Multivariate Normal distributio The k-variate Normal distributio is give by: f x,, xk f x e k / / where x μ x μ x x x xk μ k k k k k kk 6
Margial joit probability fuctio Defiitio: Margial joit probability fuctio Let,,, q, q+, k deote k discrete radom variables with joit probability fuctio p(x, x,, x q, x q+, x k ) the the margial joit probability fuctio of,,, q is,,,, p x x p x x q q x x q Whe,,, q, q+, k is cotiuous, the the margial joit desity fuctio of,,, q is,,,, f x x f x x dx dx q q q Coditioal joit probability fuctio Defiitio: Coditioal joit probability fuctio Let,,, q, q+, k deote k discrete radom variables with joit probability fuctio p(x, x,, x q, x q+, x k ) the the coditioal joit probability fuctio of,,, q give q+ = x q+,, k = x k is p x,, xk p x,, xq xq,, qq k xk p x,, x q k q k For the cotiuous case, we have: qqk f x,, x x,, x q q k f x,, x f x,, x q k q k k 7
Coditioal joit probability fuctio Defiitio: Idepedece of sects of vectors Let,,, q, q+, k deote k cotiuous radom variables with joit probability desity fuctio f(x, x,, x q, x q+, x k ) the the variables,,, q are idepedet of q+,, k if,, k q,, q qk q,, k f x x f x x f x x A similar defiitio for discrete radom variables. Coditioal joit probability fuctio Defiitio: Mutual Idepedece Let,,, k deote k cotiuous radom variables with joit probability desity fuctio f(x, x,, x k ) the the variables,,, k are called mutually idepedet if,, f x x f x f x f x k k k A similar defiitio for discrete radom variables. 8
Multivariate margial pdfs - Example Let,, Z deote 3 joitly distributed radom variable with joit desity fuctio the f x, y, z K x yz x y z,, otherwise Fid the value of K. Determie the margial distributios of, ad Z. Determie the joit margial distributios of,, Z, Z Multivariate margial pdfs - Example Solutio: Determiig the value of K. f x, y, z dxdydz K x yz dxdydz 3 x x K xyz dydz K yz dydz 3 3 x y y K y z dz K z dz 3 3 y if z z 7 K K K 3 4 3 4 K 7 9
Multivariate margial pdfs - Example The margial distributio of. f x f x, y, zdydz x yzdydz 7 y y x y z dz x z dz 7 7 y z x z x for x 7 4 7 4 Multivariate margial pdfs - Example The margial distributio of,. f xy, fxyzdz,, x yzdz 7 z xz y 7 z z x y for x, y 7
Multivariate margial pdfs - Example Fid the coditioal distributio of:. Z give = x, = y,. give = x, Z = z, 3. give = y, Z = z, 4., Z give = x, 5., Z give = y 6., give Z = z 7. give = x, 8. give = y 9. give Z = z. Z give = x,. Z give = y. give Z = z Multivariate margial pdfs - Example The margial distributio of,. f x, y x y for x, y 7 Thus the coditioal distributio of Z give = x, = y is f f x, y, z x yz 7 x, y x y 7 x yz for z x y
Multivariate margial pdfs - Example The margial distributio of. f x x for x 7 4 The, the coditioal distributio of, Z give = x is f x, y, z x yz 7 f x x 7 4 x yz for y, z x 4 Expectatios for Multivariate Distributios Defiitio: Expectatio Let,,, deote joitly distributed radom variable with joit desity fuctio f(x, x,, x ) the E g,, g x,, x f x,, x dx,, dx
Expectatios for Multivariate Distributios - Example Let,, Z deote 3 joitly distributed radom variable with joit desity fuctio the f x, y, z Determie E[Z]. Solutio: otherwise 7 x yz x, y, z E Z xyz x yzdxdydz 7 7 3 x yz xy z dxdydz Expectatios for Multivariate Distributios - Example E Z xyz x yzdxdydz 7 7 3 x yz xy z dxdydz 4 x x x 3 7 yz y z dydz yz y z dydz 4 7 x 3 y y y 3 3 z z dz z z dz 7 3 7 3 y 3 3 z z 3 3 7 7 7 4 9 7 4 9 7 36 84 3
Some Rules for Expectatios Rule. E x f x,, x dx dx i i Thus you ca calculate E[ i ] either from the joit distributio of,, or the margial distributio of i. Proof: x f x,, x dx,, dx i x f x dx i i i i x f x,, x dx dx dx dx dx i i i i x f x dx i i i i Some Rules for Expectatios Rule. E a a ae ae This property is called the Liearity property. Proof:,, ax a x f x x dxdx a x f x,, xdx dx a x f x,, x dx dx 4
Some Rules for Expectatios Rule 3 3. (The Multiplicative property) Suppose,, q are idepedet of q+,, k the,, q q,, k E g,, q E h q,, k E g h I the simple case whe k =, ad g()= & h()=: E E E if ad are idepedet Some Rules for Expectatios Rule 3 Proof:,, q q,, k E g h,,,,,, g x x h x x f x x dx dx q q k k,, q q,, k,, q,, g x x h x x f x x f x x dx dx dx dx q k q q k hxq,, xkfxq,, xk gx,, xq f x,, xq dxdx q dxq dxk E g,, q,,,, h x x f x x dx dx q k q k q k 5
Some Rules for Expectatios Rule 3 E g,, q,,,, h x x f x x dx dx q k q k q k,, q q,, k E g E h Some Rules for Variace Rule. Var Var Var Cov, Proof: E w here C ov, = Thus, E Var where E E Var E Var Cov, Var 6
Some Rules for Variace Rule Note: If ad are idepedet, the E Cov, = ad Var Var Var = E E E E = Some Rules for Variace Rule - Defiitio: Correlatio coefficiet For ay two radom variables ad the defie the correlatio coefficiet to be: xy = Var Cov, Cov, Var Thus Cov, = ad Var if ad are idepedet. 7
Some Rules for Variace Rule - Recall xy = Var Cov, Cov, Var Property. If ad are idepedet, the =. (Cov(,)=.) The coverse is ot ecessarily true. That is, = does ot imply that ad are idepedet. Example: y\x 6 8 f y (y)...4.. 3...4 f x (x).4..4 E()=8, E()=, E()=6 Cov(,) =6 8* = P(=6,=)= P(=6)*P(=)=.4* *.=.8=> & are ot idepedet Some Rules for Variace Rule - Property. ad if there exists a ad b such that P ba where = + if b > ad = - if b< Proof: Let U ad V. We will pick b to miimize g(b). Let gb E V bu g b E V bu for all b. E V bvu b U E V be VU b E U 8
Some Rules for Variace Rule - Takig first derivatives of g(b) w.r.t b g b E V bu E V be VU b E U EVU gb EVU be U => b b mi E U Sice g(b), the g(b mi ) g b mi E V bmie VU bmie U E VU E VU E V E VU E[U ] E U E U E VU E U E V Some Rules for Variace Rule - EVU E V EVU Thus, or E U E U E V E E E => 9
Some Rules for Variace Rule - Note: g b mi E V bmie VU bmie U If ad oly if E V b U mi This will be true if P bmi mi i.e., P b a a b where mi PV bmiu Some Rules for Variace Rule - Summary: ad if there exists a ad b such that where ad b a P b a b E E mi Cov, = = Var b mi
Some Rules for Variace Rule a b a b ab. Var Var Var Cov, Proof Var a b E a b a b with a b E a b a b Thus, Var a b E a b a b Ea ab b a Var abcov, b Var Some Rules for Variace Rule 3 3. Var a a Var aa Cov, aa Cov, aa 3Cov, 3 aa Cov, a a Cov, a Var a i i a Var aa Cov, i i i j i j i j ai Var i if,, are mutually idepedet
The mea ad variace of a Biomial RV We have already computed this by other methods:. Usig the probability fuctio p(x).. Usig the momet geeratig fuctio m (t). Now, we will apply the previous rules for mea ad variaces. Suppose that we have observed idepedet repetitios of a Beroulli trial Let,, be mutually idepedet radom variables each havig Beroulli distributio with parameter p ad defied by i if repetitio i is S (prob p) if repetitio i is F (prob q) The mea ad variace of a Biomial RV E p q p ( p) i Var[ ] ( p) Now = + + has a Biomial distributio with parameters ad pthe, is the total umber of successes i the repetitios. E E p p p i ( p p var p ( p) p ) qp var pq pq pq q ( p) p ( p) ( p)
Coditioal Expectatio Defiitio: Coditioal Joit Probability Fuctio Let,,, q, q+, k deote k cotiuous radom variables with joit probability desity fuctio f(x, x,, x q, x q+, x k ) the the coditioal joit probability fuctio of,,, q give q+ = x q+,, k = x k is qqk f x,, x x,, x q q k f x,, x f x,, x q k q k k 3
Defiitio: Coditioal Joit Probability Fuctio Let U = h(,,, q, q+, k ) the the Coditioal Expectatio of U give q+ = x q+,, k = x k is E U x q,, x k,,,,,, qq k h x x f x x x x dx dx k q q k q Note: This will be a fuctio of x q+,, x k. Coditioal Expectatio of a Fuctio - Example Let,, Z deote 3 joitly distributed RVs with joit desity fuctio the 7 x yz x, y, z f x, y, z otherwise Determie the coditioal expectatio of U = + + Z give = x, = y. Itegratio over z, gives us the margial distributio of,: f xy, x y for x, y 7 4
Coditioal Expectatio of a Fuctio - Example The, the coditioal distributio of Z give = x, = y is f f x, y, z x yz 7 x, y x y 7 x yz for z x y Coditioal Expectatio of a Fuctio - Example The coditioal expectatio of U = + + Z give = x, = y. x yz x y x y E U x, y x y z dz x y z x yz dz x y yz y x y x z x x y dz 3 z z y yx y x x x yz x y 3 z z y yx y x x x y x y 3 5
Coditioal Expectatio of a Fuctio - Example Thus the coditioal expectatio of U = + + Z give = x, = y. E U x, y y 3 y x y x x y x x y y x x yx y x y 3 x y 3 x x y y A Useful Tool: Iterated Expectatios Theorem Let (x, x,, x q, y, y,, y m ) = (x, y) deote q + m RVs. Let U(x, x,, x q, y, y,, y m ) = g(x, y). The, EU E E U y y Var U E Var U Var E U y y y y The first result is commoly referred as the Law of iterated expectatios. The secod result is commoly referred as the Law of total variace or variace decompositio formula. 6
A Useful Tool: Iterated Expectatios Proof: (i the simple case of variables ad ) First, we prove the Law of iterated expectatios. Thus U g,,, E U g x y f x y dxdy E U E g, g x, y f x y dx hece, y f x y g x, y dx f E E U EU y f y dy A Useful Tool: Iterated Expectatios E E U EU y f y dy x, y y f g x, y dx f ydy f g xy, fxydx, dy,, g xy f xydxdy EU 7
A Useful Tool: Iterated Expectatios Now, for the Law of total variace: Var U E U E U E E U E EU E VarU E U E E U E VarU E E U E EU E Var U Var E U A Useful Tool: Iterated Expectatios - Example Example: Suppose that a rectagle is costructed by first choosig its legth, ad the choosig its width. Its legth is selected form a expoetial distributio with mea = / = 5. Oce the legth has bee chose its width,, is selected from a uiform distributio form to half its legth. Fid the mea ad variace of the area of the rectagle A =. 8
A Useful Tool: Iterated Expectatios - Example Solutio: 5 x f x e x 5 for f yx if y x x f x y f x f y x, x 5 x 5 5 e = 5x e if y x, x x We could compute the mea ad variace of A = from the joit desity f(x,y) A Useful Tool: Iterated Expectatios - Example, E A E xyf x y dxdy x x x 5x 5 5 5 xy e dydx ye dydx E A E x y f x, y dxdy x x x 5x 5 5 5 x y e dydx xy e dydx ad Var A E A E A x x 9
A Useful Tool: Iterated Expectatios - Example x yx x 5 x y 5 5 5 y 3 x 3 5 5 x 5 5 8 3 3 5 E A ye dydx e dx x e dx x e dx 3 3 3 5 5 5 5.5 A Useful Tool: Iterated Expectatios - Example x 3 y x x 5 x y 5 E A 5xy e dydx 5xe dx 3 y Thus Var A E A E A 5 5 5 5 x 5 5 x e dx x e dx 4 x 4 5 3 8 6 5 5 5 5 5 4 5 5 4 6 6 5 4! 4 5 5 5.5 93.75 3
A Useful Tool: Iterated Expectatios - Example Now, let s use the previous theorem. That is, ad E A E E E Var A Var E Var Var E Now E E 4 4 Var Var 48 ad 4 This is because give, has a uiform distributio from to / A Useful Tool: Iterated Expectatios - Example Thus E A E E E E E 4 4 4 d where momet for the expoetial dist' with Note Thus 5 k! k for the expoetial dist k 5 EA 4 4.5 5 The same aswer as previously calculated!! Ad o itegratio eeded! 3
A Useful Tool: Iterated Expectatios - Example E Now 4 4 48 Also Var A Var ad Var E Var Var E 4 4 4! 5 E Var E 48 48 4 48 4 Var E Var 4 4 Var 4 4 E E 4 4 5 A Useful Tool: Iterated Expectatios - Example Thus Var A Var Var E Var 4 4 Var 4!! 5 5 5 4!! 4 4 4 5 5 4 4 5 4 4 E Var Var E 4 5 5 5 4 5 4 4 5 5 93.75 4 4 8 The same aswer as previously calculated!! Ad o itegratio eeded! 3
The Multivariate MGF Defiitio: Multivariate MGF Let,,, q be q radom variables with a joit desity fuctio give by f(x, x,, x q ). The multivariate MGF is m ( t) E [exp( t' )] where t = (t, t,, t q ) ad = (,,, q ). If,,, are idepedet radom variables, the m ( t) i m ( t i i ) The MGF of a Multivariate Normal Defiitio: MGF for the Multivariate Normal Let,,, q be ormal radom variables. The multivariate ormal MGF is m ( t) E[exp( t' )] exp( t'μ t' t) where t= (t, t,, t q ), = (,,, q ) ad μ= (μ, μ,, μ q ). 33
Review: The Trasformatio Method Theorem Let deote a radom variable with probability desity fuctio f(x) ad U = h(). Assume that h(x) is either strictly icreasig (or decreasig) the the probability desity of U is: dh u dx g u f h ( u) f x du du ( ) The Trasformatio Method (may variables) Theorem Let x, x,, x deote radom variables with joit probability desity fuctio f(x, x,, x ) Let u = h (x, x,, x ). u = h (x, x,, x ). u = h (x, x,, x ). defie a ivertible trasformatio from the x s to the u s 34
The Trasformatio Method (may variables) The the joit probability desity fuctio of u, u,, u is give by:,,,, g u u f x x where J f x,, x J d x,, x d u,, u Jacobia of the trasformatio d x,, x d u,, u dx dx du du det dx dx du du Example: Distributio of x+y ad x-y Suppose that x, x are idepedet with desity fuctios f (x ) ad f (x ) Fid the distributio of u = x + x ad u = x - x Solutio: Solvig for x ad x, we get the iverse trasformatio: J x u u x The Jacobia of the trasformatio d x, x d u, u dx du det dx du u u dx du dx du 35
Example: Distributio of x+y ad x-y J d x, x d u, u det The joit desity of x, x is f(x, x ) = f (x ) f (x ) Hece the joit desity of u ad u is:,, g u u f x x J u u u u f f Example: Distributio of x+y ad x-y u u u u g u, u f f From We ca determie the distributio of u = x + x, g u g u u du u u u u f f du u u u u dv v u v du put the, 36
Example: Distributio of x+y ad x-y Hece u u u u g u f f du f v f u v dv This is called the covolutio of the two desities f ad f. Example (): Covolutio formula -The Gamma distributio Let ad be two idepedet radom variables such that ad have a expoetial distributio with parameter We will use the covolutio formula to fid the distributio of U = +. (We already kow the distributio of U: gamma.) g U ( u) u f U e ( u -u y) f ( y) dy dy ue -u This is the gamma distributio whe α=. u e -(u-y) e -y dy 37
Example (): The ex-gaussia distributio Let ad be two idepedet radom variables such that:. has a expoetial distributio with parameter. has a ormal (Gaussia) distributio with mea ad stadard deviatio. We will use the covolutio formula to fid the distributio of U = +. (This distributio is used i psychology as a model for respose time to perform a task.) Example (): The ex-gaussia distributio Now f x x e x x f y e The desity of U = + is: g u f v f u v dv x uv v e e dv 38
Example (): The ex-gaussia distributio or u v v g u e dv e u v v dv e v u v u v dv u v u v e e dv Example (): The ex-gaussia distributio or u v u v e e dv u u v u vu e e dv u u v u vu e e dv e u u P V 39
Example (): The ex-gaussia distributio Where V has a Normal distributio with mea ad variace. That is, g u e V u u u Where Φ(z) is the cdf of the stadard Normal distributio.9 The ex-gaussia distributio.6 g(u).3 3 4
Distributio of Quadratic Forms We will preset differet theorems whe the RVs are ormal variables: Theorem 7.. If y ~ N(μ y, Σ y ), the z = Ay ~N(Aμ y, A Σ y A ), where A is a matrix of costats. Theorem 7.. Let the vector y ~N(, I ). The y y ~. Theorem 7.3. Let the vector y ~N(, σ I ) ad M be a symmetric idempotet matrix of rak m. The, y My/σ ~ tr(m) Proof: Sice M is symmetric it ca be diagoalized with a orthogoal matrix Q. That is, Q MQ = Λ. (Q Q=I) Sice M is idempotet all these roots are either zero or oe. Thus, I Q' MQ Note: dim(i) = rak(m) (the umber of o-zero roots is the rak of the matrix). Also, sice Σ i λ i =tr(i), => dim(i)=tr(m). Let v=q y. E(v) = Q E(y)= Var(v) = E[vv ]=E[Q yyq] =Q E(σ I )Q = σ Q I Q = σ I =>v ~ N(,σ I ) The, tr( M ) tr( M ) y' My v' Q' MQv I vi v' v v i i i Thus, y My/σ is the sum of tr(m) N(,) squared variables. It follows a tr(m) 4
Theorem 7.4. Let the vector y ~ N(μ y, Σ y ). The, (y -μ y ) Σ - y (y -μ y ) ~ Proof: Recall that there exists a o-sigular matrix A such that AA = Σ y. Let v = A - (y -μ y ) (a liear combiatio of ormal variables) => v ~ N(, I ) => v Σ - y v ~ (usig Theorem 7.3, where =tr(σ - y ). Theorem 7.5 Let the vector y ~ N(, I) ad M be a matrix. The, the characteristic fuctio of y My is I-itM -/ Proof: E [ e y' My y ity' My ] () / y e ity' My y' y/ dx () y'( IitM) y/ This is the ormal desity with Σ - =(I-itM), except for the determiat I-itM -/, which should be i the deomiator. e / y e dx. Theorem 7.6 Let the vector y ~ N(, I), M be a idempotet matrix of rak m, let L be a idempotet matrix of rak s, ad suppose ML =. The y My ad y Ly are idepedetly distributed variables. Proof: By Theorem 7.3 both quadratic forms distributed variables. We oly eed to prove idepedece. From Theorem 7.5, we have y' My y' Ly E [ e y E [ e y ity' My ity' Ly ] I itm ] I itl / / The forms will be idepedetly distributed if That is, φ y (M+L)y = φ y My φ y Ly ity'( ML) y / / / y '( ML) y Ey[ e ] I it( M L) I itm I itl Sice ML = M L, the above result will be true oly whe ML=. 4