1.9.4 Hierarchical Models and Mixture Distributions

Transcription

1 36 CHAPTER 1 ELEMENTS OF PROBABILITY DISTRIBUTION THEORY 194 Hierarchical Models and Mixture Distributions When a random variable depends on another random variable, then we may try to model it in a hierarchical way This leads to a mixture distribution We explain this situation by the following example Example 17 Denote by N a number of eggs laid by an insect Suppose that each egg survives, independently of other eggs, with the same probability p Let X denote the number of survivors Then X N n is a binomial rv with n trials and probability of success p, ie, X (N n Bin(n, p Assume that the number of eggs the insect lays follows a Poisson distribution with parameter λ, ie, N Poisson(λ That is, we can write the following hierarchical model, X N Bin(N, p N Poisson(λ Then, the marginal distribution of the random variable X can be written as P(X x P(X x, N n n0 P(X x N np(n n n0 nx {( n x p x (1 p n x }{ e λ λ n since X N n is Binomial, N is Poisson and the conditional probability is zero if n < x Multiplying this expression by λx and then simplifying gives, λ x n! P(X x (n x!x! px (1 p n xe λ λ n λ x n! λ x nx ( (λpx e λ n x (1 pλ x! (n x! nx (λpx e λ e (1 pλ x! (λpx e λp x! n! },

2 19 TWO-DIMENSIONAL RANDOM VARIABLES 37 That is, X Poisson(λp Then, we can easily get the expected number of surviving eggs as E(X λp Note, that in the above example pλ p E(N E(pN E ( E(X N since X N Bin(N, p The following theorem generalizes this fact Theorem 115 If X 1 and X are any two random variables, then provided that the expectations exist E X 1 E ( E(X 1 X, Proof We will show the result for a continuous bi-variate rv Proof for the discrete case goes along the same lines, just replace integrals and pdf with sums and pmf For any two continuous random variables X 1 and X with a joint pdf f X (x 1, x we can write E X 1 ( x 1 f X (x 1, x dx 1 dx x 1 f(x 1 x dx 1 f X (x dx E(X 1 x f X (x dx E ( E(X 1 X In fact, we have the following, more general result E[g(X 1 ] E[E(g(X 1 X ] Another very useful result relates the variance of a hierarchical distribution with variances of conditional distributions It is given in the following theorem Theorem 116 If X 1 and X are any two random variables, then provided that the expectations exist var X 1 E[var(X 1 X ] + var[e(x 1 X ],

3 38 CHAPTER 1 ELEMENTS OF PROBABILITY DISTRIBUTION THEORY Proof We have var(x 1 X E(X1 X [E(X 1 X ] and hence E[var(X 1 X ] E[E(X1 X ] E{[E(X 1 X ] } E(X1 E{[E(X 1 X ] } Also, var[e(x 1 X ] E{[E(X 1 X ] } {E[E(X 1 X ]} E{[E(X 1 X ] } [E(X 1 ] Therefore, E[var(X 1 X ] + var[e(x 1 X ] E(X1 [E(X 1] var(x 1 Example 18 A new drug is tested on n patients in phase III of a clinical trial Assume that a response can either be efficacy (success or no efficacy (failure It is expected that each patient s chance to react positively to the drug is different and the probability of success follows a Beta(α, β distribution Then, the response for each patient could be modeled as a hierarchical Bernoulli rv, that is, X i P i Bern(P i, i 1,,, n, P i Beta(α, β We are interested in the expected total number of efficacious responses and also in its variance Let us denote by Y the total number of successes Then Y X i We will use Theorems 115 and 116 to calculate its expectation and variance E Y E(X i i1 i1 E ( E(X i P i i1 E(P i i1 i1 α α + β nα α + β Now, we will calculate the variance of Y Random variables X i are independent, hence ( var(y var X i var(x i i1 i1

4 19 TWO-DIMENSIONAL RANDOM VARIABLES 39 By Theorem 116 we have var(x i var ( E(X i P i + E ( var(x i P i, i 1,,, n The first term in the above expression is var ( E(X i P i var(p i αβ (α + β (α + β + 1, since P i Beta(α, β The second term of the expression for the variance of X i is E ( var(x i P i E ( P i (1 P i Here we used the fact that Γ(α + β Γ(αΓ(β Γ(α + β Γ(αΓ(β Γ(α + β Γ(αΓ(β p i (1 p i p α 1 i (1 p i β 1 dp i p α i (1 p i β dp i Γ(α + 1Γ(β + 1 Γ(α + β + αβ (α + β(α + β + 1 Γ(α + β + Γ(α + 1Γ(β + 1 pα i (1 p i β is a pdf of a Beta(α + 1, β + 1 random variable and we also used the property of Gamma function that Γ(z zγ(z 1 Hence, var(x i var ( E(X i P i + E ( var(x i P i αβ (α + β (α + β αβ (α + β(α + β + 1 αβ (α + β Finally, as the variance of X i does not depend on i, we get var(y nαβ (α + β

5 40 CHAPTER 1 ELEMENTS OF PROBABILITY DISTRIBUTION THEORY Exercise 117 The number of spam messages Y sent to a server in a day has Poisson distribution with parameter λ 1 Each spam message independently has probability p 1/3 of not being detected by the spam filter Let X denote the number of spam messages getting through the filter Calculate the expected daily number of spam messages which get into the server Also, calculate the variance of X 195 Covariance and Correlation Covariance and correlation are two measures of the strength of a relationship between two rvs We will use the following notation E X 1 µ X1 E X µ X var(x 1 σ X 1 var(x σ X Also, we assume that σx 1 and σx are finite positive values A simplified notation µ 1, µ, σ1, σ will be used when it is clear which rvs we refer to Definition 119 The covariance of X 1 and X is defined by cov(x 1, X E[(X 1 µ X1 (X µ X ] (116 Definition 10 The correlation of X 1 and X is defined by ρ (X1,X corr(x 1, X cov(x 1, X σ X1 σ X (117 Some useful properties of the covariance and correlation are given in the following two theorems Theorem For any random variables X 1 and X, cov(x 1, X E(X 1 X µ X1 µ X

6 19 TWO-DIMENSIONAL RANDOM VARIABLES 41 If random variables X 1 and X are independent then cov(x 1, X 0 Then also ρ (X1,X 0 3 For any random variables X 1 and X and any constants a and b, var(ax 1 + bx a var(x 1 + b var(x + ab cov(x 1, X Exercise 118 Prove Theorem 117 Theorem 118 For any random variables X 1 and X 1 1 ρ (X1,X 1, ρ (X1,X 1 iff there exist numbers a 0 and b such that P(X ax 1 + b 1 If ρ (X1,X 1 then a > 0, and if ρ (X1,X 1 then a < 0 Exercise 119 Prove Theorem 118 Hint: Consider roots (and so the discriminant of the quadratic function of t: g(t E[(X µ X t + (Y µ Y ] 196 Bivariate Normal Distribution Here we use matrix notation A bivariate rv is treated as a random vector ( X1 X The expectation of a bivariate random vector is written as ( ( X1 µ1 µ E X E X X µ

7 4 CHAPTER 1 ELEMENTS OF PROBABILITY DISTRIBUTION THEORY and its variance-covariance matrix is ( ( var(x1 cov(x V 1, X σ 1 ρσ 1 σ cov(x, X 1 var(x ρσ 1 σ σ Then the joint pdf of a normal bi-variate rv X is given by f X (x where x (x 1, x T 1 π det(v exp { 1 (x µt V 1 (x µ The determinant of V is ( σ det V det 1 ρσ 1 σ ρσ 1 σ σ (1 ρ σ1σ Hence, the inverse of V is V 1 1 ( σ ρσ 1 σ det V ρσ 1 σ σ1 1 ( 1 ρ }, (118 σ1 ρσ1 1 σ 1 ρσ1 1 σ Then the exponent in formula (118 can be written as 1 (x µt V 1 (x µ ( 1 (1 ρ (x 1 µ 1, x µ ( 1 (x1 µ 1 (1 ρ σ 1 σ1 ρσ1 1 σ 1 ρσ1 1 σ ρ (x 1 µ 1 (x µ σ 1 σ + (x µ σ So, the joint pdf of the two-dimensional normal rv X is 1 f X (x πσ 1 σ (1 ρ { ( 1 (x1 µ 1 exp (1 ρ σ 1 Note that when ρ 0 it simplifies to f X (x 1 πσ 1 σ exp ( x1 µ x µ ρ (x 1 µ 1 (x µ + (x } µ σ 1 σ σ { 1 ( (x1 µ 1 σ 1 + (x } µ, σ which can be written as a product of the marginal distributions of X 1 and X Hence, if X (X 1, X T has a bivariate normal distribution and ρ 0 then the variables X 1 and X are independent

8 19 TWO-DIMENSIONAL RANDOM VARIABLES 43 Figure 1: Bivariate Normal pdf