Generalized Linear Model. Badr Missaoui

Transcription

1 Badr Missaoui

2 Logistic Regression Outline Generalized linear models Deviance Logistic regression.

3 All models we have seen so far deal with continuous outcome variables with no restriction on their expectations, and (most) have assumed that mean and variance are unrelated (i.e. variance is constant). Many outcomes of interest do not satisfy this. Examples : binary outcomes, Poisson count outcomes. A Generalized Linear Model (GLM) is a model with two ingredients : a link function and a variance function. The link relates the means of the observations to predictors : linearization The variance function relates the means to the variances.

4 The data involve 462 males between the ages of 15 and 64. The outcome Y is the presence (Y = 1) or absence Y = 0 of heart disease Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) e-06 *** sbp tobacco ** ldl ** adiposity famhistpresent e-05 *** typea ** obesity alcohol age ***

5 Motivation Classical linear model Y = Xβ + ε where ε N(0, σ 2 ). That means, Y N(Xβ, σ 2 ) In the GLM, we specify that Y P(Xβ)

6 We write the GLM as and E(Y i ) = µ i η i = g(µ i ) = X i β where the function g called a link function which belongs to an exponential family.

7 The exponential family density are specifying two components, the canonical parameter θ and the dispersion parameter φ. Let Y = (Y i ) i=1...n be a sequence of random variables. Y i has an exponential density if ( ) yi θ i b(θ i ) f Yi (y i ; θ i, φ) = exp + c(y i, φ) a i (φ) where the functions b, c are specific to each distribution and a i (φ) = φ/w i.

8 Law Law µ σ 2 B(m, p) p y (1 p) m y. ( ) m m y k=0 δ {k} mp mp(1 p) P(µ) µ y e µ. m k=0 } 1 k! δ k µ µ N (µ, σ 2 ) exp { (y µ)2.dy µ σ 2 2σ 2 } IG(µ, λ) λ exp { λ(y µ)2 dy 2µy. µ µ 3 /λ 2πy 3

9 We write l(y; θ, φ) = log f (y; θ, φ) for the log-likelihood function of Y. Using the facts that ( ) l E θ ( ) l Var θ = 0 = E ( 2 ) l θ 2 We have and E(y) = b (θ) Var(y) = b (θ)a(φ)

10 Gaussian case ] 1 f (y; θ, φ) = [ σ 2π exp (y µ)2 2σ 2 ( yµ µ 2 /2 = exp σ 2 1 ( )) y 2 2 σ 2 + log(2πσ2 ) We can write( θ = µ, φ = σ 2, a(φ) ) = φ, b(θ) = θ 2 /2 and c(y, φ) = 1 y log(2πσ 2 ) σ 2 Binomial case ( ) n f (y; θ, φ) = µ y (1 µ) n y y = exp ( y log We can write θ = log µ c(y, φ) = log ( ) n y 1 µ ( ) µ + n log(1 µ) + log 1 µ, b(θ) = n log(1 µ) and ( )) n y

11 Recall that in ordinary linear models, the MLE of β satisfies ˆβ = (X T X) 1 X T Y if X has full rank. In GLM, the MLE ˆβ does not exist in closed form and can be approximately estimated via iterative weighted least squares.

12 For n observations, the log-likelihood function is n L(β) = l(y i ; θ, φ) Computing i=1 l i = l i θ i µ i η i 1 = x ij β j θ i µ i η i β j g (µ i ) 1 b (θ i ) y i µ i φ/w i The likelihood equations are L i n 1 µ i = x ij β j g (µ i ) 2 (y i µ i ) = 0 j = 1,.., p Var(y i ) η i Put and i=1 { } W = diag g (µ i ) 2 Var(y i ) i=1,...,n { µ η = diag µi η i } i=1,...,n

13 These likelihood equations are X T W 1 µ (y µ) = 0 η These equations are non-linear in β and require an iterative method (e.g Newton-Raphson). The Fisher s Information matrix is and in general term I = X T W 1 X ( 2 ) L(β) [I] jk = E = β j β k n i=1 x ij x jk Var(y i ) ( µi η i ) 2

14 Let ˆµ 0 = Y be the initial estimate. Then, set ˆη 0 = g(ˆµ 0 ), and form the adjusted variable Z 0 = ˆη 0 + (Y ˆµ 0 ) η µ µ=ˆµ 0 Calculate ˆβ 1 by the least squares regression of Z 0 on X, that means So, Set ˆβ 1 = argmin β (Z 0 Xβ) T W 1 0 (Z 0 Xβ) ˆβ 1 = (X T W 1 0 X) 1 X T W 1 0 Z 0 ˆη 1 = X ˆ β 1, ˆµ 1 = g 1 (ˆη 1 ) Repeat until changes in ˆβ m are sufficiently small.

15 Estimation In theory, ˆβ m ˆβ as m, but in practice, the algorithm may fail to converge. Under some conditions, ˆβ N(β, I 1 (β)) In practice, the asymptotic covariance matrix of ˆβ is estimated by φ(x T Wm 1 X) 1 where W m is the weight matrix from the m th iteration. If φ is unknown, it is estimated by ˆφ = 1 n p n i=1 w i (y i ˆµ) 2 V (ˆµ) where V (ˆµ i ) = var(y i )/a(φ) = w i var(y i )/φ

16 Confidence interval [ ] 1 1 CI α (β i ) = ˆβ j u 1 α/2 n ˆσ βj ; ˆβ j + u 1 α/2 n ˆσ βj where u 1 α/2 is the 1 α/2 quantile of N(0, 1) and [ ] 1 I( ˆβ). ˆσ βj = 1 n To test the hypothesis if φ is unknown jj H 0 : β j = 0 against H 1 : β j 0 ˆβ j N(0, 1) φ(x T Wm 1 X) 1 (j, j) ˆβ j t n p ˆφ(X T Wm 1 X) 1 (j, j)

17 Goodness-of-Fit H 0 : the true model is M versus H 1 : the true is M sat The likelihood ratio test for this hypothesis is called the deviance. For any submodel M, dev(m) = 2(ˆl sat ˆl M ) Under H 0, dev(m) χ 2 p sat p.

18 Goodness-of-Fit The scaled deviance for GLM is D(y, ˆµ) = 2 [l(ˆµ sat, φ; y) l(ˆµ, φ; y)] = n { 2w i yi (θ(ˆµ sat i ) θ(ˆµ i )) b(ˆµ sat } i ) + b(ˆµ i /φ = i=1 n D (y i ; ˆµ i )/φ i=1 = D (y; ˆµ)/φ

19 Tests We use the deviance to compare two models having p 1 and p 2 parameters respectively, where p 1 < p 2. Let ˆµ 1 and ˆµ 2 denote the corresponding MLEs. If φ is unknown, D(y, ˆµ 1 ) D(y, ˆµ 2 ) χ 2 p 2 p 1 D (y, ˆµ 1 ) D (y, ˆµ 2 ) (p 2 p 1 ) ˆφ F 1 α,p2 p 1,n p 2

20 Goodness-of-Fit The deviance residuals for a given model are d i = sign(y i ˆµ i ) D (y i ; ˆµ i ) A poorly fitting point will make a large contribution to the deviance, so d i will be large.

21 Diagnostics The Pearson residuals are defined by r i = y i ˆµ i (1 hii )V (ˆµ) where h ii is the ith diagonal element of The deviance residuals are H = X(X T Wm 1 X) 1 X T Wm 1 ˆε i = sign(y i ˆµ i ) D (y i ; ˆµ i ) 1 h ii

22 Diagnostics The Anscombe residuals is defined as a transformation of the Pearson residual r A i = t(y i ) t(ˆµ i ) t (ˆµ i ) φv (ˆµ i )(1 h ii ) The aim in introducing the function t is to make the residuals as Gaussian as possible. We consider t(x) = x 0 V (µ) 1/3 dµ

23 Diagnostics Influential points using the Cook s distance C i = 1 p ( ˆβ (i) ˆβ) T X T W m X( ˆβ (i) ˆβ) r 2 i h ii p(1 h ii ) 2 The outliers points : if h ii > 2p/n or h ii > 3p/n, then we consider that ith point is an outlier.

24 Model Selection Model selection can be done using the AIC and BIC. Forward, Backward and stepwise approach can be used.

25 Logistic regression Logistic regression is a generalization of regression that is used when the outcome Y is binary 0, 1. As example, we assume that P(Y i = 1 X i ) = eβ 0+β 1 X i 1 + e β 0+β 1 X i Note that E(Y i X i ) = P(Y i = 1 X i )

26 Logistic regression Define the logit function ( ) z logit(z) = log 1 z We can write where π i = P(Y i = 1 X i ) logit(π i ) = β 0 + β 1 X i The extension to several covariates is logit(π i ) = β 0 + p β j x ij i=1

27 How do we estimate the parameters? Can be fit using maximum likelihood. The likelihood function is L(β) = n f (y i X i ; β) = L(β) = i=1 n i=1 The estimator ˆβ has to be found numerically. π y i i (1 π i ) 1 y i

28 Usually, we use the reweighted least squares First set a starting values of β (0) Compute ex i β (k) ˆπ i = 1 + e X i β(k) Define weighted matrix W whose i th diagonal is ˆπ i (1 ˆπ i ) Define the adjusted response vector Z = Xβ (k) + W 1 (Y ˆπ) Take ˆβ (k+1) = (X T WX) 1 X T WZ which is the weighted linear regression of Z on X

29 Model selection and diagnostics Diagnostics : the Pearson χ 2 Y i ˆπ i ˆπi (1 ˆπ i ) The deviance residuals [ sign(y i ˆπ i ) 2 Y i log ( Yi ) ( )] 1 Yi + (1 Y i ) log ˆπ i 1 ˆπ i

30 To fit this model, we use the glm command. Call: glm(formula = chd ~., family = binomial, data = SAheart) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) e-06 *** row.names sbp tobacco ** ldl ** adiposity famhistpresent e-05 *** typea ** obesity alcohol age *** --- Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: on 461 degrees of freedom Residual deviance: on 451 degrees of freedom AIC: Number of Fisher Scoring iterations: 5

31 To fit this model, we use the glm command. Start: AIC= chd ~ row.names + sbp + tobacco + ldl + adiposity + famhist + typea + obesity + alcohol + age Df Deviance AIC - alcohol adiposity row.names sbp <none> obesity ldl tobacco typea age famhist etc... Step: AIC= chd ~ tobacco + ldl + famhist + typea + age Df Deviance AIC <none> ldl typea tobacco famhist age

32 Suppose Y i Binomial(n i, π i ) We can fit the logistic model as before Pearson residuals r i = logit(π i ) = X i β Y i n i ˆπ i ni ˆπ i (1 ˆπ i ) Deviation residuals d i = sign(y i Ŷi) 2 [ Y i log ( Yi ) ( )] ni Y i + (n i Y i ) log ˆµ i n i ˆµ i

33 Goodness-of-Fit test The Pearson test and deviance χ 2 = i D = i r 2 i d 2 i both have a χ 2 n p distribution if the model is correct.

34 To fit this model, we use the glm command. Call: glm(formula = cbind(y, n - y) ~ x, family = binomial) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) e-16 *** x e-16 *** --- Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: on 7 degrees of freedom Residual deviance: on 6 degrees of freedom AIC: Number of Fisher Scoring iterations: 4

35 To test the correctness of the model > pvalue = 1-pchisq(out$dev,out$df.residual) > print(pvalue) [1] > r=resid(out,type="deviance") > p=out$linear.predictors > plot(p,r,pch=19,xlab="linear predictor", ylab="deviance residuals") > print(sum(r^2)) [1] > cooks.distance(out) Note that the residuals give back the deviance test, and the p-value is large indicating no evidence of a lack of fit.