Logistic regression residuals

Transcription

1 Logistic regression residuals There are (at least) three different definitions of residuals The raw residuals The Pearson residuals y i n i ˆπ i y i n i ˆπ i ni ˆπ i (1 ˆπ i ) The deviance residuals sign(y i n i ˆπ i ) 2y i log y i n i ˆπ i + 2(n i y i ) log n i y i n i (1 ˆπ i ). Slide 1/9 Niels Richard Hansen Regression May 18, 2010

2 Deviance and χ 2 -statistic The Pearson χ 2 -statistic is X 2 = i (o i e i ) 2 e i where o i is observed and e i is expected. For logistic regression X 2 = N i=1 (y i n i ˆπ i ) n i ˆπ i (1 ˆπ i ). By Taylor expansion of s s log(s/t) around t the deviance is approximately equal to X 2. Slide 2/9 Niels Richard Hansen Regression May 18, 2010

3 Hosmer-Lemeshow When the distribution of the deviance/χ 2 -statistic can not be approximated by a χ 2 -distribution one can fall back on the Hosmer-Lemeshow goodness-of-fit test. Divide the explanatory variables into g roughly equal sized groups (sizes m k, k = 1,..., g) according to fitted value of π. Approximate the probability in group k = 1,..., g as π k = the average of the estimated probabilities in group k. Compute the ordinary χ 2 -statistic for the g groups using the expected value e = m k π k. The Hosmer-Lemeshow statistic is usually compared with a χ 2 -distribution with g 2 degrees of freedom, which is justified by simulation experience. Slide 3/9 Niels Richard Hansen Regression May 18, 2010

4 Model selection AIC, BIC If the model is correct and Y is a new observation independent of the observation used for estimation the approximation 2E(l(ˆπ 1, Y ) l(ˆπ 2, Y )) 2l(ˆπ 1, y) 2l(ˆπ 2, y) + 2(p q) = D 1 + 2p (D 2 + 2q) can be justified with p and q the dimensions of parameter spaces for the two models. AIC = 2l(ˆπ) + 2p where D is the deviance small value of AIC is best. BIC = 2l(ˆπ) + p log(number of observations) Slide 4/9 Niels Richard Hansen Regression May 18, 2010

5 Odds and interpretations... again What does it mean that the odds-ratio between two models is 2? We gamble, and the odds are 50 to 1 against it means that if we win, we win 50 kroner per 1 kroner at stake. If the odds-ratio is 2, the odds are 25 to 1 against for the second model we win 25 kroner per 1 kroner at stake. Is the odds-ratio a sensible comparison of risks? In epidemiology?? Well... we read of directly from the odds-ratio how the pay-off changes. Slide 5/9 Niels Richard Hansen Regression May 18, 2010

6 Relative risks Alternatively, we can measure the difference between two models by the relative risk RR = π A π B If odds are 1 to 1 and 3 to 1, the odds-ratio is 3 but the relative risk is 2. For small probabilities odd-ratio = π A(1 π B ) π B (1 π A ) π A π B = relative risk Slide 6/9 Niels Richard Hansen Regression May 18, 2010

7 Multinomial models If the response variable takes values in {1,..., J} with J > 2 we have a multinomial model. It is not a response distribution that fits directly into the framework of univariate, exponential families... Two possibilities: The multinomial distribution arises as a conditional distribution in a Poisson model, which can be used with a log-link function. We can generalize the logistic regression model; fix a reference category and assume that In the latter case log π j π 1 = (1, x T )β j, j = 2,..., J. π j = Slide 7/9 Niels Richard Hansen Regression May 18, 2010 e (1,xT )β j 1 + J j=2 e(1,xt )β j.

8 Feedforward neural network The multinomial regression model can be perceived as a simple, feedforward neural network with no hidden layers: The input layer consisting of one node per explanatory variable. The output layer consisting of one node per class label. The network computes the probabilities p(y j x) by weighing the signal x i by w ij along the link from the i th input node to the j th output node to obtain the linear predictor η j = i j w ij x i and the probabilities are given by the softmax function for j = 1,..., J. Slide 8/9 Niels Richard Hansen Regression May 18, 2010 p(y j x) = e η j J k=1 eη k

9 Cumulative log-odds models If the J categories are ordered 1 < 2 <... < J, the cumulative logit model reads log π π j π j π J = (1, x T )β j, j = 1,..., J 1 The β ij -parameter has the interpretation of a log-odds ratio but for the cumulative probability. The proportional odds model is the model assumption log π π j = β 0j + x T β, j = 1,..., J 1 π j π J We assume that all explanatory variables affect the cumulative odds for the J 1 first categories in the same way parallel on a log-scale. This is most easy to interpret if the ordinal response variable is given by cut points for a latent response variable. Slide 9/9 Niels Richard Hansen Regression May 18, 2010