Lecture 6: Poisson regression

Transcription

1 Lecture 6: Poisson regression Claudia Czado TU München c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

2 Overview Introduction EDA for Poisson regression Estimation and testing in Poisson regression Residuals in Poisson regression c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

3 Introduction Want regression models for count response data. See Cameron and Trivedi (1998) and Winkelmann (1997) for books on regression models for count data. Example: Canadian car insurance Y i = Number of claims in within a group i of insured with t i insurance years x i = covariate of group i c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

4 Poisson regression Model: Y i P oisson(t i µ(x i, β)) i = 1,..., n ind., i.e. P (Y i = y i ) = e t iµ(x i,β)(t i µ(x i,β)) y i y i! µ(x i, β) := e x i tβ 0 E(Y i ) = t i µ(x i, β) = V ar(y i ) t i gives the time length in which events occur, t i known. Likelihood: l(y, β) = n P (Y i = y i ) = e n t i µ(x i,β) n (t i µ(x i, β)) y i y i! c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

5 For the Poisson model EDA for Poisson regression Y i P oisson(t i e x i tβ ) ind. ln(µ i ) = ln(t i ) + x t i β ( ) If we only have a single obs. for each x i we can estimate ( ) by ln(y i ) = ln(t i ) + x t i β Therefore ln(y i ) ln(t i ) needs to be linear in x i if the Poisson model is valid. If Y i = 0 holds we need to consider ln(y i + c) for c small. Similar as for the binary regression we can investigate main and interaction effects. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

6 If there are several obs. denoted by Y i1,..., Y ini with t i1,..., t ini for each design vector x i we can plot x i versus ln 1 n i n i j=1 Y ij ln 1 n i n i j=1 t ij In this case we can check whether E(Y i ) = V ar(y i ) is valid for Poisson regression: plot ˆµ 0 i = 1 n i n i j=1 Y ij versus s 0 i = 1 n i 1 and we expect this plot to be centered around the 45 o - line. n i j=1 (Y ij ˆµ 0 i ) 2 c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

7 Example: Canadian Automobile Insurance Claims Source: Bailey and Simon (1960) Description: The data give the Canadian automobile insurance experience for policy years 1956 and 1957 as of June 30, The data includes virtually every insurance company operating in Canada and was collated by the Statistical Agency (Canadian Underwriters Association - Statistical Department) acting under instructions from the Superintendent of Insurance. The data given here is for private passenger automobile liability for non-farmers for all of Canada excluding Saskatchewan. The variable Merit measures the number of years since the last claim on the policy. The variable Class is a collation of age, sex, use and marital status. The variables Insured and Premium are two measures of the risk exposure of the insurance companies. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

8 Variable Description: Merit 3 licensed and accident free 3 years 2 licensed and accident free 2 years 1 licensed and accident free 1 year 0 all others Class 1 pleasure, no male operator < 25 2 pleasure, non-principal male operator < 25 3 business use 4 unmarried owner and principal operator< 25 5 married owner and principal operator < 25 c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

9 Insured Earned car years Premium Earned premium in 1000 s (adjusted to what the premium would have been had all cars been written at 01 rates) Claims Cost Number of claims Total cost of the claim in 1000 s of dollars c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

10 > cancar.data Merit Class Insured Premium Claims Cost Data c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

11 Exploratory Data Analysis: Merit Rating Class empirical log mean empirical log mean Merit Rating Class and Merit Class Merit and Class empirical log mean empirical log mean Merit Class Linear Effect for Merit Rating, but quadratic effect for Class if considered as metric variables. Interaction effects present, especially when Merit=2 and 3. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

12 Likelihood analysis in Poisson regression loglikelihood: ln l(y, β) = n t i µ(x i, β) + n y i ln(t i µ(x i, β)) + const. ind. of β s j (β) := ln l(y,β) β j = n t i e x i tβ x ij + n y i e x i t β e x i t β x ij s(β) = = n (y i t i e x tβ i )x ij = 0 j = 1,..., p s 1(β). s p (β) = X t (Y µ) = 0 µ = t i e x i t β. t i e x i t β score equations MLE ˆβ solves s(ˆβ) = 0. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

13 Fisher information in Poisson regression 2 ln l(y,β) β r β s = ( n ) β r (y i t i e x tβ i )x is = n i rs := E t i e x i tβ x is x ir ( ) 2 ln l(y,β) β r β s = n t i e x i tβ x is x ir I(β) := (i rs ) r,s=1,...,p = X t D(β)X where D(β) := diag(..., t i e x i tβ,...) I(ˆβ) 1 = estimated asymptotic covariance matrix of ˆβ c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

14 Poisson regression as GLM P (Y i = y i ) = exp{ t i µ(x i, β) + y i ln(t i µ(x i, β)) ln(y i!)} If µ(x i, β) = e x i t β = exp{y i ln(t i µ(x i, β)) t } {{ } i µ(x i, β) ln(y } {{ } i!)} } {{ } θ i b(θ i )=e θ i c(y i,φ) a(φ) = 1 φ = 1 µ i = E(Y i ) = b (θ i ) = e θ i = t i µ(x i, β) = t i e x i tβ = e x i t β+ln(t i ) η i = x t i β = ln(µ i) ln(t i ) = θ i ln(t i ) η i = g(µ i ) g(µ i ) = ln(µ i ) ln(t i ) ln(t i ) known offset extended link function If t i = 1 i θ i = η i g(µ i ) = ln(µ i ) canonical link. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

15 Properties of the MLE Since ˆβ solves s(ˆβ) = X t (Y ˆµ) = 0 X t Y = X t ˆµ where ˆµ = (..., t i e x i t ˆβ,...) t i) If X contains intercept n Y i = t }{{} 1 n Y = 1 nt ˆµ = n 1st column of X ˆµ i ii) If { 1 i X = (D 1, D 2,..., D k ) D ij = th obs. has level j 0 otherwise j = 1,..., k, i = 1,..., n, k = p X t Y = (N 1,..., N k ) where N j = number of obs. with level i Fitted marginal totals (X t ˆµ) are the same as the marginal totals (X t Y). c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

16 iii) If two factors A with I levels and B with J levels are considered with interaction model Y A B then the observed totals of each cell is the same as the fitted totals. If we only have a single observation for each cell, then we have a saturated model. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

17 Deviance analysis in Poisson regression - After the EDA identifies important covariates one can use the partial deviance test to test for significance of individual or groups of covariates - Goodnes of fit can be checked with the residual deviance test - The deviance is given by D = 2 n {y i log ( yi ˆµ i ) (y i ˆµ i )} a χ 2 n p, where ˆµ i := t i e x i t ˆβ. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

18 Remark: The second term of D will be zero if an intercept is used since n y i = n ˆµ i. Pearson χ 2 P statistic for Poisson GLM s is given by χ 2 P = n (y i ˆµ i ) 2 ˆµ i a χ 2 n p Note: a assumes n fixed, but t i µ i i. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

19 Poisson Regression: Class and Merit as factors > f.main_claims ~ offset(log(insured)) + Merit + Class > r.main_glm(f.main,family=poisson) > summary(r.main,cor=f) Call: glm(formula = Claims ~ offset(log(insured)) + Merit + Class, family = poisson) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Value Std. Error t value (Intercept) Merit Merit Merit Class Class Class Class Null Deviance: on 19 degrees of freedom Residual Deviance: 580 on 12 degrees of freedom Residual Deviance too large, not a good fit. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

20 With all interaction terms > f.inter_claims ~ offset(log(insured)) + Merit * Class > r.inter_glm(f.inter,family=poisson) > summary(r.inter,cor=f) Call: glm(formula = Claims ~ offset(log(insured)) + Merit * Class, family = poisson) c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

21 Coefficients: Value Std. Error t value (Intercept) Merit Merit Merit Class Class Class Class Merit1Class Merit2Class Merit3Class Merit1Class Merit2Class Merit3Class Merit1Class Merit2Class Merit3Class Merit1Class Merit2Class Merit3Class Null Deviance: on 19 degrees of freedom Residual Deviance: 0 on 0 degrees of freedom This model is the saturated model since only one observation per cell. Interaction effects when Merit=3 or 2 are very significant. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

22 With some interaction terms > f.inter1_claims ~ offset(log(insured)) + Merit + Class + Class:(Merit == 3) + Class:(Merit == 2) > r.inter1_glm(f.inter1,family=poisson) > summary(r.inter1,cor=f) Call: glm(formula = Claims ~ offset(log(insured)) + Merit + Class + Class:(Merit == 3) + Class:(Merit == 2), family = poisson) c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

23 Coefficients: (4 not defined because of singularities) Value Std. Error t value (Intercept) Merit Merit Merit Class Class Class Class Merit == 3 NA NA NA Merit == 2 NA NA NA Merit == 3Class Merit == 3Class Merit == 3Class Merit == 3Class Merit == 3Class5 NA NA NA Merit == 2Class Merit == 2Class Merit == 2Class Merit == 2Class Merit == 2Class5 NA NA NA Null Deviance: on 19 degrees of freedom Residual Deviance: 5.5 on 4 degrees of freedom The Residual Deviance is now only 5.5 on 4 df with a p-value of.24, reasonable good fit. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

24 Class and Merit as metric variables > Merit.m_as.numeric(Merit) > Class.m_as.numeric(Class) > f.main.m_claims ~ offset(log(insured))+merit.m + poly(class.m,2) > r.main.m_glm(f.main.m,family=poisson) > summary(r.main.m,cor=f) Call: glm(formula = Claims ~ offset(log(insured)) + Merit.m + poly(class.m, 2), family = poisson) Coefficients: Value Std. Error t value (Intercept) Merit.m poly(class.m, 2) poly(class.m, 2) Null Deviance: on 19 degrees of freedom Residual Deviance: 1000 on 16 degrees of freedom c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

25 With interaction > f.inter.m_claims ~ offset(log(insured)) + Merit.m*poly(Class.m,2) > r.inter.m_glm(f.inter.m,family=poisson) > summary(r.inter.m) Call: glm(formula = Claims ~ offset(log(insured)) + Merit.m * poly(class.m, 2), family = poisson) Deviance Residuals: Min 1Q Median 3Q Max c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

26 Coefficients: Value Std. Error (Intercept) Merit.m poly(class.m, 2) poly(class.m, 2) Merit.mpoly(Class.m, 2) Merit.mpoly(Class.m, 2) t value (Intercept) Merit.m poly(class.m, 2)1 4.1 poly(class.m, 2) Merit.mpoly(Class.m, 2)1 9.0 Merit.mpoly(Class.m, 2)2-4.6 Null Deviance: on 19 degrees of freedom Residual Deviance: 580 on 14 degrees of freedom Residual deviance is still too high the models with Merit and Class as factors fit better. Note this is not a saturated model. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

27 Residuals in Poisson regression Pearson residuals: r P i := y i ˆµ i ˆµi Deviance residuals: r D i := sign(y i.ˆµ i )[y i log ( yi ˆµ i ) (y i ˆµ i )] 1/2 If t i const resid(r.object,type= deviance ) resid(r.object,type= pearson ) in Splus i, it is better to consider the standardized response Y i = Y i /t i r P i := y i/t i ˆµ i /t i ˆµi /t i = 1 t i ti r P i = 1 ti r P i standardized Pearson residuals Interpretation: ˆµ i = t i e x i t ˆβ is the estimated number of events in 0 to t i units ˆµ i t i estimated number of events in 0 to 1 units. = the c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

28 Claims~offset+Merit+Class Residual Analysis Claims~offset+Merit+Class+Class:(Merit==3) +Class:(Merit==2) stand. Pearson residual stand. Pearson residual Index Index Claims~offset+Merit.m+poly(Class.m,2) Claims~offset+Merit.m*poly(Class.m,2) stand. Pearson residual stand. Pearson residual Index Index Residuals for model Claims~offset(log(Insured))+Merit+Class+Class:(Merit==3)+Class:(Merit==2) look best. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

29 Interpretation The estimated number of claims per 1000 earned car years for the main effects only and the interaction model are given by: > fit.main_tapply((fitted(r.main)/insured)*1000,list(merit,class),mean) > fit.main > fit.inter1_tapply((fitted(r.inter1)/insured)*1000,list(merit,class),mean) > fit.inter > fit.obs <- tapply((claims/insured) * 1000,list(Merit, Class), mean) > fit.obs c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

30 Observed Claims Expected Claims (Main Effects) observed no of claims Class Merit estimated no of claims Class Merit Expected Claims (With Interaction) estimated claims Class Merit c (Claudia Czado, TU Munich) ZFS/IMS Göttingen

31 References Bailey, R. and L. Simon (1960). Two studies in automobile insurance ratemaking. ASTIN Bulletin V.1, N.4, Cameron, A. and P. Trivedi (1998). Regression analysis of count data. Cambridge University Press. Winkelmann, R. (1997). Econometric analysis of count data (Second, revised and enlarged ed.). Berlin: Springer-Verlag. c (Claudia Czado, TU Munich) ZFS/IMS Göttingen