# Generalized Linear Model Theory

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Appendix B Generalized Linear Model Theory We describe the generalized linear model as formulated by Nelder and Wedderburn (1972), and discuss estimation of the parameters and tests of hypotheses. B.1 The Model Let y 1,..., y n denote n independent observations on a response. We treat y i as a realization of a random variable Y i. In the general linear model we assume that Y i has a normal distribution with mean µ i and variance σ 2 Y i N(µ i, σ 2 ), and we further assume that the expected value µ i is a linear function of p predictors that take values x i = (x i1,..., x ip ) for the i-th case, so that µ i = x iβ, where β is a vector of unknown parameters. We will generalize this in two steps, dealing with the stochastic and systematic components of the model. G. Rodríguez. Revised November 2001

2 2 APPENDIX B. GENERALIZED LINEAR MODEL THEORY B.1.1 The Exponential Family We will assume that the observations come from a distribution in the exponential family with probability density function f(y i ) = exp{ y iθ i b(θ i ) a i (φ) + c(y i, φ)}. (B.1) Here θ i and φ are parameters and a i (φ), b(θ i ) and c(y i, φ) are known functions. In all models considered in these notes the function a i (φ) has the form a i (φ) = φ/p i, where p i is a known prior weight, usually 1. The parameters θ i and φ are essentially location and scale parameters. It can be shown that if Y i has a distribution in the exponential family then it has mean and variance E(Y i ) = µ i = b (θ i ) (B.2) var(y i ) = σ 2 i = b (θ i )a i (φ), (B.3) where b (θ i ) and b (θ i ) are the first and second derivatives of b(θ i ). When a i (φ) = φ/p i the variance has the simpler form var(y i ) = σ 2 i = φb (θ i )/p i. The exponential family just defined includes as special cases the normal, binomial, Poisson, exponential, gamma and inverse Gaussian distributions. Example: The normal distribution has density f(y i ) = 1 exp{ 1 (y i µ i ) 2 2πσ 2 2 σ 2 }. Expanding the square in the exponent we get (y i µ i ) 2 = y 2 i + µ2 i 2y iµ i, so the coefficient of y i is µ i /σ 2. This result identifies θ i as µ i and φ as σ 2, with a i (φ) = φ. Now write f(y i ) = exp{ y iµ i 1 2 µ2 i σ 2 This shows that b(θ i ) = 1 2 θ2 i and variance: y2 i 2σ log(2πσ2 )}. (recall that θ i = µ i ). Let us check the mean E(Y i ) = b (θ i ) = θ i = µ i, var(y i ) = b (θ i )a i (φ) = σ 2.

3 B.1. THE MODEL 3 Try to generalize this result to the case where Y i has a normal distribution with mean µ i and variance σ 2 /n i for known constants n i, as would be the case if the Y i represented sample means. Example: In Problem Set 1 you will show that the exponential distribution with density f(y i ) = λ i exp{ λ i y i } belongs to the exponential family. In Sections B.4 and B.5 we verify that the binomial and Poisson distributions also belong to this family. B.1.2 The Link Function The second element of the generalization is that instead of modeling the mean, as before, we will introduce a one-to-one continuous differentiable transformation g(µ i ) and focus on η i = g(µ i ). (B.4) The function g(µ i ) will be called the link function. Examples of link functions include the identity, log, reciprocal, logit and probit. We further assume that the transformed mean follows a linear model, so that η i = x iβ. (B.5) The quantity η i is called the linear predictor. Note that the model for η i is pleasantly simple. Since the link function is one-to-one we can invert it to obtain µ i = g 1 (x iβ). The model for µ i is usually more complicated than the model for η i. Note that we do not transform the response y i, but rather its expected value µ i. A model where log y i is linear on x i, for example, is not the same as a generalized linear model where log µ i is linear on x i. Example: The standard linear model we have studied so far can be described as a generalized linear model with normal errors and identity link, so that η i = µ i. It also happens that µ i, and therefore η i, is the same as θ i, the parameter in the exponential family density.

5 B.3. TESTS OF HYPOTHESES 5 Finally, we obtain an improved estimate of β regressing the working dependent variable z i on the predictors x i using the weights w i, i.e. we calculate the weighted least-squares estimate ˆβ = (X WX) 1 X Wz, (B.8) where X is the model matrix, W is a diagonal matrix of weights with entries w i given by (B.7) and z is a response vector with entries z i given by (B.6). The procedure is repeated until successive estimates change by less than a specified small amount. McCullagh and Nelder (1989) prove that this algorithm is equivalent to Fisher scoring and leads to maximum likelihood estimates. These authors consider the case of general a i (φ) and include φ in their expression for the iterative weight. In other words, they use wi = φw i, where w i is the weight used here. The proportionality factor φ cancels out when you calculate the weighted least-squares estimates using (B.8), so the estimator is exactly the same. I prefer to show φ explicitly rather than include it in W. Example: For normal data with identity link η i = µ i, so the derivative is dη i /dµ i = 1 and the working dependent variable is y i itself. Since in addition b (θ i ) = 1 and p i = 1, the weights are constant and no iteration is required. In Sections B.4 and B.5 we derive the working dependent variable and the iterative weights required for binomial data with link logit and for Poisson data with link log. In both cases iteration will usually be necessary. Starting values may be obtained by applying the link to the data, i.e. we take ˆµ i = y i and ˆη i = g( ˆµ i ). Sometimes this requires a few adjustments, for example to avoid taking the log of zero, and we will discuss these at the appropriate time. B.3 Tests of Hypotheses We consider Wald tests and likelihood ratio tests, introducing the deviance statistic. B.3.1 Wald Tests The Wald test follows immediately from the fact that the information matrix for generalized linear models is given by I(β) = X WX/φ, (B.9)

6 6 APPENDIX B. GENERALIZED LINEAR MODEL THEORY so the large sample distribution of the maximum likelihood estimator ˆβ is multivariate normal ˆβ N p (β, (X WX) 1 φ). (B.10) with mean β and variance-covariance matrix (X WX) 1 φ. Tests for subsets of β are based on the corresponding marginal normal distributions. Example: In the case of normal errors with identity link we have W = I (where I denotes the identity matrix), φ = σ 2, and the exact distribution of ˆβ is multivariate normal with mean β and variance-covariance matrix (X X) 1 σ 2. B.3.2 Likelihood Ratio Tests and The Deviance We will show how the likelihood ratio criterion for comparing any two nested models, say ω 1 ω 2, can be constructed in terms of a statistic called the deviance and an unknown scale parameter φ. Consider first comparing a model of interest ω with a saturated model Ω that provides a separate parameter for each observation. Let ˆµ i denote the fitted values under ω and let ˆθ i denote the corresponding estimates of the canonical parameters. Similarly, let µ O = y i and θ i denote the corresponding estimates under Ω. The likelihood ratio criterion to compare these two models in the exponential family has the form 2 log λ = 2 n i=1 y i ( θ i ˆθ i ) b( θ i ) + b( ˆθ i ). a i (φ) Assume as usual that a i (φ) = φ/p i for known prior weights p i. Then we can write the likelihood-ratio criterion as follows: 2 log λ = D(y, ˆµ). (B.11) φ The numerator of this expression does not depend on unknown parameters and is called the deviance: n D(y, ˆµ) = 2 p i [y i ( θ i ˆθ i ) b( θ i ) + b( ˆθ i )]. i=1 (B.12) The likelihood ratio criterion 2 log L is the deviance divided by the scale parameter φ, and is called the scaled deviance.

7 B.3. TESTS OF HYPOTHESES 7 Example: Recall that for the normal distribution we had θ i = µ i, b(θ i ) = 1 2 θ2 i, and a i (φ) = σ 2, so the prior weights are p i = 1. Thus, the deviance is D(y, ˆµ) = 2 {y i (y i ˆµ i ) 1 2 y2 i ˆµ i 2 } = 2 { 1 2 y2 i y i ˆµ i ˆµ i 2 } = (y i ˆµ i ) 2 our good old friend, the residual sum of squares. Let us now return to the comparison of two nested models ω 1, with p 1 parameters, and ω 2, with p 2 parameters, such that ω 1 ω 2 and p 2 > p1. The log of the ratio of maximized likelihoods under the two models can be written as a difference of deviances, since the maximized log-likelihood under the saturated model cancels out. Thus, we have 2 log λ = D(ω 1) D(ω 2 ) φ (B.13) The scale parameter φ is either known or estimated using the larger model ω 2. Large sample theory tells us that the asymptotic distribution of this criterion under the usual regularity conditions is χ 2 ν with ν = p 2 p 1 degrees of freedom. Example: In the linear model with normal errors we estimate the unknown scale parameter φ using the residual sum of squares of the larger model, so the criterion becomes 2 log λ = RSS(ω 1) RSS(ω 2 ) RSS(ω 2 )/(n p 2 ). In large samples the approximate distribution of this criterion is χ 2 ν with ν = p 2 p 1 degrees of freedom. Under normality, however, we have an exact result: dividing the criterion by p 2 p 1 we obtain an F with p 2 p 1 and n p 2 degrees of freedom. Note that as n the degrees of freedom in the denominator approach and the F converges to (p 2 p 1 )χ 2, so the asymptotic and exact criteria become equivalent. In Sections B.4 and B.5 we will construct likelihood ratio tests for binomial and Poisson data. In those cases φ = 1 (unless one allows overdispersion and estimates φ, but that s another story) and the deviance is the same as the scaled deviance. All our tests will be based on asymptotic χ 2 statistics.

8 8 APPENDIX B. GENERALIZED LINEAR MODEL THEORY B.4 Binomial Errors and Link Logit We apply the theory of generalized linear models to the case of binary data, and in particular to logistic regression models. B.4.1 The Binomial Distribution First we verify that the binomial distribution B(n i, π i ) belongs to the exponential family of Nelder and Wedderburn (1972). The binomial probability distribution function (p.d.f.) is f i (y i ) = ( ni y i ) π y i i (1 π i) n i y i. (B.14) Taking logs we find that log f i (y i ) = y i log(π i ) + (n i y i ) log(1 π i ) + log Collecting terms on y i we can write π i log f i (y i ) = y i log( ) + n i log(1 π i ) + log 1 π i ( ni y i ( ni y i ). ). This expression has the general exponential form log f i (y i ) = y iθ i b(θ i ) a i (φ) + c(y i, φ) with the following equivalences: Looking first at the coefficient of y i we note that the canonical parameter is the logit of π i Solving for π i we see that π i = π i θ i = log( ). (B.15) 1 π i eθ i 1 + e θ i, so 1 π i = e θ i. If we rewrite the second term in the p.d.f. as a function of θ i, so log(1 π i ) = log(1 + e θ i ), we can identify the cumulant function b(θ i ) as b(θ i ) = n i log(1 + e θ i ).

9 B.4. BINOMIAL ERRORS AND LINK LOGIT 9 The remaining term in the p.d.f. is a function of y i but not π i, leading to ) c(y i, φ) = log ( ni Note finally that we may set a i (φ) = φ and φ = 1. Let us verify the mean and variance. Differentiating b(θ i ) with respect to θ i we find that µ i = b (θ i ) = n i e θ i 1 + e θ i = n iπ i, in agreement with what we knew from elementary statistics. Differentiating again using the quotient rule, we find that e θ i v i = a i (φ)b (θ i ) = n i (1 + e θ i ) 2 = n iπ i (1 π i ), again in agreement with what we knew before. In this development I have worked with the binomial count y i, which takes values 0(1)n i. McCullagh and Nelder (1989) work with the proportion p i = y i /n i, which takes values 0(1/n i )1. This explains the differences between my results and their Table 2.1. B.4.2 Fisher Scoring in Logistic Regression Let us now find the working dependent variable and the iterative weight used in the Fisher scoring algorithm for estimating the parameters in logistic regression, where we model y i. η i = logit(π i ). (B.16) It will be convenient to write the link function in terms of the mean µ i, as: π i µ i η i = log( ) = log( ), 1 π i n i µ i which can also be written as η i = log(µ i ) log(n i µ i ). Differentiating with respect to µ i we find that dη i dµ i = 1 µ i + 1 n i µ i = n i µ i (n i µ i ) = 1 n i π i (1 π i ). The working dependent variable, which in general is z i = η i + (y i µ i ) dη i dµ i,

10 10 APPENDIX B. GENERALIZED LINEAR MODEL THEORY turns out to be z i = η i + The iterative weight turns out to be y i n i π i n i π i (1 π i ). w i = [ 1/ b (θ i )( dη ] i ) 2, dµ i = 1 n i π i (1 π i ) [n iπ i (1 π i )] 2, (B.17) and simplifies to w i = n i π i (1 π i ). (B.18) Note that the weight is inversely proportional to the variance of the working dependent variable. The results here agree exactly with the results in Chapter 4 of McCullagh and Nelder (1989). Exercise: Obtain analogous results for Probit analysis, where one models η i = Φ 1 (µ i ), where Φ() is the standard normal cdf. Hint: To calculate the derivative of the link function find dµ i /dη i and take reciprocals. B.4.3 The Binomial Deviance Finally, let us figure out the binomial deviance. Let ˆµ i denote the m.l.e. of µ i under the model of interest, and let µ i = y i denote the m.l.e. under the saturated model. From first principles, D = 2 [y i log( y i n i ) + (n i y i ) log( n i y i n i ) y i log( ˆµ i n i ) (n i y i ) log( n i ˆµ i n i )]. Note that all terms involving log(n i ) cancel out. Collecting terms on y i and on n i y i we find that D = 2 [y i log( y i ˆµ i ) + (n i y i ) log( n i y i n i µ i )]. (B.19) Alternatively, you may obtain this result from the general form of the deviance given in Section B.3.

11 B.5. POISSON ERRORS AND LINK LOG 11 Note that the binomial deviance has the form D = 2 o i log( o i e i ), where o i denotes observed, e i denotes expected (under the model of interest) and the sum is over both successes and failures for each i (i.e. we have a contribution from y i and one from n i y i ). For grouped data the deviance has an asymptotic chi-squared distribution as n i for all i, and can be used as a goodness of fit test. More generally, the difference in deviances between nested models (i.e. the log of the likelihood ratio test criterion) has an asymptotic chi-squared distribution as the number of groups k or the size of each group n i, provided the number of parameters stays fixed. As a general rule of thumb due to Cochrane (1950), the asymptotic chisquared distribution provides a reasonable approximation when all expected frequencies (both ˆµ i and n i ˆµ i ) under the larger model exceed one, and at least 80% exceed five. B.5 Poisson Errors and Link Log Let us now apply the general theory to the Poisson case, with emphasis on the log link function. B.5.1 The Poisson Distribution A Poisson random variable has probability distribution function f i (y i ) = e µ i µ y i i y i! (B.20) for y i = 0, 1, 2,.... The moments are E(Y i ) = var(y i ) = µ i. Let us verify that this distribution belongs to the exponential family as defined by Nelder and Wedderburn (1972). Taking logs we find log f i (y i ) = y i log(µ i ) µ i log(y i!). Looking at the coefficient of y i we see immediately that the canonical parameter is θ i = log(µ i ), (B.21)

12 12 APPENDIX B. GENERALIZED LINEAR MODEL THEORY and therefore that the canonical link is the log. Solving for µ i we obtain the inverse link µ i = e θ i, and we see that we can write the second term in the p.d.f. as b(θ i ) = e θ i. The last remaining term is a function of y i only, so we identify c(y i, φ) = log(y i!). Finally, note that we can take a i (φ) = φ and φ = 1, just as we did in the binomial case. Let us verify the mean and variance. Differentiating the cumulant function b(θ i ) we have µ i = b (θ i ) = e θ i = µ i, and differentiating again we have v i = a i (φ)b (θ i ) = e θ i = µ i. Note that the mean equals the variance. B.5.2 Fisher Scoring in Log-linear Models We now consider the Fisher scoring algorithm for Poisson regression models with canonical link, where we model η i = log(µ i ). (B.22) The derivative of the link is easily seen to be dη i dµ i = 1 µ i. Thus, the working dependent variable has the form The iterative weight is [ w i = 1/ b (θ i )( dη ] i ) 2 dµ [ ] i 1 = 1/ µ i µ 2, i z i = η i + y i µ i µ i. (B.23)

13 B.5. POISSON ERRORS AND LINK LOG 13 and simplifies to w i = µ i. (B.24) Note again that the weight is inversely proportional to the variance of the working dependent variable. B.5.3 The Poisson Deviance Let ˆµ i denote the m.l.e. of µ i under the model of interest and let µ i = y i denote the m.l.e. under the saturated model. From first principles, the deviance is D = 2 [y i log(y i ) y i log(y i!) y i log( ˆµ i ) + ˆµ i + log(y i!)]. Note that the terms on y i! cancel out. Collecting terms on y i we have D = 2 [y i log( y i ˆµ i ) (y i ˆµ i )]. (B.25) The similarity of the Poisson and Binomial deviances should not go unnoticed. Note that the first term in the Poisson deviance has the form D = 2 o i log( o i e i ), which is identical to the Binomial deviance. The second term is usually zero. To see this point, note that for a canonical link the score is log L β = X (y µ), and setting this to zero leads to the estimating equations X y = X ˆµ. In words, maximum likelihood estimation for Poisson log-linear models and more generally for any generalized linear model with canonical link requires equating certain functions of the m.l.e. s (namely X ˆµ) to the same functions of the data (namely X y). If the model has a constant, one column of X will consist of ones and therefore one of the estimating equations will be yi = ˆµ i or (yi ˆµ i ) = 0,

14 14 APPENDIX B. GENERALIZED LINEAR MODEL THEORY so the last term in the Poisson deviance is zero. This result is the basis of an alternative algorithm for computing the m.l.e. s known as iterative proportional fitting, see Bishop et al. (1975) for a description. The Poisson deviance has an asymptotic chi-squared distribution as n with the number of parameters p remaining fixed, and can be used as a goodness of fit test. Differences between Poisson deviances for nested models (i.e. the log of the likelihood ratio test criterion) have asymptotic chi-squared distributions under the usual regularity conditions.

### SAS Software to Fit the Generalized Linear Model

SAS Software to Fit the Generalized Linear Model Gordon Johnston, SAS Institute Inc., Cary, NC Abstract In recent years, the class of generalized linear models has gained popularity as a statistical modeling

: Table of Contents... 1 Overview of Model... 1 Dispersion... 2 Parameterization... 3 Sigma-Restricted Model... 3 Overparameterized Model... 4 Reference Coding... 4 Model Summary (Summary Tab)... 5 Summary

### Poisson Models for Count Data

Chapter 4 Poisson Models for Count Data In this chapter we study log-linear models for count data under the assumption of a Poisson error structure. These models have many applications, not only to the

### Generalized Linear Models. Today: definition of GLM, maximum likelihood estimation. Involves choice of a link function (systematic component)

Generalized Linear Models Last time: definition of exponential family, derivation of mean and variance (memorize) Today: definition of GLM, maximum likelihood estimation Include predictors x i through

### Logit Models for Binary Data

Chapter 3 Logit Models for Binary Data We now turn our attention to regression models for dichotomous data, including logistic regression and probit analysis. These models are appropriate when the response

### Logistic Regression (1/24/13)

STA63/CBB540: Statistical methods in computational biology Logistic Regression (/24/3) Lecturer: Barbara Engelhardt Scribe: Dinesh Manandhar Introduction Logistic regression is model for regression used

### i=1 In practice, the natural logarithm of the likelihood function, called the log-likelihood function and denoted by

Statistics 580 Maximum Likelihood Estimation Introduction Let y (y 1, y 2,..., y n be a vector of iid, random variables from one of a family of distributions on R n and indexed by a p-dimensional parameter

### Statistiek (WISB361)

Statistiek (WISB361) Final exam June 29, 2015 Schrijf uw naam op elk in te leveren vel. Schrijf ook uw studentnummer op blad 1. The maximum number of points is 100. Points distribution: 23 20 20 20 17

### Sufficient Statistics and Exponential Family. 1 Statistics and Sufficient Statistics. Math 541: Statistical Theory II. Lecturer: Songfeng Zheng

Math 541: Statistical Theory II Lecturer: Songfeng Zheng Sufficient Statistics and Exponential Family 1 Statistics and Sufficient Statistics Suppose we have a random sample X 1,, X n taken from a distribution

### L10: Probability, statistics, and estimation theory

L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian

### Lecture 6: Poisson regression

Lecture 6: Poisson regression Claudia Czado TU München c (Claudia Czado, TU Munich) ZFS/IMS Göttingen 2004 0 Overview Introduction EDA for Poisson regression Estimation and testing in Poisson regression

### Lecture 8: Gamma regression

Lecture 8: Gamma regression Claudia Czado TU München c (Claudia Czado, TU Munich) ZFS/IMS Göttingen 2004 0 Overview Models with constant coefficient of variation Gamma regression: estimation and testing

### Parametric Models. dh(t) dt > 0 (1)

Parametric Models: The Intuition Parametric Models As we saw early, a central component of duration analysis is the hazard rate. The hazard rate is the probability of experiencing an event at time t i

### Models for Count Data With Overdispersion

Models for Count Data With Overdispersion Germán Rodríguez November 6, 2013 Abstract This addendum to the WWS 509 notes covers extra-poisson variation and the negative binomial model, with brief appearances

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### Statistics - Written Examination MEC Students - BOVISA

Statistics - Written Examination MEC Students - BOVISA Prof.ssa A. Guglielmi 26.0.2 All rights reserved. Legal action will be taken against infringement. Reproduction is prohibited without prior consent.

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Introduction to General and Generalized Linear Models

Introduction to General and Generalized Linear Models General Linear Models - part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby

### 15.1 The Structure of Generalized Linear Models

15 Generalized Linear Models Due originally to Nelder and Wedderburn (1972), generalized linear models are a remarkable synthesis and extension of familiar regression models such as the linear models described

### Linear Models for Continuous Data

Chapter 2 Linear Models for Continuous Data The starting point in our exploration of statistical models in social research will be the classical linear model. Stops along the way include multiple linear

### Computer exercise 4 Poisson Regression

Chalmers-University of Gothenburg Department of Mathematical Sciences Probability, Statistics and Risk MVE300 Computer exercise 4 Poisson Regression When dealing with two or more variables, the functional

### Logistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression

Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max

### Factorial experimental designs and generalized linear models

Statistics & Operations Research Transactions SORT 29 (2) July-December 2005, 249-268 ISSN: 1696-2281 www.idescat.net/sort Statistics & Operations Research c Institut d Estadística de Transactions Catalunya

### NON-LIFE INSURANCE PRICING USING THE GENERALIZED ADDITIVE MODEL, SMOOTHING SPLINES AND L-CURVES

NON-LIFE INSURANCE PRICING USING THE GENERALIZED ADDITIVE MODEL, SMOOTHING SPLINES AND L-CURVES Kivan Kaivanipour A thesis submitted for the degree of Master of Science in Engineering Physics Department

### Local classification and local likelihoods

Local classification and local likelihoods November 18 k-nearest neighbors The idea of local regression can be extended to classification as well The simplest way of doing so is called nearest neighbor

### Joint Probability Distributions and Random Samples (Devore Chapter Five)

Joint Probability Distributions and Random Samples (Devore Chapter Five) 1016-345-01 Probability and Statistics for Engineers Winter 2010-2011 Contents 1 Joint Probability Distributions 1 1.1 Two Discrete

### GLM, insurance pricing & big data: paying attention to convergence issues.

GLM, insurance pricing & big data: paying attention to convergence issues. Michaël NOACK - michael.noack@addactis.com Senior consultant & Manager of ADDACTIS Pricing Copyright 2014 ADDACTIS Worldwide.

### Regression Models for Time Series Analysis

Regression Models for Time Series Analysis Benjamin Kedem 1 and Konstantinos Fokianos 2 1 University of Maryland, College Park, MD 2 University of Cyprus, Nicosia, Cyprus Wiley, New York, 2002 1 Cox (1975).

### The Delta Method and Applications

Chapter 5 The Delta Method and Applications 5.1 Linear approximations of functions In the simplest form of the central limit theorem, Theorem 4.18, we consider a sequence X 1, X,... of independent and

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Change of Continuous Random Variable

Change of Continuous Random Variable All you are responsible for from this lecture is how to implement the Engineer s Way (see page 4) to compute how the probability density function changes when we make

### Probability Calculator

Chapter 95 Introduction Most statisticians have a set of probability tables that they refer to in doing their statistical wor. This procedure provides you with a set of electronic statistical tables that

### ELEC-E8104 Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems

Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems Minimum Mean Square Error (MMSE) MMSE estimation of Gaussian random vectors Linear MMSE estimator for arbitrarily distributed

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Underwriting risk control in non-life insurance via generalized linear models and stochastic programming

Underwriting risk control in non-life insurance via generalized linear models and stochastic programming 1 Introduction Martin Branda 1 Abstract. We focus on rating of non-life insurance contracts. We

### TESTING THE ONE-PART FRACTIONAL RESPONSE MODEL AGAINST AN ALTERNATIVE TWO-PART MODEL

TESTING THE ONE-PART FRACTIONAL RESPONSE MODEL AGAINST AN ALTERNATIVE TWO-PART MODEL HARALD OBERHOFER AND MICHAEL PFAFFERMAYR WORKING PAPER NO. 2011-01 Testing the One-Part Fractional Response Model against

### Multivariate Normal Distribution

Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4-7/21/2011 Slide 1 of 41 Last Time Matrices and vectors Eigenvalues

### Overview Classes. 12-3 Logistic regression (5) 19-3 Building and applying logistic regression (6) 26-3 Generalizations of logistic regression (7)

Overview Classes 12-3 Logistic regression (5) 19-3 Building and applying logistic regression (6) 26-3 Generalizations of logistic regression (7) 2-4 Loglinear models (8) 5-4 15-17 hrs; 5B02 Building and

### Cointegration. Basic Ideas and Key results. Egon Zakrajšek Division of Monetary Affairs Federal Reserve Board

Cointegration Basic Ideas and Key results Egon Zakrajšek Division of Monetary Affairs Federal Reserve Board Summer School in Financial Mathematics Faculty of Mathematics & Physics University of Ljubljana

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Automated Biosurveillance Data from England and Wales, 1991 2011

Article DOI: http://dx.doi.org/10.3201/eid1901.120493 Automated Biosurveillance Data from England and Wales, 1991 2011 Technical Appendix This online appendix provides technical details of statistical

### Household s Life Insurance Demand - a Multivariate Two Part Model

Household s Life Insurance Demand - a Multivariate Two Part Model Edward (Jed) W. Frees School of Business, University of Wisconsin-Madison July 30, 1 / 19 Outline 1 2 3 4 2 / 19 Objective To understand

### Estimation and Inference in Cointegration Models Economics 582

Estimation and Inference in Cointegration Models Economics 582 Eric Zivot May 17, 2012 Tests for Cointegration Let the ( 1) vector Y be (1). Recall, Y is cointegrated with 0 cointegrating vectors if there

### The Probit Link Function in Generalized Linear Models for Data Mining Applications

Journal of Modern Applied Statistical Methods Copyright 2013 JMASM, Inc. May 2013, Vol. 12, No. 1, 164-169 1538 9472/13/\$95.00 The Probit Link Function in Generalized Linear Models for Data Mining Applications

### Regression, least squares

Regression, least squares Joe Felsenstein Department of Genome Sciences and Department of Biology Regression, least squares p.1/24 Fitting a straight line X Two distinct cases: The X values are chosen

### VI. Introduction to Logistic Regression

VI. Introduction to Logistic Regression We turn our attention now to the topic of modeling a categorical outcome as a function of (possibly) several factors. The framework of generalized linear models

### Continuous Random Variables. and Probability Distributions. Continuous Random Variables and Probability Distributions ( ) ( ) Chapter 4 4.

UCLA STAT 11 A Applied Probability & Statistics for Engineers Instructor: Ivo Dinov, Asst. Prof. In Statistics and Neurology Teaching Assistant: Neda Farzinnia, UCLA Statistics University of California,

### GENERALIZED LINEAR MODELS IN VEHICLE INSURANCE

ACTA UNIVERSITATIS AGRICULTURAE ET SILVICULTURAE MENDELIANAE BRUNENSIS Volume 62 41 Number 2, 2014 http://dx.doi.org/10.11118/actaun201462020383 GENERALIZED LINEAR MODELS IN VEHICLE INSURANCE Silvie Kafková

### Institute of Actuaries of India Subject CT3 Probability and Mathematical Statistics

Institute of Actuaries of India Subject CT3 Probability and Mathematical Statistics For 2015 Examinations Aim The aim of the Probability and Mathematical Statistics subject is to provide a grounding in

### Common probability distributionsi Math 217/218 Probability and Statistics Prof. D. Joyce, 2016

Introduction. ommon probability distributionsi Math 7/8 Probability and Statistics Prof. D. Joyce, 06 I summarize here some of the more common distributions used in probability and statistics. Some are

### The Exponential Family

The Exponential Family David M. Blei Columbia University November 3, 2015 Definition A probability density in the exponential family has this form where p.x j / D h.x/ expf > t.x/ a./g; (1) is the natural

### CS540 Machine learning Lecture 14 Mixtures, EM, Non-parametric models

CS540 Machine learning Lecture 14 Mixtures, EM, Non-parametric models Outline Mixture models EM for mixture models K means clustering Conditional mixtures Kernel density estimation Kernel regression GMM

### Parametric Models Part I: Maximum Likelihood and Bayesian Density Estimation

Parametric Models Part I: Maximum Likelihood and Bayesian Density Estimation Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Fall 2015 CS 551, Fall 2015

### Examination 110 Probability and Statistics Examination

Examination 0 Probability and Statistics Examination Sample Examination Questions The Probability and Statistics Examination consists of 5 multiple-choice test questions. The test is a three-hour examination

### GLM I An Introduction to Generalized Linear Models

GLM I An Introduction to Generalized Linear Models CAS Ratemaking and Product Management Seminar March 2009 Presented by: Tanya D. Havlicek, Actuarial Assistant 0 ANTITRUST Notice The Casualty Actuarial

### LOGISTIC REGRESSION. Nitin R Patel. where the dependent variable, y, is binary (for convenience we often code these values as

LOGISTIC REGRESSION Nitin R Patel Logistic regression extends the ideas of multiple linear regression to the situation where the dependent variable, y, is binary (for convenience we often code these values

### Tilted Normal Distribution and Its Survival Properties

Journal of Data Science 10(2012), 225-240 Tilted Normal Distribution and Its Survival Properties Sudhansu S. Maiti 1 and Mithu Dey 2 1 Visva-Bharati University and 2 Asansol Engineering College Abstract:

### TRANSFORMATIONS OF RANDOM VARIABLES

TRANSFORMATIONS OF RANDOM VARIABLES 1. INTRODUCTION 1.1. Definition. We are often interested in the probability distributions or densities of functions of one or more random variables. Suppose we have

### Stat260: Bayesian Modeling and Inference Lecture Date: February 1, Lecture 3

Stat26: Bayesian Modeling and Inference Lecture Date: February 1, 21 Lecture 3 Lecturer: Michael I. Jordan Scribe: Joshua G. Schraiber 1 Decision theory Recall that decision theory provides a quantification

### MATH4427 Notebook 2 Spring 2016. 2 MATH4427 Notebook 2 3. 2.1 Definitions and Examples... 3. 2.2 Performance Measures for Estimators...

MATH4427 Notebook 2 Spring 2016 prepared by Professor Jenny Baglivo c Copyright 2009-2016 by Jenny A. Baglivo. All Rights Reserved. Contents 2 MATH4427 Notebook 2 3 2.1 Definitions and Examples...................................

### Multilevel Modeling of Complex Survey Data

Multilevel Modeling of Complex Survey Data Tihomir Asparouhov 1, Bengt Muthen 2 Muthen & Muthen 1 University of California, Los Angeles 2 Abstract We describe a multivariate, multilevel, pseudo maximum

### 7 Generalized Estimating Equations

Chapter 7 The procedure extends the generalized linear model to allow for analysis of repeated measurements or other correlated observations, such as clustered data. Example. Public health of cials can

### Pearson s Goodness of Fit Statistic as a Score Test Statistic

Pearson s Goodness of Fit Statistic as a Score Test Statistic Gordon K. Smyth Abstract For any generalized linear model, the Pearson goodness of fit statistic is the score test statistic for testing the

### 11. Analysis of Case-control Studies Logistic Regression

Research methods II 113 11. Analysis of Case-control Studies Logistic Regression This chapter builds upon and further develops the concepts and strategies described in Ch.6 of Mother and Child Health:

### Lecture 12: Generalized Linear Models for Binary Data

Lecture 12: Generalized Linear Models for Binary Data Dipankar Bandyopadhyay, Ph.D. BMTRY 711: Analysis of Categorical Data Spring 2011 Division of Biostatistics and Epidemiology Medical University of

### Lecture 18: Logistic Regression Continued

Lecture 18: Logistic Regression Continued Dipankar Bandyopadhyay, Ph.D. BMTRY 711: Analysis of Categorical Data Spring 2011 Division of Biostatistics and Epidemiology Medical University of South Carolina

### Introduction to Predictive Modeling Using GLMs

Introduction to Predictive Modeling Using GLMs Dan Tevet, FCAS, MAAA, Liberty Mutual Insurance Group Anand Khare, FCAS, MAAA, CPCU, Milliman 1 Antitrust Notice The Casualty Actuarial Society is committed

### Penalized regression: Introduction

Penalized regression: Introduction Patrick Breheny August 30 Patrick Breheny BST 764: Applied Statistical Modeling 1/19 Maximum likelihood Much of 20th-century statistics dealt with maximum likelihood

### It is important to bear in mind that one of the first three subscripts is redundant since k = i -j +3.

IDENTIFICATION AND ESTIMATION OF AGE, PERIOD AND COHORT EFFECTS IN THE ANALYSIS OF DISCRETE ARCHIVAL DATA Stephen E. Fienberg, University of Minnesota William M. Mason, University of Michigan 1. INTRODUCTION

### Chapter 1. Probability, Random Variables and Expectations. 1.1 Axiomatic Probability

Chapter 1 Probability, Random Variables and Expectations Note: The primary reference for these notes is Mittelhammer (1999. Other treatments of probability theory include Gallant (1997, Casella & Berger

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct

### Time Series Analysis

Time Series Analysis hm@imm.dtu.dk Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby 1 Outline of the lecture Identification of univariate time series models, cont.:

### Non-Parametric Estimation in Survival Models

Non-Parametric Estimation in Survival Models Germán Rodríguez grodri@princeton.edu Spring, 2001; revised Spring 2005 We now discuss the analysis of survival data without parametric assumptions about the

### CCNY. BME I5100: Biomedical Signal Processing. Linear Discrimination. Lucas C. Parra Biomedical Engineering Department City College of New York

BME I5100: Biomedical Signal Processing Linear Discrimination Lucas C. Parra Biomedical Engineering Department CCNY 1 Schedule Week 1: Introduction Linear, stationary, normal - the stuff biology is not

### Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation

Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation Abstract This article presents an overview of the logistic regression model for dependent variables having two or

### 1 Review of complex numbers

1 Review of complex numbers 1.1 Complex numbers: algebra The set C of complex numbers is formed by adding a square root i of 1 to the set of real numbers: i = 1. Every complex number can be written uniquely

### Lecture 14: GLM Estimation and Logistic Regression

Lecture 14: GLM Estimation and Logistic Regression Dipankar Bandyopadhyay, Ph.D. BMTRY 711: Analysis of Categorical Data Spring 2011 Division of Biostatistics and Epidemiology Medical University of South

### Applied Statistics. J. Blanchet and J. Wadsworth. Institute of Mathematics, Analysis, and Applications EPF Lausanne

Applied Statistics J. Blanchet and J. Wadsworth Institute of Mathematics, Analysis, and Applications EPF Lausanne An MSc Course for Applied Mathematicians, Fall 2012 Outline 1 Model Comparison 2 Model

### MATHEMATICAL METHODS OF STATISTICS

MATHEMATICAL METHODS OF STATISTICS By HARALD CRAMER TROFESSOK IN THE UNIVERSITY OF STOCKHOLM Princeton PRINCETON UNIVERSITY PRESS 1946 TABLE OF CONTENTS. First Part. MATHEMATICAL INTRODUCTION. CHAPTERS

### Calibration of Stock Betas from Skews of Implied Volatilities

Calibration of Stock Betas from Skews of Implied Volatilities Jean-Pierre Fouque University of California Santa Barbara Joint work with Eli Kollman (Ph.D. student at UCSB) Joint Seminar: Department of

### An extension of the factoring likelihood approach for non-monotone missing data

An extension of the factoring likelihood approach for non-monotone missing data Jae Kwang Kim Dong Wan Shin January 14, 2010 ABSTRACT We address the problem of parameter estimation in multivariate distributions

### Lecture 13: Introduction to generalized linear models

Lecture 13: Introduction to generalized linear models 21 November 2007 1 Introduction Recall that we ve looked at linear models, which specify a conditional probability density P(Y X) of the form Y = α

### Logistic Regression (a type of Generalized Linear Model)

Logistic Regression (a type of Generalized Linear Model) 1/36 Today Review of GLMs Logistic Regression 2/36 How do we find patterns in data? We begin with a model of how the world works We use our knowledge

### Maximum Likelihood Estimation

Math 541: Statistical Theory II Lecturer: Songfeng Zheng Maximum Likelihood Estimation 1 Maximum Likelihood Estimation Maximum likelihood is a relatively simple method of constructing an estimator for

### 4. Introduction to Statistics

Statistics for Engineers 4-1 4. Introduction to Statistics Descriptive Statistics Types of data A variate or random variable is a quantity or attribute whose value may vary from one unit of investigation

### Statistics in Retail Finance. Chapter 2: Statistical models of default

Statistics in Retail Finance 1 Overview > We consider how to build statistical models of default, or delinquency, and how such models are traditionally used for credit application scoring and decision

### Linear Classification. Volker Tresp Summer 2015

Linear Classification Volker Tresp Summer 2015 1 Classification Classification is the central task of pattern recognition Sensors supply information about an object: to which class do the object belong

### Bivariate Distributions

Chapter 4 Bivariate Distributions 4.1 Distributions of Two Random Variables In many practical cases it is desirable to take more than one measurement of a random observation: (brief examples) 1. What is

Lecture 16: Generalized Additive Models Regression III: Advanced Methods Bill Jacoby Michigan State University http://polisci.msu.edu/jacoby/icpsr/regress3 Goals of the Lecture Introduce Additive Models

### Life Table Analysis using Weighted Survey Data

Life Table Analysis using Weighted Survey Data James G. Booth and Thomas A. Hirschl June 2005 Abstract Formulas for constructing valid pointwise confidence bands for survival distributions, estimated using

### Inferential Statistics

Inferential Statistics Sampling and the normal distribution Z-scores Confidence levels and intervals Hypothesis testing Commonly used statistical methods Inferential Statistics Descriptive statistics are

### Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur

Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur Module No. #01 Lecture No. #15 Special Distributions-VI Today, I am going to introduce

### Title. Syntax. Menu. Description. stata.com. lrtest Likelihood-ratio test after estimation. modelspec2. lrtest modelspec 1.

Title stata.com lrtest Likelihood-ratio test after estimation Syntax Menu Description Options Remarks and examples Stored results Methods and formulas References Also see Syntax lrtest modelspec 1 [ modelspec2

### Statistics in Retail Finance. Chapter 6: Behavioural models

Statistics in Retail Finance 1 Overview > So far we have focussed mainly on application scorecards. In this chapter we shall look at behavioural models. We shall cover the following topics:- Behavioural

### Chapter 6: Point Estimation. Fall 2011. - Probability & Statistics

STAT355 Chapter 6: Point Estimation Fall 2011 Chapter Fall 2011 6: Point1 Estimat / 18 Chap 6 - Point Estimation 1 6.1 Some general Concepts of Point Estimation Point Estimate Unbiasedness Principle of

### Journal of Statistical Software

JSS Journal of Statistical Software August 26, Volume 16, Issue 8. http://www.jstatsoft.org/ Some Algorithms for the Conditional Mean Vector and Covariance Matrix John F. Monahan North Carolina State University

### PREDICTIVE DISTRIBUTIONS OF OUTSTANDING LIABILITIES IN GENERAL INSURANCE

PREDICTIVE DISTRIBUTIONS OF OUTSTANDING LIABILITIES IN GENERAL INSURANCE BY P.D. ENGLAND AND R.J. VERRALL ABSTRACT This paper extends the methods introduced in England & Verrall (00), and shows how predictive