Simple Linear Regression

Transcription

1 Simple Linear Regression Example: Body density Aim: Measure body density (weight per unit volume of the body) (Body density indicates the fat content of the human body.) Problem: Body density is difficult to measure directly. Research suggests that skinfold thickness can accurately predict body density. Skinfold thickness is measures by pinching a fold of skin between calipers.. Body Density ( kg m ) Skinfold Thickness (mm) Questions: Are body density and skinfold thickness related? How accurately can we predict body density from skinfold thickness? Regression: predict response variable for fixed value of explanatory variable describe linear relationship in data by regression line fitted regression line is affected by chance variation in observed data Statistical inference: accounts for chance variation in data Simple Linear Regression, Feb 7, - -

2 Population Regression Line Simple linear regression studies the relationship between a response variable Y and a single explanatory variable X. We expect that different values of X will produce different mean responses of Y. For given X = x, we consider the subpopulation with X = x: this subpopulation has mean µ Y X=x = E(Y X = x) (cond. mean of Y given X = x) and variance σ Y X=x = var(y X = x) (cond. variance of Y given X = x) Linear regression model with constant variance: E(Y X = x) = µ Y X=x = a + b x (population regression line) var(y X = x) = σ Y X=x = σ The population regression line connects the conditional means of the response variable for fixed values of the explanatory variable. This population regression line tells how the mean response of Y varies with X. The variance (and standard deviation) does not depend on x. Simple Linear Regression, Feb 7, - -

3 Conditional Mean Sample (x, y ),..., (x n, y n ) Sampling probability f(x, y) fix x = x rescale by f X (x ) f(x, y) Conditional probability f(y x ) = f XY (x, y) f X (x ) E(Y X = x ) = y f Y X (y x ) dy conditional mean Simple Linear Regression, Feb 7, - -

4 The Linear Regression Model Simple linear regression Y i = a + b x i + ε i, i =,..., n where Y i x i ε i response (also dependent variable) predictor (also independent variable) error Assumptions: Predictor x i is deterministic (fixed values, not random). Errors have zero mean, E(ε i ) =. Variation about mean does not depend on x i, i.e. var(ε i ) = σ. Errors ε i are independent. Often we additionally assume: The errors are normally distributed, ε i iid N (, σ ). For fixed x the response Y is normally distributed with Y N (a + b x, σ ). Simple Linear Regression, Feb 7, - -

5 Data: (Y, x ),..., (Y n, x n ) Least Squares Estimation Aim: Find straight line which fits data best: Ŷ i = a + b x i fitted values for coefficients a and b a - intercept b - slope Least Squares Approach: Minimize squared distance between observed Y i and fitted Ŷi: L(a, b) = n (Y i Ŷi) = n (Y i a b x i ) Set partial derivatives to zero (normal equations): L a = L b = n n (Y i a b x i ) = (Y i a b x i ) x i = Solution: Least squares estimators where â = Ȳ S XY S XX X ˆb = S XY S XX S XY = n (Y i Ȳ )(x i x) (sum of squares) S XX = n (x i x) Simple Linear Regression, Feb 7, - -

6 Least squares predictor Ŷ Ŷ i = â + ˆb x i Residuals ˆε i : ˆε i = Y i Ŷi = Y i â ˆb x i Residual sum of squares (SS Residual ) Least Squares Estimation SS Residual = n ˆε i = n (Y i Ŷi) Estimation of σ ˆσ = n n (Y i Ŷi) = n SS Residual Regression standard error s e = ˆσ = SS Residual /(n ) Variation accounting: SS Total = n (Y i Ȳ ) total variation SS Model = n (Ŷi Ȳ ) variation explained by linear model SS Residual = n (Y i Ŷi) remaining variation Simple Linear Regression, Feb 7, - -

7 Least Squares Estimation Example: Body density Scatter plot with least squares regression line:. Body Density ( kg m ) Skinfold Thickness (mm) Calculation of least squares estimates: x ȳ S XX S XY S Y Y SS Residual ˆb = S XY S XX =.7. =. â = ȳ ˆb x = =.7 ˆσ = RSS n =.87 9 =. s e = ˆσ =. =.9 Simple Linear Regression, Feb 7, - 7 -

8 Least Squares Estimation Example: Returns on Treasury bills and inflation Using STATA:. infile ID BODYD SKINT using bodydens.txt, clear (9 observations read). regress BODYD SKINT Source SS df MS Number of obs = F(, 9) =.89 Model Prob > F =. Residual R-squared = Adj R-squared =.77 Total Root MSE = BODYD Coef. Std. Err. t P> t [9% Conf. Interval] SKINT _cons twoway (lfitci BODYD SKINT, range(.)) (scatter BODYD SKINT), xtitle(skin thickn > ess) ytitle(body density) scheme(scolor) legend(off) Body density Skin thickness Simple Linear Regression, Feb 7, - 8 -