Introduction to Linear Regression and Correlation Analysis

Transcription

1 Introduction to Linear Regression and Correlation Analsis

2 Goals After this, ou should be able to: Calculate and interpret the simple correlation between two variables Determine whether the correlation is significant Calculate and interpret the simple linear regression equation for a set of data Understand the assumptions behind regression analsis Determine whether a regression model is significant

3 Goals After this, ou should be able to: Calculate and interpret confidence intervals for the regression coefficients Recognize regression analsis applications for purposes of prediction and description Recognize some potential problems if regression analsis is used incorrectl Recognize nonlinear relationships between two variables

4 Scatter Plots and Correlation A scatter plot (or scatter diagram) is used to show the relationship between two variables Correlation analsis is used to measure strength of the association (linear relationship) between two variables Onl concerned with strength of the relationship No causal effect is implied

5 Scatter Plot Eamples Linear relationships Curvilinear relationships

6 Scatter Plot Eamples Strong relationships Weak relationships

7 Scatter Plot Eamples No relationship

8 Correlation Coefficient The population correlation coefficient ρ (rho) measures the strength of the association between the variables The sample correlation coefficient r is an estimate of ρ and is used to measure the strength of the linear relationship in the sample observations

9 Features of ρ and r Unit free Range between -1 and 1 The closer to -1, the stronger the negative linear relationship The closer to 1, the stronger the positive linear relationship The closer to 0, the weaker the linear relationship

10 Eamples of Approimate r Values r = -1 r = -.6 r = 0 r = +.3 r = +1

11 Calculating the Correlation Coefficient Sample correlation coefficient: r = [ ( ( )( ) ) ][ ( ) ] where: or the algebraic equivalent: r = [n( r = Sample correlation coefficient n = Sample size = Value of the independent variable = Value of the dependent variable n ) ( ) ][n( ) ( ) ]

12 Calculation Eample Trunk Diameter Tree Height Σ=713 Σ=14111 Σ=314 Σ=73 Σ=

13 Calculation Eample Tree Height, 70 r = n [n( ) ( ) ][n( ) ( ) ] = 8(314) (73)(31) [8(713) (73) ][8(14111) (31) ] 30 0 = Trunk Diameter, r = relativel strong positive linear association between and

14 Significance Test for Correlation Hpotheses H 0 : ρ = 0 H A : ρ 0 (no correlation) (correlation eists) Test statistic t = r (with n degrees of freedom) 1 r n

15 Eample: Produce Stores Is there evidence of a linear relationship between tree height and trunk diameter at the.05 level of significance? H 0 : ρ = 0 (No correlation) H 1 : ρ 0 (correlation eists) α =.05, df = 8 - = 6 t = r 1 r n = = 4.68

16 Eample: Test Solution r t = 1 r n d.f. = 8- = 6 α/=.05 = = 4.68 α/=.05 Decision: Reject H 0 Conclusion: There is evidence of a linear relationship at the 5% level of significance Reject H 0 Reject H t 0 α/ Do not reject H -t 0 α/

17 Introduction to Regression Analsis Regression analsis is used to: Predict the value of a dependent variable based on the value of at least one independent variable Eplain the impact of changes in an independent variable on the dependent variable Dependent variable: the variable we wish to eplain Independent variable: the variable used to eplain the dependent variable

18 Simple Linear Regression Model Onl one independent variable, Relationship between and is described b a linear function Changes in are assumed to be caused b changes in

19 Tpes of Regression Models Positive Linear Relationship Relationship NOT Linear Negative Linear Relationship No Relationship

20 Population Linear Regression The population regression model: Dependent Variable Population intercept Population Slope Coefficient Independent Variable = β + β ε Random Error term, or residual Linear component Random Error component

21 Linear Regression Assumptions Error values (ε) are statisticall independent Error values are normall distributed for an given value of The probabilit distribution of the errors is normal The probabilit distribution of the errors has constant variance The underling relationship between the variable and the variable is linear

22 Population Linear Regression (continued) Observed Value of for i = β + β ε Predicted Value of for i ε i Random Error for this value Slope = β 1 Intercept = β 0 i

23 Estimated Regression Model The sample regression line provides an estimate of the population regression line Estimated (or predicted) value Estimate of the regression intercept Estimate of the regression slope ŷ = b + b i 0 1 Independent variable The individual random error terms e i have a mean of zero

24 Least Squares Criterion b 0 and b 1 are obtained b finding the values of b 0 and b 1 that minimize the sum of the squared residuals e = ( ŷ) = ( (b 0 + b 1 ))

25 The Least Squares Equation The formulas for b 1 and b 0 are: b 1 ( = ( )( ) ) algebraic equivalent: b = n 1 ( ) n and b0 = b1

26 Interpretation of the Slope and the Intercept b 0 is the estimated average value of when the value of is zero b 1 is the estimated change in the average value of as a result of a oneunit change in

27 Finding the Least Squares Equation The coefficients b 0 and b 1 will usuall be found using computer software Other regression measures will also be computed as part of computer-based regression analsis

28 Simple Linear Regression Eample A real estate agent wishes to eamine the relationship between the selling price of a home and its size (measured in square feet) A random sample of 10 houses is selected Dependent variable () = house price in $1000s Independent variable () = square feet

29 Sample Data for House Price Model House Price in $1000s () Square Feet ()

30 Graphical Presentation House price model: scatter plot and regression line Intercept = House Price ($1000s) Square Feet Slope = house price = (square feet)

31 Interpretation of the Intercept, b 0 house price = (square feet) b 0 is the estimated average value of Y when the value of X is zero (if = 0 is in the range of observed values) Here, no houses had 0 square feet, so b 0 = just indicates that, for houses within the range of sizes observed, $98,48.33 is the portion of the house price not eplained b square feet

32 Interpretation of the Slope Coefficient, b 1 house price = (square feet) b 1 measures the estimated change in the average value of Y as a result of a one-unit change in X Here, b 1 = tells us that the average value of a house increases b.10977($1000) = $109.77, on average, for each additional one square foot of size

33 Least Squares Regression Properties The sum of the residuals from the least squares regression line is 0 ( ) The sum of the squared residuals is a minimum (minimized ( ˆ) ) The simple regression line alwas passes through the mean of the variable and the mean of the variable The least squares coefficients are unbiased estimates of β 0 and β 1 ( ˆ) = 0

34 Eplained and Uneplained Variation Total variation is made up of two parts: SST = SSE + SSR Total sum of Squares Sum of Squares Error Sum of Squares Regression = SST ( ) SSE = ( ŷ) SSR = (ŷ ) where: = Average value of the dependent variable = Observed values of the dependent variable ŷ = Estimated value of for the given value

35 Eplained and Uneplained Variation SST = total sum of squares Measures the variation of the i values around their mean SSE = error sum of squares Variation attributable to factors other than the relationship between and SSR = regression sum of squares Eplained variation attributable to the relationship between and

36 Eplained and Uneplained Variation i SST = ( i - ) SSE = ( i - i ) _ SSR = ( i - ) _ X i

37 Coefficient of Determination, R The coefficient of determination is the portion of the total variation in the dependent variable that is eplained b variation in the independent variable The coefficient of determination is also called R-squared and is denoted as R R = SSR where 0 R 1 SST

38 Coefficient of Determination, R Coefficient of determination SSR R = = SST sum of squares eplained b regression total sum of squares Note: In the single independent variable case, the coefficient of determination is R = r where: R = Coefficient of determination r = Simple correlation coefficient

39 Eamples of Approimate R Values R = 1 R = 1 Perfect linear relationship between and : 100% of the variation in is eplained b variation in R = +1

40 Eamples of Approimate R Values 0 < R < 1 Weaker linear relationship between and : Some but not all of the variation in is eplained b variation in

41 Eamples of Approimate R Values R = 0 No linear relationship between and : R = 0 The value of Y does not depend on. (None of the variation in is eplained b variation in )

42 Calculating R Values There are several equivalent was to calculate R in matri notation.

43 Estimate of Regression Error Variance The residuals from the OLS regression are the estimates of the population errors. It follows that the estimate of the population error variance σ would be the estimated residual variance, s. Where SSE = Sum of squared errors n = Sample size k = number of independent variables in the model

44 Standard Error of Estimate The standard deviation of the variation of observations around the regression line is estimated b s ε = n SSE k 1 Where SSE = Sum of squared errors n = Sample size k = number of independent variables in the model

45 The Standard Deviation of the Regression Slope The standard error of the regression coefficient estimates (b) is estimated b where: s b1 = Estimate of the standard error of the least squares slope s ε = SSE n = Sample standard error of the estimate

46 Comparing Standard Errors Variation of observed values from the regression line Variation in the slope of regression lines from different possible samples small s ε small s b1 large s ε large s b1

47 Inference on the OLS estimates t test for a population slope Is there a linear relationship between and? Null and alternative hpotheses H 0 : β 1 = 0 (no linear relationship) H 1 : β 1 0 (linear relationship does eist) Test statistic t = d.f. b1 β s = b 1 n 1 where: b 1 = Sample regression slope coefficient β 1 = Hpothesized slope s b1 = Estimator of the standard error of the slope

48 Inference about the Slope: t Test House Price in $1000s () Square Feet () Estimated Regression Equation: house price = (sq.ft.) The slope of this model is Does square footage of the house affect its sales price?

49 Inferences about the Slope: t Test Eample H 0 : β 1 = 0 H A : β 1 0 Test Statistic: t = 3.39 From Ecel output: Coefficients Intercept b 1 Standard Error sb 1 t Stat t P-value Square Feet d.f. = 10- = 8 α/=.05 α/=.05 Reject H 0 Reject H t 0 α/ Do not reject H -t 0 α/ Decision: Reject H 0 Conclusion: There is sufficient evidence that square footage affects house price

50 Calculating a Confidence Interval Confidence Interval Estimate of the Slope: b ± t s 1 α/ b 1 d.f. = n - Ecel Printout for House Prices: Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Intercept Square Feet At 95% level of confidence, the confidence interval for the slope is (0.0337, )

51 Regression Analsis for Description Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Intercept Square Feet Since the units of the house price variable is $1000s, we are 95% confident that the average impact on sales price is between $33.70 and $ per square foot of house size This 95% confidence interval does not include 0. Conclusion: There is a significant relationship between house price and square feet at the.05 level of significance

52 Confidence Interval for the Average, Given Confidence interval estimate for the mean of given a particular p Size of interval varies according to distance awa from mean, ŷ 1 (p ) ± t α /sε + n ( )

53 Confidence Interval for an Individual, Given Confidence interval estimate for an Individual value of given a particular p ŷ 1 ( ) ± t s 1+ + p α / ε n ( ) This etra term adds to the interval width to reflect the added uncertaint for an individual case

54 Interval Estimates for Different Values of Prediction Interval for an individual, given p = b 0 + b 1 Confidence Interval for the mean of, given p p

55 Eample: House Prices House Price in $1000s () Square Feet () Estimated Regression Equation: house price = (sq.ft.) Predict the price for a house with 000 square feet

56 Eample: House Prices Predict the price for a house with 000 square feet: house price = (sq.ft.) = (000) = The predicted price for a house with 000 square feet is ($1,000s) = $317,850

57 Confidence Interval for Prediction of Average Confidence Interval Estimate for E() p Find the 95% confidence interval for the average price of,000 square-foot houses Predicted Price Y i = ($1,000s) 1 (p ) tα/ sε + = ± n ( ) ŷ ± 37.1 The confidence interval endpoints are , or from $80, $354,900

58 Confidence Interval for Prediction of Specific Prediction Interval Estimate for p Find the 95% confidence interval for an individual house with,000 square feet Predicted Price Y i = ($1,000s) 1 (p ) tα/ sε 1+ + = ± 10.8 n ( ) ŷ ± The prediction interval endpoints are , or from $15, $40,070