Regression and Correlation

Transcription

1 STP3 Brief Class Notes Instructor: Ela Jackiewicz Chapter Regression and Correlation In this chapter we will consider bivariate data in forms of ordered pairs (x, y). Both x and y can be observed or y can be observed for specific values of x that are selected by the researcher. Data must display a linear trend on the scatterplot for us to consider fitting a linear equation that will expres in terms of x. We will use the following vocabulary with respect to x and y: y response variable or dependent variable x predictor variable/explanatory variable or independent variable Our line is called Least Squares Regression Line and will have a form: y=b 0 b x, where y is a predicted value of y for given x. The name of the line reflects the fact that our line has smallest possible sum of squared errors (of all lines that can be possibly fitted to the given data). Errors or residuals are : e= y y Formulas for the slope and y-intercept of Least-Squares Regression equation: y =b 0 b x : b = (x i x)(y i y) (x i x) = r, b 0 =y b x = where x i x n = y i y n (standard deviations of x-s and y-s respectively) Following example will illustrate the computations. In our example we consider people on the diet, X=# of times per week person eats out, Y= change in weight (in pounds),(before-after) after three months period of dieting x i y i x i x (x i x) y i y ( y i y) (x i x)( y i y) SS(total) x=5/5=5 y=0/5=8 b = /= b 0 =8 5 =8, so our equation is : y=8 x Equation tells us that for every time increase in x (times to eat out in a week), y (weight change) decreases by pounds. The more often you go out to eat the less pounds are lost Units of the slope are : (units of y)/ (units of x) Total Sum of squares SS total = y i y gives total variability in y values

2 STP3 Brief Class Notes Instructor: Ela Jackiewicz Linear correlation Coefficient of X and Y, r : r= x n i x y i y = n ( x i x )( y y i ) = x i x y i y, x i x y i y Linear Correlation coefficient gives information about strength and direction of a linear trend in our data. Values close to or - indicate extremely strong linear association between X and Y. Linear correlation r is symmetric, if X and Y are reversed, value of r remains the same. Sign(r)=sign of the slope, r is unit free and r. In our example r= 0 = - 0., indicating very strong negative linear trend. Warning about correlation: strong (positive or negative) correlation does not have to imply causal relationship between x and y. Making predictions using our equation: Predict number of pounds lost if dieting person eats out 5 times per week: y=8-0=8 lb (this prediction is for x within data range) Warning about making predictions: It is safe to predict y for x within data range. Extrapolation - making predictions for values of the predictor variable outside the range of the observed values of the predictor variable (x) Grossly incorrect predictions can result from extrapolation. For ex. Prediction for x=5 i=- (weight gain of lb). One may think that value is reasonable, weight gain is to be expected if you eat out so often, but because we have no data in that range, we may not be sure that our particular trend will hold as far out of the range of x values. Besides Total sum of squares there are two other important sums of squares that we will use. We can now obtain residual sum of squares, and regression sum of squares by obtaining predicted values of y for each x, by using our equation. x i y i ŷ i y i ŷ ( y i ŷ ) ŷ i y ( ŷ i y ) SS(resid) SS(reg)

3 STP3 Brief Class Notes Instructor: Ela Jackiewicz Residual Sum of Squares: SS resid = y i y i by the regression equation gives total variability in y values unexplained Regression Sum of Squares: the regression equation SS reg = y i y gives total variability in y valuexplained by Regression Identity: SS(total)=SS(reg)+SS(resid) We can verify that in our example : 0=88+8 We will use the three sums of squares in farther definitions. Problems we may encounter with our data: Outlier a data point that lies far from the regression line, relative to other data points Influential observation a data point whose removal causes the regression to change considerably. It is usually separated in the x-direction from the other data points. It pulls the regression line towards itself. Assessing the fit of our regression line: = We can use SS(resid) to compute Residual Standard Deviation: s SS(resid) e n This measure tells us how close are the data points to our fitted regression line, the smaller is this value, the better is the fit of our regression line. If is smaller than (standard deviation of y-s), it means a good fit. The units of are the same as units of y. In our example = (8/3)=.5lb<5.5= indicating a good fit. ( = In the nice data, if it is not too small, we expect roughly : 8% of observed y-s to be within ± of the regression line 5% of observed y-s points to be within ± of the regression line and.% of observed y-s points to be within ±3 of the regression line 0 5 =5.5 ) In other words these percentages (roughly) of data points are expected to be within a vertical distances (parallel to the y axis) above and below the regression line. Another value that tells us about the fit of the regression line is: Coefficient of Determination: r SS reg SS resid = = SS total SS total (square of linear corr. coef. r) This measure gives us percentage of total variability in y values that ixplained by our regression line. If this is close to (00%), then that indicates a good fit. 0 r

4 STP3 Brief Class Notes Instructor: Ela Jackiewicz In our example r = 8 0 = 88 0 =0.83, it indicates a good fit, 83% of total variability in y-values ixplained by the regression line. There is an approximate relationship of r to and. This approximation is good for large data r If this ratio is small, it indicates a good fit Above formula can also be expressed as r s e and used to approximate r for large data. In our example 0.83 =0. and approximation, since our data is small. = =0.8, and r 0. not a great Hypothesis test and CI for the slope and correlation coefficient: Assuming that the following linear model applies to X and Y: Linear Model has a form: Y = Y X, where Y X is a linear function of X, so Y = 0 X Y X = population mean Y value for given X, Y X = population standard deviation of Y-s for given X value, population correlation coefficient is (rho), Term ϵ in our model represents random error, we include it in the model to reflect that Y varieven if X is fixed. We can test hypothesis related to the true slope and true correlation coefficient. The terms we compute for Least Squares Regression are estimating parameters in our model: b 0 estimates 0, b estimates, r estimates, estimates σ Y X =σ ϵ, since it is independent of x If we want to test if slope is zero (indicating no relationship between x and y), our hypotheses are: H 0 : =0 vs H a :β 0 or H a :β > 0 or H a :β < 0 Choice of alternative depends on the question. Test statistics: t s = b SE b df=n-, Standard Error of the slope b : SE b = (x i x) = n, the second formula iasier to compute,

5 STP3 Brief Class Notes Instructor: Ela Jackiewicz We can also compute Confidence interval for a true slope: 5% CI for : b ±t.05 SE b Equivalently we may test hypotheses involving true correlation coefficient. Our hypotheses are: H 0 : =0 vs H a :ρ 0 or H a :ρ> 0 or H a :ρ< 0 Test statistics: t s =r n r, df=n- iquivalent to the one before, but easier to compute. In our example we may ask the question: Assuming that a linear model applies, test the hypothesis that there is no relationship between x and y against an alternative that y decreases with increasing x. Use 5% significance level. Our test hypotheses are: H 0 : =0 and H a :β < 0 or ( H 0 :ρ=0 and H a :ρ< 0 ) We will use second test statistics: t=. 5 = 3.8 P-value=0.05< Null is rejected, we have evidence for alternative. Yes, y decreases with increasing x, there is statistically significant negative linear relationship between x and y. Different (Computational) formulas for the slope, r and all three sums of squares are: b = S xy, SS resid =Syy S xy, SS total =S S yy, SS reg = S xy, xx S xx S xx r= S xy S xx S yy, r = S xy S xx S yy where: S xx = x i x i CALCULATOR n, S yy = y i y i n, S xy = x i y i x i y i n ) To use above computational formulas and/or find basic statistics for x and y use STAT EDIT place x-s and y-s on L and L respectively, then STAT CALC, select -Var Stats <enter> -Var Stats L, L <enter> )To find the regression line: STAT EDIT place x-s and y-s on L and L respectively STAT CALC, select LinReg(ax+b) <enter> LinReg(ax+b) L, L <enter>

6 STP3 Brief Class Notes Instructor: Ela Jackiewicz Output should display slope, y intercept, r and r.if you do not see r and r, then go to the CATALOG, select DIAGNOSTICS ON <enter><enter>, next time you run the regression it will show both values. 3) To perform the test: STAT TESTS use LinRegT-Test, place Xlist L Ylist L, then select appropriate alternative hypothesis (They use β for β, test is for both β and ρ ) In the output value of s=, you also get values of and there - ) To compute CI for β use STAT TESTS use LinRegT-interval, place Xlist L Ylist L, then select appropriate C-level If you do not have the CI on your calculator, use s from the LinRegT-Test output to compute SE b = n and compute CI by hand. Least squares Linear Regression Example Is number of beers consumed a good predictor of BAC (blood alcohol content)? Can you drive legally after beers (legal limit is.08)? Random sample of students from Ohio State University participated in a study. Each student was assigned randomly number of beers to drink and after 30 minutes a police officer measured their BAC in grams of alcohol per deciliter of blood. The results are given in the table below. # of beers=x BAC=y Answer following questions: Graph these data to confirm existence of a linear trend. Obtain LS regression line for these data. What is the slope of your line telling you about change in BAC after you drink a next beer? Do you think y-intercept value is sensible? Compute linear correlation coefficient and coefficient of determination. What % of total variability in BAC ixplained by the regression line? Are r and r indicating a good fit? Clearly answer both questions posed at the beginning of the problem. Can we use our equation to safely predict BAC for one that drinks 5 beers? Given SS(resid)=.005, compute, compare it to and decide if it indicates a good fit. Test the hypothesis of no linear relationship between x and y against directional alternative hypothesis that y increases with increasing x. Obtain 5% CI for the true slope β