5. Linear regression and correlation

Transcription

1 Statistics for Engineers Linear regression and correlation If we measure a response variable at various values of a controlled variable, linear regression is the process of fitting a straight line to the mean value of at each. For eample ou might measure fuel efficienc at various 50 values of an eperimentall controlled eternal temperature, and then tr to fit a straight line 0 to the results (assuming ou think there is an 150 underling linear relationship). 0 e.g. the plot of against suggests that it is reasonable to fit a straight line Model Sa we take n measurements of a function () obtaining for each i a value. When plotted on a scatter diagram, there is a straight line relationship between and, apart from random variation in each measurement. Model: where a + b i is the linear relation and e i is the random error. We assume ( ) for all i, and 's are independent, and want to estimate a and b, using the data. The likelihood P(D a,b) can be found since e i are Normal, i.e. ( ) we have the log likelihood, hence since (* + ) ( ) The maimum likelihood estimator can be found b maimizing this log likelihood. This is equivalent to minimizing ( ) since σ is a constant. Minimizing E is minimizing the squared error. Even when the random error is more complicated than a simple Normal, E can still be defined, and least-squared values can be calculated, though the ma not have a ver clear interpretation. In fact regression is often used completel blindl, without knowing the model the samples are drawn from, and can still be useful to identif correlations between variables.

2 Statistics for Engineers 5- Least squares estimates of a and b The least-squares estimates and must satisf and, i.e. (1) ( ) () ( ) Therefore, a and b satisf: ( ) ( ) ( ) ( ) ( ) Solving (1) gives:. Substituting into () then gives: ( ) ( + Solving for gives the final answer Here ( )( ) ( ) ( ) It can be shown that a and b are unbiased. Note variance. is just proportional to the sample

3 Statistics for Engineers 5-3 The fitted regression line is: i.e. the regression line passes through ( ) ( ) Eample: The data has been observed for various values of, as follows: Fit the simple linear regression model using least squares. Answer : n = 9,,,, Hence ( ) ( ) So the fit is approimatel

4 Statistics for Engineers 5-4 Estimating : variance of about the fitted line Estimated error is: The mean error is zero, so the ordinar sample variance of the 's is In fact, this is biased since two parameters, a and b have been estimated. The unbiased estimate is: ( ) Using ( ) then (( ) ( )) (( ) ( )( ) ( ) ) ( ) ( * Confidence interval for the slope, b Recall that for Normal data with unknown variance, confidence interval for is: s X tn 1 n s to X tn 1 n The quantit is the estimate of, the variance of X It can be shown that var( b ) = /S, estimated b /S (n- df). Confidence interval for b is b tn S to b tn S

5 Statistics for Engineers 5-5 Predictions One common reason for fitting a linear regression model is to predict given. Predicted value for the mean at is Confidence interval for mean at given It can be shown that var( ) = var( a b ) = 1 n S Therefore, confidence interval for mean is: i t n 1 n S Etrapolation: using the fitted model to predict outside the range of 's used estimating a and b. This ma be misleading, as the approimate linear relation ma not continue to hold beond the range for which ou have observations. Eample: Using the previous data, what is the mean value of at and the 95% confidence interval? Answer The epected value is For the confidence interval need (with ) For 95% confidence need for Q=0.975, i.e..

6 Statistics for Engineers 5-6 Hence confidence interval is ( ( ) ) ( ( ), Confidence interval for a prediction Often we want to predict the range a future data point might lie, rather than just calculate the mean. This confidence interval for a single response (measurement of at ) is given b ( ( ) ) This is larger because it is a combination of the uncertaint in the mean, and the epected scatter of a given point about the mean. Eample: Using the previous data, what is the 95% confidence interval for a measurement of at Answer ( ( ) ) ( ( ), Correlation Regression tries to model the relation between and. Correlation tries to measure the strength of the linear association between and.

7 Statistics for Engineers A B Same fitted line in both cases, but stronger linear association in case B. What does correlation mean? If and are positivel correlated, then if is high is epected to be high, if is low then is epected to be low. In other words, on average ( )( ) is epected to be positive: both if and are below the mean, or if and are above the mean. Similarl for a negative correlation ( )( ) is epected to be negative. We can therefore use ( )( ) to quantif the correlation. It is often convenient to normalize b the variance of the and, giving the definition of the correlation coefficient: This is sometimes called the Pearson product-moment. The range is: : r = 1: there is a line with positive slope going through all the points; r = -1: there is a line with negative slope going through all the points; r = 0: there is no linear association between and. Eample: from the previous data, hence The magnitude of r measures how nois the data is, but not the slope. Also onl means that there is no linear relationship, and does not impl the variables are independent there could be man more complicated relationships that do not fit a straight line:

8 Statistics for Engineers 5-8 In general it is not eas to quantif the error on the estimated correlation coefficient. Possibilities include subdividing the points and assessing the spread in r values. Also, of course does not impl that changes in cause changes in - additional tpes of evidence are needed. Eample: Earnings and height: error Correlation r J Polit Econ. 08; 116(3):

9 Statistics for Engineers 5-9 So there is strong evidence for a -3% correlation. This doesn t mean being tall causes ou earn more (though it could). For eample height could be correlated with cognitive abilit, and cognitive abilit causes ou to earn more. In fact this appears to be the case: height is correlated with intelligence, both higher height and higher intelligence being caused b better health and nutrition during development. There could also be a genetic component (mabe smart women slightl prefer tall men perhaps because it is an indicator of health and nutrion and the then have tall smart children). Determining the the reason for an empirical correlation is usuall etremel difficult.