Regression. In this class we will:

Transcription

1 AMS 5 REGRESSION

2 Regression The idea behind the calculation of the coefficient of correlation is that the scatter plot of the data corresponds to a cloud that follows a straight line. This idea can be formalized by regression methods. In this class we will: Consider the definition of simple linear regression Find a method to predict an individual value Use the normal curve to estimate the percentile ranks Describe the regression effect Compute the regression errors and its RMS Study the behavior of regression errors

3 Regression The regression method describes how one variable depends on another. The Northern California temperature data have average altitude of 3,524 feet and a SD of 1,839 feet; average temperature of 70.3 degrees and SD 6.5 degrees. The correlation between temperature and altitude is

4 Regression The idea behind the calculation of the coefficient of correlation is that the scatter plot of the data corresponds to a cloud that follows a straight line. This idea can be formalized by regression methods. In this class we will: Consider the definition of simple linear regression Find a method to predict an individual value Use the normal curve to estimate the percentile ranks Describe the regression effect Compute the regression errors and its RMS Study the behavior of regression errors

5 Regression The cloud of points shows a mild negative association between the two variables, as does the value of r. Can we use the values of altitude to estimate the average values of temperature?

6 Regression How does the regression line work? Associated with an increase of one SD in x there is an increase of r SDs in y on average. Clearly, if the correlation coefficient is negative, then the average value of y decreases as x increases. In the temperature and altitude example, an increase of height of 1,839 feet produces a increase of = degrees in the average temperature.

7 Regression How do we use the method to predict an individual value? If we consider two variables x and y and we want to predict the value of y for a specific value of x, we use the average value of y that corresponds to the value of x according to the regression method. Example: The first year GPAs and the Math SAT for the students of a university produce the following data average SAT score = 550, SD = 80 average 1st-year GPA = 2.6, SD = 0.6 r = 0.40 We want to predict the 1st-year GPA of a student with a SAT score of 650.

8 Regression The student's SAT score in standard units is = so the score is 1.25 SDs above average. An increase of one SD above the average SAT score produces an increase of 0,4 0,6 GPA points. This implies that our student will have an increase of = 0.3 points of GPA above average. Since the average GPA is 2.6, the predicted GPA is = 2.9 This is the average GPA that we expect for students with STA scores around 650.

9 Regression WARNING: You can use the regression method on new subjects provided that they are similar to the ones that were used to produce the averages, SDs and r used in the regression method. In the previous example the method will not be valid for students of a different institution.

10 Regression We can use the regression method and the normal curve to produce estimates of the percentile ranks. Example: In the previous example suppose a student has a percentile rank of 90% for the SAT scores. That is, only 10% of the scores are higher than his. What is the predicted percentile rank for the 1st-year GPA of this student? Using the normal curve we have that a 90% probability corresponds to z score of 1.3. This means that the student's SAT score is 1.3 SDs above average. This corresponds to being SDs above the average GPA and this corresponds to an accumulated probability, under the normal curve, of approximately 69%.

11 Regression So the percentile rank on 1st-year GPA of a student with a percentile rank on SAT score of 90% is predicted to be 69%. Notice that the student with a SAT percentile rank of 90% was `pulled down' to only 69% by the regression method. Why is that? Suppose the correlation was perfect, r = 1, then 90% will convert to 90%. The other extreme is that there is no correlation, so, in the absence of any information, the best guess is the median or 50% percentile. The regression method produces a rank that is somewhere between these two extremes.

12 Example The shoe size and the heights of 14 men are recorded. The shoe size average is with a SD of The average height is inches with a SD of 2.45 inches. The correlation is What is the average height of a man that uses shoes of size 11.5? We convert 11.5 to standard units = so the shoe size is units above average. This means that the height will be = 1.95 inches above average. So the average height of a man with shoe size 11.5 will be = inches.

13 Regression effect Galton, a British statistician, studied the relationship between the height of the fathers and the sons in 1,078 families. He noticed that tall fathers tended to have shorter sons and short fathers tended to have taller sons. He termed this fact regression to mediocrity. This is where the term regression comes from. Example: Children are tested for IQ before and after taking a preschool program. In both cases the scores average 100 and the SD is 15. So, on average, there seems to be no effect. Nevertheless children below average in the first test had an average gain of 5 IQ and those above average had an average loss of 5 IQ. This is regression effect.

14 Regression effect A model for the test-retest situation is observed test score = true score + chance error Suppose that the chance error can be either positive or negative. Suppose that the true scores in the population follow the normal curve with an average of 100 and a SD of 15. Consider the children who scored 140 on the first test. There are two possibilities: true score below 140, with a positive chance error true score above 140, with a negative chance error Which one is more likely? According to the normal curve, the first possibility is more likely, since the mean is 100 and so the interval above 140 has less probability than the one below 140. Under this scenario, the second test is more likely to produce a value below 140.

15 Regression effect A symmetric situation is valid for those scoring, say, 80 IQ. It is likely that the true test is above 80 with a negative chance error, and so the second score is likely to be above the first. In other words, if a students scores above average in the first test, it is likely that the true score is lower than the observed one. If the student takes the test again, chances are that the second score will be lower than the first. A symmetric situation is true for a person scoring below average in the first test. This explains the regression effect.

16 Regression errors The regression method can be used to predict y from x. But actual values differ from predictions. These are the regression errors. error = actual value of y - predicted value of y Some of the errors defined in this way are positive and some are negative. Reflecting the fact that some observations are above and some are below the regression line. How do we measure the error in a regression? The overall size of the error is measured using the root-meansquare (RMS), as we did to obtain the SD. This is equal to where N is the number of points in the scatter diagram.

17 Regression errors What if we ignore the values of x? Then our prediction for y is the average of y. In this case the RMS error coincides with the SD of y.

18 Computing the RMS error We saw that the error that corresponds to a prediction where the values of x are ignored corresponds to the SD of y. The overall size of the error for a regression using x has to be smaller than the SD. How much smaller? 2 RMS error= 1 r SD of y We observe the following features The units of the RMS error are the same as the units of the variable being predicted. Perfect correlation corresponds to zero RMS error. Zero correlation corresponds to maximum RMS error (equal to SD of y).

19 Computing the RMS error Example 1: In the California temperature example we had that the SD of y is 6.5 degrees and the correlation is -0.76, then degrees 4.22 degrees So, in this case, knowing the altitude reduces the SD from 6.5 to 4.22 degrees. Example 2: In the shoe sizes examples we had that the SD of y is 2.45 inches and the correlation is 0.93, then inches 0.90 inches So we observe that, knowing the shoe size produces a dramatic reduction of the SD from 2.45 to 0.90.

20 Plotting the residuals Prediction errors are usually called residuals. It is important to explore the graphical properties of residuals to find out about the goodness of the fit by the regression line. In a residual plot the x coordinates are the same as for the original data. The y coordinates correspond to the values of the residuals. So there is one point for each point in the original scatter diagram.

21 Thus, if everything is OK with the regression line, we expect to see a cloud of points around the zero line in the y axis. Plotting the residuals We expect to see no trends or clusters in the residuals There should be about the same number of positive as negative residuals A histogram of the residuals should look symmetric around zero

22 Problem The following results are taken from a study of about 1,000 families: average height of husband 68 inches, SD 2.7 inches average height of wife 63 inches, SD 2.5, r 0.25 Predict the height of a wife when the height of her husband is inches The husband is 4 inches above average height. This is 4/2.7 = 1.5 SD above the average. So the wife is predicted to have r 1.5 = this corresponds to = 1 inch inches This the husband is right on the average, so the wife will be right on the average as well.

23 Prediction for data in a vertical strip Example: A law school finds the following relationship between LSAT scores and first-year scores average LSAT score = 162, SD = 6 average first-year score = 68, SD = 10, r=0.60 Q: About what percentage of the students had first-year scores over 75? A: We use the normal curve approximation. Converting to standard units = 0.7 this corresponds to a right hand tail of 14% under the normal curve.

24 Prediction for data in a vertical strip Q: Of the students who scored 165 on the LSAT, about what percentage had first-year scores over 75? A: We first convert to standard units for the x variable: = 0.5 then convert to standard units for the y variable r 0.5= = 0.3 which corresponds to = 3 points above average or 68+3 = 71. Since the data corresponding to a strip are a smaller and more homogeneous sample, the corresponding SD will be smaller. How much smaller?

25 Prediction for data in a vertical strip Example: A law school finds the following relationship between LSAT scores and first-year scores average LSAT score = 162, SD = 6 average first-year score = 68, SD = 10, r=0.60 Q: About what percentage of the students had first-year scores over 75? A: We use the normal curve approximation. Converting to standard units = 0.7 this corresponds to a right hand tail of 14% under the normal curve.

26 Prediction for data in a vertical strip We expect the dispersion in the y variable to be about the same for each vertical strip. This is given by the RMS error, thus the new SD is r SD of y= = 8 points This new SD can be used to convert to standard units = and, using the normal curve, we obtain an area of 31% above 0.5. This is the percentage of students scoring more than 75 in the first year among those who scored 165 in the LSAT. Notice that this percentage is higher than the 14% we obtained before. This is because we have focus on a smaller portion of the sample, obtaining a smaller SD.

27 Prediction for data in a vertical strip In summary, when considering data for a vertical strip: Convert to standard units in the x variable. Obtain the predicted value of the y variable. Calculate the SD for the y variable in the strip using RMS error. Convert to standard units in the y variable and use the normal curve.

28 Slope and intercept All lines can be determined by a slope and an intercept. The intercept is the height of the line when x = 0. The slope is the rate at which y increases, per unit increase in x. If the slope is negative then y decreases as x increases.

29 Slope and intercept How do you get the slope of a regression line? Example: A sample of 555 California men age in 1993 was surveyed to find out about education and income. The data are summarized by average education 12.5 years; SD 4 years average income $21,500; SD $16,000; r 0.35 This means that, for every increase of one SD in education, there is an increase of r SD in income. Thus, 4 extra years of education are worth an extra 0.35 $16,000 = $5,600 of income. So, each extra year is worth 0.35 $16, this, is the slope of the regression line. = $1, 400

30 Slope and intercept The intercept of the regression line is given by the value of y when x = 0. This is 12.5 years below average in education. Since each year costs $1,400, a man with no education should have an income which is below average by 12.5 years $1,400 per year = $17,500 since the average income is $21,500, the income of a man with no education is $21,500 -$17,500 = $4,000. This is the intercept of the regression line. This corresponds to the change in y associated with one unit increase in x.

31 Slope and intercept This is given by average of y - slope average of x The equation for the regression line is called the regression equation and can be written as y= slope x+ intercept So, for our example, we have that predicted income = $1,400 per year education + $4,000

32 Slope and intercept Q: What is the predicted income of a man with an education of 15 years? A: Using the regression equation we have y = $1, $4,000 = $25,000 we can plug in any value of education and obtain the expected income for that level of education. Warning: It is usually a bad idea to use the regression line for extrapolations.

33 Example Back to our shoe size example. The shoe size and the heights of 14 men are recorded. The shoe size average is with a SD of The average height is inches with a SD of 2.45 inches. The correlation is r SD of height The slope of the regression line is = = 1.88 SD of shoe size 1.21 To obtain the intercept we consider a show size of zero. This is units below average and so will correspond to a height that is = inches below average. So it corresponds to a height of = inches. The regression line is height = 1.88 shoe size inches Q: What is the predicted height of a man with a show size of 9? A: Using the regression equation we have inches = inches

34 Least Square Consider a cloud of points produced by obtaining the scatter diagram of observations corresponding to two variables x and y. There are many lines that we can draw through the cloud. Which is the straight line that fits the points best? The regression line is a possible solution to this problem. This is the reason why the regression line is called the least squares line.

35 Least Square Example: Let b be the length of a spring with no load. If a load x is attached to the spring the stretch is proportional to x. Thus the length of the string is y = mx + b. where m and b are constants that depend on the string. An experiment is run to determine the constants for a given spring, the data are shown in the table. The correlation coefficient is r = 0.999, so the points are very close to straight line. But they are not exactly on a straight line. This is probably due to measurement error. The regression line for these data produces estimates of b and m, given, respectively, by the intercept and the slope of the line. The values are m 0.5c per kg, and b cm. These are the least squares estimates of m and b.

36 Problem Find the regression equation for predicting final score from midterm score, based on the following information: average midterm score = 70, SD = 10 average final score = 55, SD = 20, r = 0.60 The slope of the line can be obtained as r SD of final = = 1.2 SD of midterm 10 A score of 0 in the midterm will correspond to a final score that is = 84 units below average. So the intercept is = -29 units of the final score. Thus, the regression equation is final score = 1.2 midterm score - 29