Yiming Peng, Department of Statistics. February 12, 2013

Transcription

1 Regression Analysis Using JMP Yiming Peng, Department of Statistics February 12, 2013

2 2

3 Presentation and Data Short Courses Regression Analysis Using JMP Download Data to Desktop 3

4 Presentation Outline Simple Linear Regression Multiple Linear Regression Questions/Comments

5 Presentation Outline (if time permits) Regression with Binary Response Variables Individual Goals/Interests

6 1. Definition 2. Scatterplot and Correlation 3. Model and Estimation 4. Coefficient of Determination (R 2 ) 5. Assumptions 6. Caution 7. Example

7 Simple Linear Regression (SLR) is used to study the relationship between a variable of interest and another variable. Both variables must be quantitative (continuous) Variable of interest known as Response or Dependent Variable (Y) Other variable known as Explanatory or Independent Variable (X)

8 Objectives How is the response variable affected by changes in explanatory variable? We would like a numerical description of how both variables vary together. Determine the significance of the explanatory variable in explaining the variability in the response (not necessarily causation). Predict values of the response variable for given values of the explanatory variable.

9 Scatterplots are used to graphically examine the relationship between two quantitative variables. Student Beers BAC

10 After plotting two variables on a scatterplot, we describe the relationship by examining the form, direction, and strength of the association. We look for an overall pattern Form: linear, curved, clusters, no pattern Direction: positive, negative, no direction Strength: how closely the points fit the form and deviations from that pattern which cause significant changes in the direction of the overall pattern. Outliers

11 No Relationship Non-Linear Relationship Positive Linear Relationship Negative Linear Relationship Y Y Y

12 An outlier is a data value that has a very low probability of occurrence (i.e., it is unusual or unexpected). An outlier is a data value that has a very low probability of occurrence (i.e., it is unusual or unexpected). In a scatterplot, outliers are points that fall outside of the overall pattern of the relationship.

13 Correlation Measures the direction and strength of a linear relationship between two quantitative variables. Pearson Correlation Coefficient Assumption of normality Calculation: Spearman s Rho and Kendall s Tau are used for non-normal quantitative variables.

14 Properties of Pearson Correlation Coefficient -1 r 1 Positive values of r: as one variable increases, the other increases Negative values of r: as one variable increases, the other decreases Values close to 0 indicate no linear relationship between the two variables Values close to +1 or -1 indicated strong linear relationships r doesn t distinguish explanatory and response variables r has no unit Important note: Correlation does not imply causation

15 Pearson Correlation Coefficient: General Guidelines 0 r < 0.2 : Very Weak linear relationship 0.2 r < 0.4 : Weak linear relationship 0.4 r < 0.6 : Moderate linear relationship 0.6 r < 0.8 : Strong linear relationship 0.8 r < 1.0 : Very Strong linear relationship

16 The Simple Linear Regression Model Basic Model: response = deterministic + stochastic Deterministic: model of the linear relationship between X and Y Stochastic: Variation, uncertainty, and miscellaneous factors Model y i = value of the response variable for the i th observation x i = value of the explanatory variable for the i th observation β 0 = y-intercept β 1 = slope ε i = random error, iid Normal(0,σ 2 )

17 But which line best describes our data?

18 The least-squares regression line is the unique line such that the sum of the total vertical (y) distances is zero and sum of the squared vertical (y) distances between the data points and the line is the smallest possible. Distances between the points and line are squared so all are positive values. This is done so that distances can be properly added.

19 Least Square Estimation Predicted Values Residuals The distinction between explanatory and response variables is essential in regression

20 Interpretation of Parameters β 0 : Value of Y when X=0 β 1 : Change in the value of Y with an increase of 1 unit of X (also known as the slope of the line) Hypothesis Testing β 0 - Test whether the true y-intercept is different from 0 Null Hypothesis: β 0 =0 Alternative Hypothesis: β 0 0 β 1 - Test whether the slope is different from 0 Null Hypothesis: β 1 =0 Alternative Hypothesis: β 1 0

21 Analysis of Variance (ANOVA) for Simple Linear Regression Source Df Sum of Squares Mean Square F Ratio P-value Model 1 SSR SSR/1 F 1 =MSR/MSE P(F>F 1,1-α,1,n-2) Error n-2 SSE SSE/(n-2) Total n-1 SST

22

23 Coefficient of Determination (R 2 ) Percent variation in the response variable (Y) that is explained by the least squares regression line 0 R 2 1 Calculation: Prediction

24 Assumptions of Simple Linear Regression 1. Independence Residuals are independent of each other Related to the method in which the data were collected or time related data Tested by plotting time collected vs. residuals Parametric test: Durbin-Watson Test 2. Constant Variance Variance of the residuals is constant Tested by plotting predicted values vs. residuals Parametric test: Brown-Forsythe Test

25 Assumptions of Simple Linear Regression 3. Normality Residuals are normally distributed Tested by evaluating histograms and normal-quantile plots of residuals Parametric test: Shapiro Wilkes test

26 Constant Variance: Plot of Fitted Values vs. Residuals Good Residual Plot: No Pattern Bad Residual Plot: Variability Increasing Predicted Values Predicted Values

27 Normality: Histogram and Q-Q Plot of Residuals Normal Assumption Appropriate Normal Assumption Not Appropriate

28 Some Remedies Non-Constant Variance: Weight Least Squares Non-normality: Box-Cox Transformation Dependence: Auto-Regressive Models

29 Do not use a regression on inappropriate data. Pattern in the residuals Presence of large outliers Use residual plots for help. Clumped data falsely appearing linear Recognize when the correlation/regression is performed on averages. A relationship, however strong, does not itself imply causation. Beware of lurking variables.

30 A lurking variable is a variable not included in the study design that does or may have an effect on the variables studied. Lurking variables can falsely suggest a relationship. What is the lurking variable here? Some more obvious than others. Strong positive association between the number firefighters at a fire site and the amount of damage a fire does. Negative association between moderate amounts of wine-drinking and death rates from heart disease in developed nations.

31 Example Dataset: Fitness A researcher was interested in the relationship between oxygen uptake and a number of potential explanatory variables separately, including age, weight, running time, running pulse rate, rest pulse rate, and max pulse rate. Filename: Fitness0.jmp (JMP sample data)

32 Questions/Comments about Simple Linear Regression

33 Multiple Linear Regression 1. Definition 2. Categorical Explanatory Variables 3. Model and Estimation 4. Adjusted Coefficient of Determination 5. Assumptions 6. Model Selection 7. Example

34 Multiple Linear Regression Explanatory Variables Two Types: Continuous and Categorical Continuous Predictor Variables Examples Time, Grade Point Average, Test Score, etc. Coded with one parameter β # x # Categorical Predictor Variables Examples Sex, Political Affiliation, Marital Status, etc. Actual value assigned to Category not important Ex) Sex - Male/Female, M/F, 1/2, 0/1, etc. Coded Differently than continuous variables

35 Multiple Linear Regression Similar to simple linear regression, except now there is more than one explanatory variable, which may be continuous and/or categorical. Model y i = value of the response variable for the i th observation x #i = value of the explanatory variable # for the i th observation β 0 = y-intercept β # = parameter corresponding to explanatory variable # ε i = random error, iid Normal(0,σ 2 )

36 Multiple Linear Regression Least Square Estimation Predicted Values Residuals

37 Multiple Linear Regression Interpretation of Parameters β 0 : Value of Y when X=0 Β # : Change in the value of Y with an increase of 1 unit of X # in the presence of the other explanatory variables Hypothesis Testing β 0 - Test whether the true y-intercept is different from 0 Β # - Null Hypothesis: β 0 =0 Alternative Hypothesis: β 0 0 Test of whether the value change in Y with an increase of 1 unit in X# is different from 0 in the presence of the other explanatory variables. Null Hypothesis: β # =0 Alternative Hypothesis: β # 0

38 Multiple Linear Regression Adjusted Coefficient of Determination (ADJ R 2 ) Percent variation in the response variable (Y) that is explained by the least squares regression line with explanatory variables x 1, x 2,,x p Calculation of R 2 : The R 2 value will increase as explanatory variables added to the model The adjusted R 2 introduces a penalty for the number of explanatory variables.

39 Multiple Linear Regression Other Model Evaluation Statistics Akaike Information Criterion (AIC or AICc) Schwartz Information Criterion (SIC) Bayesian Information Criterion (BIC) Mallows C p Prediction Sum of Squares (PRESS)

40 Multiple Linear Regression Model Selection 2 Goals: Complex enough to fit the data well Simple to interpret, does not overfit the data Study the effect of each explanatory variable on the response Y Continuous Variable Graph Y versus X Categorical Variable - Boxplot of Y for categories of X

41 Multiple Linear Regression Model Selection cont. Multicollinearity Correlations among explanatory variables resulting in an increase in variance Reduces the significance value of the variable Occurs when several explanatory variables are used in the model Diagnostic Variance Inflation Factor (VIF) Correlation Matrix

42 Multiple Linear Regression Algorithmic Model Selection Backward Selection: Start with all explanatory variables in the model and remove those that are insignificant Forward Selection: Stepwise Selection: Start with no explanatory variables in the model and add best explanatory variables one at a time Start with two forward selection steps then alternate backward and forward selection steps until no variables to add or remove

43 Multiple Linear Regression Example Dataset: Fitness A researcher was interested in the relationship between oxygen uptake and a number of potential explanatory variables together, including age, weight, running time, running pulse rate, rest pulse rate, and max pulse rate. Filename: Fitness0.jmp (JMP sample data)

44 Multiple Linear Regression Other Multiple Linear Regression Issues Outliers Interaction Terms Higher Order Terms

45 Multiple Linear Regression Questions/Comments about Multiple Linear Regression

46 Regression with Non-Normal Response Logistic Regression with Binary Response

47 Logistic Regression Consider a binary response variable. Variable with two outcomes One outcome represented by a 1 and the other represented by a 0 Examples: Does the person have a disease? Yes or No Who is the person voting for? Outcome of a baseball game? Romney or Obama Win or loss

48 Logistic Regression Consider the linear probability model y i = β 0 + β 1 x i where y i = response for observation i x i = quantitative explanatory variable Predicted values represent the probability of Y=1 given X Issue: Predicted probability for some subjects fall outside of the [0,1] range.

49 Logistic Regression Consider the logistic regression model [ ] = P(Y i =1 x i ) = π(x i ) = exp ( β 0 + β 1 x i ) 1+ exp( β 0 + β 1 x i ) E Y i ( xi ) ( x ) π logit[ π ( xi )] = log = β0 + β1x 1 π i Predicted values from the regression equation fall between 0 and 1 i

50 Logistic Regression Interpretation of Coefficient β Odds Ratio The odds ratio is a statistic that measures the odds of an event compared to the odds of another event. Say the probability of Event 1 is π 1 and the probability of Event 2 is π 2. Then the odds ratio of Event 1 to Event 2 is: Odds _ Ratio Odds( π1) = Odds ( π ) 2 = π1 1 π1 π 2 1 π 2 Value of Odds Ratio range from 0 to Infinity Value between 0 and 1 indicate the odds of Event 2 are greater Value between 1 and infinity indicate odds of Event 1 are greater Value equal to 1 indicates events are equally likely

51 Logistic Regression Example Dataset: APACHE II Score and Mortality in Sepsis The following figure shows 30 day mortality in a sample of septic patients as a function of their baseline APACHE II Score. Patients are coded as 1 or 0 depending on whether they are dead or alive in 30 days, respectively. We wish to predict death from baseline APACHE II score in these patients. Filename: APACHE.jmp Important Note: JMP models the probability of the 0 category

52 Thank you!