A. Karpinski

Transcription

1 Chapter 5 Smple Lnear Regresson Regresson Dagnostcs and Remedal Measures Page. Resduals and regresson assumptons 5-. Resdual plots to detect lack of ft Resdual plots to detect homogenety of varance 5-4. Resdual plots to detect non-normalty Identfyng outlers and nfluental observatons Remedal Measures: 5-33 n overvew of alternatve regresson models 7. Remedal Measures: Transformatons Karpnsk

2 Smple Lnear Regresson Regresson Dagnostcs and Remedal Measures. Resduals and regresson assumptons The regresson assumptons can be stated n terms of the resduals ε ~ NID (, σ ) o ll observatons are ndependent and randomly selected from the populaton (or equvalently, the resdual terms, ε s, are ndependent) o The resduals are normally dstrbuted at each level of X o The varance of the resduals s constant across all levels of X We must also assume that the regresson model s the correct model o The relatonshp between the predctor and outcome varable s lnear o No relevant varables have been omtted o No error n the measurement of predctor varables Types of resduals o (Unstandardzed) resduals, e e = Y Yˆ resdual s the devaton of the observed value from the predcted value on the orgnal scale of the data If the regresson model fts the data perfectly, then there would be no resduals. In practce, we always have resduals, but the presence of many large resduals can ndcate that the model does not ft the data well If the resduals are normally dstrbuted, then we would expect to fnd 5% of resduals greater than σ from the mean % of resduals greater than.5σ from the mean.% of resduals greater than 3 σ from the mean It can be dffcult to eyeball standard devatons from the mean, so we often turn to standardzed resduals 4-. Karpnsk

3 o Standardzed resduals, e~ Y Yˆ Y Yˆ ~ e = = σ MSE Standardzed resduals are z-scores. Why? The average of the resduals s zero e e = = n e The standard devaton of the resduals s MSE ( e e) e Var( e) = = n n SSE = = MSE n So a standardzed resdual would be gven by: e e e Y Y e~ = = = σ MSE MSE e Because standardzed resduals are z-scores, we can easly detect outlers. When examnng standardzed resduals, we should fnd: 5% of e~ s greater than % of e~ s greater than. 5.% of e~ s greater than 3 o Studentzed resduals, e MSE s the overall varance of the resduals It turns out that the varance of an ndvdual resdual s a bt more complcated. Each resdual has ts own varance, dependng on ts dstance from X When resduals are standardzed usng resdual-specfc standard devatons, the resultng resdual s called a studentzed resdual. In large samples, t makes lttle dfference whether standardzed or studentzed are used. However, n small samples, studentzed resduals gve more accurate results. Because SPSS makes the use of studentzed resduals easy, t s good practce to examne studentzed resduals rather than standardzed resduals 4-3. Karpnsk

4 Obtanng resduals n SPSS REGRESSION /DEPENDENT dollars /METHOD=ENTER mles /SVE RESID (resd) ZRESID (zresd) SRESID (sresd). o RESID produces unstandardzed resduals o ZRESID produces standardzed resduals o SRESID produces studentzed resduals o Each resdual appears n a new data column n the data edtor RESID ZRESID SRESID You can see the dfference between standardzed and studentzed resduals s small, but t can make a dfference n how the model ft s nterpreted Because all the regresson assumptons can be stated n terms of the resduals, examnng resduals and resdual plots can be very useful n verfyng the assumptons o In general, we wll rely on resdual plots to evaluate the regresson assumptons rather than rely on statstcal tests of those assumptons 4-4. Karpnsk

5 . Resdual plots to detect lack of ft There are several reasons why a regresson model mght not ft the data well ncludng: o The relatonshp between X and Y mght not be lnear o Important varables mght be omtted from the model To detect non-lnearty n the relatonshp between X and Y, you can: o Create a scatterplot of X aganst Y Look for non-lnear relatonshps between X and Y Ths scatterplot s not nformatve regardng the regresson assumptons. o Plot the resduals aganst the X values The resduals have lnear assocaton between X and Y removed. If X and Y are lnearly related, then all that should be remanng for the resduals to capture s random error Thus, any departure from a random scatterplot ndcates problems In general, ths graph s easer to nterpret than the smple scatterplot and an added advantage of ths graph (f studentzed resduals are used) s that you can easly spot outlers In smple lnear regresson, a plot of e vs X s dentcal to a plot of e vs Yˆ. Thus, there s no need to examne both of these plots. The predcted values are the part of the Ys that have a lnear relatonshp wth X, so Ŷ and X wll always be perfectly correlated when there s only one predctor. In multple regresson, dfferent nformaton may be obtaned from a plot of e vs X and from a plot of e vs Ŷ Karpnsk

6 Example #: good lnear regresson model (n = ) o scatterplot of X aganst Y GRPH /SCTTERPLOT(BIVR)=x WITH y y x.6.8. o The X-Y relatonshp looks lnear Plot the resduals aganst the X values GRPH /SCTTERPLOT(BIVR)=x WITH sresd... Studentzed Resdual o The plot looks random so we have evdence that there s no non-lnear relatonshp between X and Y o We also see that no outlers are present o Ths graph s as good as t gets! x Karpnsk

7 Example #: nonlnear relatonshp between X and Y (n = ) o scatterplot of X aganst Y GRPH /SCTTERPLOT(BIVR)=x WITH y y x o The X-Y relatonshp looks mostly lnear.8. Plot the resduals aganst the X values GRPH /SCTTERPLOT(BIVR)=x WITH sresd Studentzed Resdual x o Ths graph has a slght U-shape, suggestng the possblty of a non-lnear relatonshp between X and Y o We also see one outler Karpnsk

8 Example #3: second nonlnear relatonshp between X and Y (n = ) o scatterplot of X aganst Y GRPH /SCTTERPLOT(BIVR)=x WITH y y x o The X-Y looks slghtly curvlnear n ths case. Plot the resduals aganst the X values GRPH /SCTTERPLOT(BIVR)=x WITH sresd... Studentzed Resdual x.6.8. o Ths graph has a strong U-shape, ndcatng a non-lnear relatonshp between X and Y o Notce that t s easer to detect the non-lnearty n the resdual plot than n the scatterplot 4-8. Karpnsk

9 o You can not determne lack-of-ft/non-lnearty from the sgnfcance tests on the regresson parameters REGRESSION /STTISTICS COEFF OUTS R NOV ZPP /DEPENDENT y /METHOD=ENTER x In ths case, we fnd evdence for a strong lnear relatonshp between X and Y, b =.887,t(98) =8.95, p <. [r =.94] Model B (Constant) X a. Dependent Varable: Y Unstandardzed Coeffcents Standard zed Coeffcen ts Coeffcents a Correlatons t Sg. Zero-order Partal Part Std. Error Beta Ths lnear relatonshp between X and Y accounts for 88.5% of the varance n Y. Model Summary Model R R Square djusted R Square Std. Error of the Estmate.94 a a. Predctors: (Constant), X Yet from the resdual plot, we know that ths lnear model s ncorrect and does not ft the data well Despte the level of sgnfcance and the large percentage of the varance accounted for, we should not report ths erroneous model Detectng the omsson of an mportant varable by lookng at the resduals s very dffcult! 4-9. Karpnsk

10 3. Resdual plots to detect homogenety of varance We assume that the varance of the resduals s constant across all levels of predctor varable(s) To examne f the resduals are homoscedastc, we can plot the resduals aganst the predcted values o If the resduals are homoscedastc, then ther varablty should be constant over the range GOOD BD BD o s prevously mentoned, plottng resduals aganst ftted values (Yˆ ) or aganst the predctor (X) produces the same plots when there s only one X varable. In multple regresson, a plot of the resduals aganst ftted values (Yˆ ) s generally preferred, but n ths case t makes no dfference o The raw resduals and the standardzed resduals do not take nto account the fact the varance of each resdual s dfferent (and depends on ts dstance from the mean of X). For plots to examne homogenety, t s partcularly mportant to use the studentzed resduals 4-. Karpnsk

11 Example #: homoscedastc model (n = ) GRPH /SCTTERPLOT(BIVR)=sresd WITH pred... Studentzed Resdual Unstandardzed Predcted Value o The band of resduals s constant across the entre length of the observed predcted values.8. Example #: heteroscedastc model (n = ) GRPH /SCTTERPLOT(BIVR)=sresd WITH pred. 4.. Studentzed Resdual Unstandardzed Predcted Value o Ths pattern where the varance ncreases as Y ncreases s a common form of heteroscedastcty Karpnsk

12 o In ths case, the unequal heteroscedastcty s also apparent from the X-Y scatterplot. But n general, volatons of the varance assumpton are easer to spot n the resdual plots GRPH /SCTTERPLOT(BIVR)=y WITH x.. 5. y x 5.. o s n the case of lookng for non-lnearty, examnng the regresson model provdes no clues that the model assumptons have been volated REGRESSION /STTISTICS COEFF OUTS R NOV ZPP /DEPENDENT y /METHOD=ENTER x. Model Summary Model R R Square djusted R Square Std. Error of the Estmate.33 a a. Predctors: (Constant), X Model (Constant) X a. Dependent Varable: Y Unstandardzed Coeffcents Standard zed Coeffcen ts Coeffcents a Correlatons t Sg. Zero-order Partal Part B Std. Error Beta Karpnsk

13 4. Resdual plots to detect non-normalty s for NOV, symmetry s more mportant than normalty There are a number of technques that we can use to check normalty of the resduals. In general, these are the same technques we used to check normalty n NOV o Boxplots or hstograms of resduals o normal P-P plot of the resduals o Coeffcents of skewness/kurtoss may also be used Normalty s dffcult to check and can be nfluenced by other volatons of assumptons. good strategy s to check and address all other assumptons frst, and then turn to checkng normalty These tests are not foolproof o Techncally, we assume that the resduals are normally dstrbuted at each level of the predctor varable(s) o It s possble (but unlkely) that the dstrbuton of resduals mght be leftskewed for some values of X and rght skewed for other values so that, on average, the resduals appear normal. o If you are concerned about ths possblty and f you have a very large sample, you could dvde the Xs nto a equal categores, and check normalty separately for each of the a subsamples (you would want at least 3-5 observatons per group). In general, ths s not necessary. Example #: Normally dstrbuted resduals (N = ) EXMINE VRIBLES=sresd /PLOT BOXPLOT HISTOGRM NPPLOT. Studentzed Resdual Mean 5% Trmmed Mean Medan Varance Std. Devaton Mnmum Maxmum Range Interquartle Range Skewness Kurtoss Descrptves Statstc Std. Error o The mean s approxmately equal to the medan o The coeffcents of skewness and kurtoss are relatvely small 4-3. Karpnsk

14 3 - - Tests of Normalty -3 N = Studentzed Resdual Studentzed Resdual Shapro-Wlk Statstc df Sg o Plots can also be obtaned drectly from the regresson command REGRESSION /DEPENDENT y /METHOD=ENTER z /RESIDULS HIST(SRESID) NORM(SRESID) /SVE sresid (sresd). 6 Hstogram Dependent Varable: Y. Normal P-P Plot of Regresson Studentzed Resdual Regresson Studentzed Resdual Observed Cum Prob o The hstogram and P-P plot are as good as they get. There are no problems wth the normalty assumpton Karpnsk

15 Example #: Non-normally dstrbuted resduals (N = ) REGRESSION /DEPENDENT y /METHOD=ENTER z /RESIDULS HIST(ZRESID) NORM(ZRESID) /SVE ZRESID (zresd). Hstogram Dependent Varable: Y. Normal P-P Plot of Regresson Standardzed Resdual Regresson Standardzed Resdual Observed Cum Prob EXMINE VRIBLES=zresd /PLOT BOXPLOT HISTOGRM NPPLOT /STTISTICS DESCRIPTIVES. Descrptves Standardzed Resdual Mean 5% Trmmed Mean Medan Varance Std. Devaton Mnmum Maxmum Range Interquartle Range Skewness Kurtoss Statstc Std. Error Tests of Normalty Standardzed Resdual Shapro-Wlk Statstc df Sg o ll sgns pont to non-normal, non-symmetrcal resduals. There s a volaton of the normalty assumpton n ths case Karpnsk

16 5. Identfyng outlers and nfluental observatons Observatons wth large resduals are called outlers But remember, when the resduals are normally dstrbuted, we expect a small percentage of resduals to be large We expect 5% of e s greater than We expect % of e s greater than.5 We expect.% of e s greater than 3 Expected number of resduals # of observatons > >.5 > o Many people use e > as a check for outlers, but ths crteron results n too many observatons beng dentfed as outlers. In large samples, we expect a large num ber of observatons to have resduals greater than o more reasonable cut-off for outlers s to use >.5 or even e > 3 e 4-6. Karpnsk

17 There are multple knds of outlers.5 #.4.3 #.. #3 # o # s an Y outler o # s an X outler o #3 and #4 are outlers for both X and Y When we examne extreme observatons, we want to know: o Is t an outler? (.e., Does t dffer from the rest of the observed data?) o Is t an nfluental observaton? (.e., Does t have an mpact on the regresson equaton?) Clearly, each of the values hghlghted on the graph s an outler, but how wll each nfluence estmaton of the regresson lne? o Outler # Influence on the ftted values: Influence on the slope: o Outler # Influence on the ftted values: Influence on the slope: o Outler #3 Influence on the ftted values: Influence on the slope: o Outler #4 Influence on the ftted values: Influence on the slope: 4-7. Karpnsk

18 Not all outlers are equally nfluental. It s not enough to dentfy outlers; we must also consder the nfluence each may have (partcularly on the estmaton of the slope) Methods of dentfyng outlers and nfluental ponts: o Examnaton of the studentzed resduals o scatterplot of studentzed resduals wth X o Examnaton of the studentzed deleton resduals o Examnaton of leverage values o Examnaton of Cook s dstance (Cook s D) o Examnaton of DFBETs Detectng extremty on the Y values: Studentzed Deleton Resduals o deleton resdua l s the dfference between the observed Y and the predcted Y ˆ based on a model wth the th () value observaton deleted d = Y Y ˆ ( ) o The deleton resdual s a measure of how much the th observaton nfluences the overall regresson equaton (that s, the ftted values) o If the th observaton has no nfluence on the regresson lne then Y = Y ˆ ( ) and d = o The greater the nfluence of the observaton, the greater the deleton resdual o Note that we cannot determne f the observaton nfluences the estmaton of the ntercept or of the slope. We can only tell that t has an nfluence on at least one of the parameters n the regresson equaton. o The sze of the deleton resduals wll be determned, n part, by the scale of the Y values. In order to create deleton resduals that do not depend on the scale of Y, we can d vde d by ts standard devaton to obtan a studentzed deleton resdual d Y Y ˆ = ( ) s(d ) o Studentzed deleton resduals can be nterpreted lke z-scores (or more precsely, lke t-scores) 4-8. Karpnsk

19 Detectng extremty on the X values: Leverage values o It can be shown (proof omtted) that the predcted value for the th observaton can be wrtten as a lnear combnaton of the observed Y values Y ˆ = hy + hy hy h n Y n Where h,h,...,h n are known as leverage values or leverage weghts h n o The leverage values are computed by only usng the X value(s). o Because the h s are computed by only usng the X value(s), h measures the role of the X value(s) n determnng how mportant Y s n affectng Y ˆ. j o Thus, leverage values are helpful n dentfyng outlyng X observatons that nfluence Y ˆ o To dentfy large leverage values, we compare h to the average leverage value. The standard rule of thumb s f the h s -3 tmes as large as the th average leverage value, then X observaton(s) for the partcpant should be examned The average leverage value s: h = p n Where p = the number of parameters n the regresson model ( for smple lnear regresson) n = the number of partcpants nd so the rule-of -thumb cutoff value s: h > p n (large samples) or 3p h > (small samples) n o Other common cut-off values nclude h >.5 (large leverage); h <. 5 (moderate leverage) < Look for a large gap n the dstrbuton of h s 4-9. Karpnsk

20 Detectng nfluence on the ftted values: Cook s Dstance (979) o Cook s D s another measure of the nfluence an outlyng observaton has on the regresson coeffcents. It combnes resduals and leverage values nto a sngle number. e h D = p * MSE ( h ) Where: e s the (unstandardzed) resdual for the th observaton p s the number of parameters n the regresson model h s the leverage for the th observaton o D for each observaton depends on two factors: The resdual: Larger resduals lead to larger D s The leverage: Larger leverage values lead to larger D s o The th observato n can be nfluental (have a large D ) by Havng a large e and only a moderate h Havng a moderate e and a large h Havng a large e and a large h o D s consdered to be large (ndcatng an nfluental observaton) f t falls at or above the 5 th percentle of the F-dstrbuton F crt ( α =. 5, dfn, dfe) dfn = # of parameters n the model = p ( for smple lnear regresson) dfe = degrees of freedom for error = N p For example, wth a smple lnear regresson model (p = ) wth 45 observatons (dfe = 45-=43) = F( α =.5, dfn, dfe) = F( α =.5,,43) =.74 D crt In ths case, observatons wth Cook s D values greater than.74 should be nvestgated as possbly beng nfluental 4-. Karpnsk

21 Detectng nfluence slope parameters: DFBETs o DFBET s a senstvty analyss of an ndvdual observatons nfluence on the slope parameter. For each observaton, Y bˆ + bˆ X + ε Y = ( ) = bˆ ( ) + bˆ ( ) X ( ) ft two regresson models: wth all observatons + ε ( ) wth observaton removed BFDET s the studentzed dfference n the slope estmates bˆ bˆ ( ) DFBET = SE bˆ ) ( ( ) o By studentzng the DFBET, they can be nterpreted lke z-scores (or more precsely, lke t-scores). DFBET > crt o When run n SPSS, you wll obtan a DFBET for each parameter n the model (a DFBET for the ntercept and a DFBET for the slope). In general, we would only be nterested n the DFBET for the slope These methods of outlers and nfluence often work well, but can be neffectve at tmes. Ideally, the dfferent procedures would dentfy the same cases, but ths does not always happen. The use of these procedures requres thought and good judgment on the part of the analyst. Once nfluental ponts are dentfed: o Check to make sure there has not been a data codng or data entry error. o Conduct a senstvty analyss to see how much your conclusons would change f the outlyng ponts were dropped. o Never drop data ponts wthout tellng your audence why those observatons were omtted. In general, t s not advsable to drop observatons from your analyss o The presence of many outlers may ndcate an mproper model Perhaps the relatonshp s not lnear Perhaps the outlers are due to a varable omtted from the model 4-. Karpnsk

22 al u esd ed R udentz St.5 #.4.3 #.. #3 # Baselne example: No outlers ncluded REGRESSION /STTISTICS COEFF OUTS R NOV ZPP /DEPENDENT y /METHOD=ENTER x /CSEWISE PLOT(SRESID) LL /SVE PRED (pred) SRESID (sresd). o The regresson model Y = X r XY =. 876 R =. 767 o Examnng resduals Casewse Dagnostcs a Case Number Stud. Resdual Y Predcted Value Resdual a. Dependent Varable: Y x Look for Studentzed Resduals larger than Karpnsk

23 o Examnng nfluence statstcs REGRESSION /DEPENDENT y /METHOD=ENTER x /SVE COOK (cook) LEVER (level) SDRESID (sdresd) SDBET. Lst var = ID cook level sdresd sdb_. d cook level sdresd SDB_ o Crtcal values Cook s D: D crt = F( α =.5,,3) =.74 p 4 Leverage: h crt > = =. 6 N 5 ~ Studentzed Deleton Resduals: d crt >. 5 Studentzed DFBET: DFBET crt > o In ths case, we do not dentfy any outlers or nfluental observatons 4-3. Karpnsk

24 Stud entze d Re sdu al Example #: Outler # ncluded REGRESSION /STTISTICS COEFF OUTS R NOV ZPP /DEPENDENT y /METHOD=ENTER x /CSEWISE PLOT(SRESID) LL /SVE PRED (pred) SRESID (sresd). o The regresson model Y = X r XY =. 89 R =. 654 (Slope s unchanged) o Examnng resduals Casewse Dagnostcs a Case Number a. Dependent Varable: Y Stud. Predcted Resdual Y Value Resdual x Look for Studentzed Resduals larger than.5 Observaton #6 looks problematc 4-4. Karpnsk

25 o Examnng nfluence statstcs REGRESSION /DEPENDENT y /METHOD=ENTER x /SVE COOK (cook) LEVER (level) SDRESID (sdresd) SDBET. Lst var = ID cook level sdresd sdb_. d cook level sdresd SDB_ o Crtcal values Cook s D: D crt = F( α =.5,,4) =.695 p 4 Leverage: h crt > = =. 54 N 6 ~ Studentzed Deleton Resduals: d crt >. 5 Studentzed DFBET: DFBET crt > o Observaton #6 Has large resdual and deleton resdual Has OK Cook s D, leverage, and DFBET Karpnsk

26 S tud entz ed Re sd ual Example #: Only outler # ncluded REGRESSION /STTISTICS COEFF OUTS R NOV ZPP /DEPENDENT y /METHOD=ENTER x /CSEWISE PLOT(SRESID) LL /SVE PRED (pred) SRESID (sresd). o The regresson model Y = X r =. 76 R =. 58 XY o Examnng resduals x Look for Studentzed Resduals larger than.5 Observaton #7 looks problematc 4-6. Karpnsk

27 o Examnng nfluence statstcs REGRESSION /DEPENDENT y /METHOD=ENTER x /SVE COOK (cook) LEVER (level) SDRESID (sdresd) SDBET. Lst var = ID cook level sdresd sdb_. d cook level sdresd SDB_ o Crtcal values Cook s D: D crt = F( α =.5,,4) =.695 p 4 Leverage: h crt > = =. 54 N 6 ~ Studentzed Deleton Resduals: d crt >. 5 Studentzed DFBET: DFBET crt > o Observaton #7 Has large resdual, deleton resdual, Cook s D, leverage, and DFBET Karpnsk

28 Stu de nt ze d R es du al Example #3: Only outler #3 ncluded REGRESSION /STTISTICS COEFF OUTS R NOV ZPP /DEPENDENT y /METHOD=ENTER x /CSEWISE PLOT(SRESID) LL /SVE PRED (pred) SRESID (sresd). o The regresson model Y = X r XY =. 9 R =. 8 (No change n slope or ntercept) o Examnng resduals x Look for Studentzed Resduals larger than.5 ll observatons are OK 4-8. Karpnsk

29 o Examnng nfluence statstcs REGRESSION /DEPENDENT y /METHOD=ENTER x /SVE COOK (cook) LEVER (level) SDRESID (sdresd) SDBET. Lst var = ID cook level sdresd. d cook level sdresd SDB_ o Crtcal values Cook s D: D crt = F( α =.5,,4) =.695 p 4 Leverage: h crt > = =. 54 N 6 ~ Studentzed Deleton Resduals: d crt >. 5 Studentzed DFBET: DFBET > o Observaton #8 Has large leverage Has OK resdual, deleton resdual, Cook s D, and DFBET crt 4-9. Karpnsk

30 Stu de nt ze d R es du al Example #4: Only outler #4 ncluded REGRESSION /STTISTICS COEFF OUTS R NOV ZPP /DEPENDENT y /METHOD=ENTER x /CSEWISE PLOT(SRESID) LL /SVE PRED (pred) SRESID (sresd). o The regresson model Y =. 4 +.X r =. 4 R =. 6 o Examnng resduals XY x Look for Studentzed Resduals larger than.5 Observaton #9 s clearly problematc 4-3. Karpnsk

31 o Examnng nfluence statstcs REGRESSION /DEPENDENT y /METHOD=ENTER x /SVE COOK (cook) LEVER (level) SDRESID (sdresd) SDBET. Lst var = ID cook level sdresd sdb_. d cook level sdresd SDB_ o Crtcal values Cook s D: D crt = F( α =.5,,4) =.695 p 4 Leverage: h crt > = =. 54 N 6 ~ Studentzed Deleton Resduals: d crt >.5 Studentzed DFBET: DFBET > o Observaton #9 Has large resdual, deleton resdual, Cook s D, leverage, and DFBET crt 4-3. Karpnsk

32 Summary and comparson:.5 #.4.3 #.. #3 # Obs Problematc? Regresson Equaton r XY e~ d D h DFB Baselne Y = X rxy =. 876 No No No No No # Y = X rxy =. 89 Yes Yes No No No # Y = X rxy =. 76 Yes Yes Yes Yes Yes #3 Y = X rxy =. 9 No No No Yes No #4 Y = X r =. 43 Yes Yes Yes Yes Yes XY o Usng a combnaton of all the methods, we (properly) dentfy outlers # and #4 as problematc. Outler # may or may not be problematc, dependng on our purposes Karpnsk

33 6. Remedal Measures: n overvew of alternatve regresson models When regresson assumptons are volated, you have two optons o Explore transformatons of X or Y so that the smple lnear regresson model can be used approprately o bandon the smple lnear regresson model and use a more approprate model. More complex regresson models are beyond the scope of ths course, but let me hghlght some possble alternatve models that could be explored. Polynomal regresson o When the regresson functon s not lnear, a model that has non-lnear terms may better ft the data. Y = β + β X + β + ε X Y 3 = β + β X + β X + β X β k X 3 k + ε o In these models, we are fttng/estmatng non-lnear regresson lnes that correspond wth polynomal curves. Ths approach s very s mlar to the trend contrasts that we conducted n NOV except: For polynomal regresson, the predctor varable (IV) s contnuous; n NOV t s categorcal. In polynomal regresson, we obtan the actual equaton of the (polynomal) lne that best fts the data. Weghted least squares regresson o The regresson queston we have been usng s known as ordnary least squares (OLS) regresson. In the OLS framework, we solve for the model parameters by mnmzng the squared resduals (squared devatons from the predcted lne) Karpnsk

34 SSE ( ˆ = e = Y Y ) = ( Y b b X o When we mnmze the resduals, solve for the parameters, and compute p-values, we need the resduals to have equal varance across the range of X values. If ths equal varance assumpton s volated, then the OLS regresson parameters wll be based. o In OLS regresson, each observaton s treated equally. But f some observatons are more precse than others (.e., they have smaller varance), t make sense that they should be gven more weght than the less precse values. o In weghted least squares regresson, each observaton s weghted by the nverse of ts varance ) w = σ SSE ( ˆ = we = w Y Y ) = w ( Y b b X Observatons wth a large varance are gven a small weght; observatons wth a small varance are gven a bg weght. o Issues wth weghted least squares regresson We do not know the varances of the resduals; ths value must be estmated. The process of estmatng these varances s not trval partcularly n small datasets. R s unnterpretable for weghted least squares regresson (but that does not stop most programs from prntng t out!). ) Karpnsk

35 Robust regresson o When assumptons are volated and/or outlers are present, OLS regresson parameters may be based. Robust regresson s a seres of approaches that computes estmates of regresson parameters usng technques that are robust to volatons of OLS assumptons o Least absolute resdual (LR) regresson estmates regresson parameters by mnmzng the sum of the absolute devatons of the Y observatons from the regresson lne: Y Yˆ = Y b b X e = Ths approach reduces the nfluence of outlers o Least medan of squares (LMS) regresson estmates regresson parameters by mnmzng the medan of the squared devatons of the Y observatons from the regresson lne: SSE = Medan( e ) = medan( Y Yˆ) = medan( Y b b X ) Ths approach also reduces the nfluence of outlers o Iteratve reweghed least squares (IRLS) regresson s a form of weghted least squares regresson where the weghts for each observaton are M- estmators of the resduals (Huber and Tukey-Bsquare estmators are the most commonly used M-estmators). Ths approach reduces the nfluence of outlers o Dsadvantages of these approaches nclude: They are not commonly ncluded n statstcal software They are robust to outlers, but they requre that other regresson assumptons be satsfed. They are not commonly used n psychology and, thus, psychologsts may regard these methods skeptcally Karpnsk

36 Non-parametrc regresson o Non-parametrc regresson technques do not estmate model parameters. For a regresson analyss, we are usually nterested n estmatng a slope, thus non-parametrc methods are of lmted utlty from an nferental perspectve. o In general, non-parametrc technques can be used to explore the shape of the regresson lne. These methods provde a smoothed curve that fts the data, but do not provde an equaton for ths lne or allow for nference on ths lne. o Prevously, we examned the followng data and concluded that the relatonshp between X and Y was non-lnear (see p. 5-8)..8.6 y x Let s examne ths data wth non-parametrc technques to explore the relatonshp between X and Y. Method of movng averages o The method of movng averages can be used to obtan a smoothed curve that fts the data. o To use ths method, you must specfy a wndow (w) the number of observatons you wll average across. Frst, average the Y values assocated wth the frst w responses (the w smallest X values), and plot that pont. Next, you dscard the frst value, add the next pont along the X axs, compute the average of ths set of w Y values, and plot that pont. Contnue movng the down the X axs, untl you have used all the ponts. Then draw a lne to connect the smoothed average ponts Karpnsk

37 o For example, f w = 3, then you take the three smallest X values, average the Y values assocated wth these ponts, and plot that pont. Next, take the nd, 3 rd, and 4 th smallest X values and repeat the process... o n example of the method of movng averages, comparng dfferent w values: Smoothng nterval = 5 Smoothng nterval = Y.5 Y X X Smoothng nterval = 5 Smoothng nterval = Y.5 Y X X If the wndow s too small, t does not smooth the data enough; f the wndow s too large, t can smooth too much and you lose the shape of the data. In ths case, w = 5 looks about rght Karpnsk

38 o Issues regardng the method of movng averages: verages are affected by outlers, so t can be preferable to use the method of movng medans The method of movng averages s partcularly useful for tme-seres data If the data are unequally spaced and/or have gaps along the X axs, the method of movng averages can provde some wacky results. In EXCEL, you can use the method of movng averages and you can specfy w. Loess smoothng o Loess stands for locally weghted scatterplot smoothng (the w got dropped somewhere along the way). o Loess smoothng s a more sophstcated method of smoothng than the method of movng averages. In each neghborhood, a regresson lne s estmated and the ftted lne s used for the smoothed lne Ths regresson s weghted to gve cases further from the mddle X value less weght. Ths process of fttng a lnear regresson lne s repeated n each neghborhood so that observatons wth large resduals n the prevous teraton receve less weght. Loess Smoothng..8.6 y x Karpnsk

39 o One partcularly useful applcaton of loess smoothng s to confrm a ftted regresson functon Ft a regresson functon and graph 95% confdence bands for the ftted lne Ft a loess smoothed curve through the data. If the loess curve stays wthn the confdence bands, the ft of the regresson lne s good. If the loess curve strays from the confdence bands, the ft of the regresson lne s not good y y x x Good Ft Poor Ft 7. Remedal Measures: Transformatons If the data do not satsfy the regresson assumptons, a transformaton appled to ether the X varable or to the Y varable may make the smple lnear regresson model approprate for the transformed data. General rules of thumb: o Transformatons on X Can be used to lnearze a non-lnear relatonshp o Transformatons on Y Can be used to fx problems of nonnormalty and unequal error varances Once normalty and homoscedastcty are acheved, t may be necessary to transform X to acheve lnearty Karpnsk

40 Prototypcal patterns and transformatons of X Non Lnear Relatonshp # o Try X = ln(x) or X = X Ln Transformaton Non Lnear Relatonshp # o Try X = X or X = exp(x) Non Lnear Relatonshp #3 X Transformaton /X Transformaton ( ) o Try X =/X or X = exp X Karpnsk

41 Transformng Y o If the data are non-normal and/or heteroscedastctc, a transformaton on Y may be useful o It can be very dffcult to determne the most approprate transformaton n Y to fx the data o One popular class of transformatons s the famly of power transformatons Y = Y λ λ = Y = Y λ = Y = Y λ =.5 Y = Y λ = Y = ln(y) by defnton λ =.5 Y = Y λ = λ = Y = Y Y = Y To determne a λ that works: Guess (tral and error) Use the Box-Cox procedure (unfortunately not mplemented n SPSS) Warnngs and cauton s about transformatons o Do not transform the data because of a small number of outlers o fter transformng the data, recheck the ft of the regresson model usng resdual analyss o Once the data are transformed, and a regresson run on the transformed data, b and b apply to the transformed data and not to the orgnal data/scale o For psychologcal data, f the orgnal data are not lnear, but the transformed data are, t can often be very dffcult to nterpret the results 4-4. Karpnsk

42 Transformatons: n example o Let s examne the relatonshp between the number of credts taken n a mnor and nterest n takng further coursework n that dscplne. o unversty collects data on students X = Number of credts completed n the mnor Y = Interest n takng another course n the mnor dscplne o Frst, let s plot the data OLS Regresson Lne 5. Interest n further coursework Loess Curve 5 Number of credts Ths relatonshp looks non-lnear. We can try a square root or a log transformaton to acheve lnearty Interest n furt her course work. 5.. ework Inte rest n further cours sqrtx o The log transformaton appears to work, so we should check the remanng assumptons lnx Karpnsk

43 REGRESSION /DEPENDENT ybr /METHOD=ENTER lnx /RESIDULS HIST(SRESID) NORM(SRESID) /SVE RESID (resd) ZRESID (zresd) SRESID (sresd) pred (pred). Hstogram Normal P-P Plot of Regresson Studentzed Resdual Dependent Varable: Interest n further coursework Dependent Varable: Interest n further coursework. 4.8 Frequency Expected Cum Prob Regresson Studentzed Resdual 3 Mean = -.3E-4 Std. Dev. =.5 N = Observed Cum Prob.8. The resduals appear to be normally dstrbuted 4.. Student zed R esdual Unstandardzed Predcted Value 5. We mght worry about an outler, but homoscedastcty seems ok Karpnsk

44 o Now, we can analyze the ln-transformed data. Model Summary b Model djusted Std. Error of R R Square R Square the Estmate.83 a a. Predctors: (Constant), lnx b. Dependent Varable: Interest n further coursework Model (Constant) lnx Unstandardzed Coeffcents Coeffcents a Standardzed Coeffcents B Std. Error Beta t Sg a. Dependent Varable: Interest n further coursework There s a strong lnear relatonshp between ln of credts taken and nterest n takng addtonal courses n the dscplne, β.8, t(98) = 3.83, p <., R =.66 = djusted o But n ths case, the non-lnear relatonshp s nterestng (and nterpretable). We would be better off wth an approach where we could model the non-lnearty than wth ths approach where we try to transform to lnearty. o In ths case, polynomal regresson may be very useful. o Note that f we had not graphed or explored our data, we would have mssed the non-lnear relatonshp altogether! Model Summary Model R R Square djusted R Square Std. Error of the Estmate.749 a a. Predctors: (Constant), Number of credts Model (Constant) Number of credts Coeffcents a Unstandardzed Coeffcents a. Dependent Varable: Interest n further coursework Standardzed Coeffcents B Std. Error Beta t Sg Karpnsk