MULTIPLE LINEAR REGRESSION IN MINITAB

Transcription

1 MULTIPLE LINEAR REGRESSION IN MINITAB Ths document shows a complcated Mntab multple regresson. It ncludes descrptons of the Mntab commands, and the Mntab output s heavly annotated. Comments n { } are used to tell how the output was created. The comments wll also cover some nterpretatons. Letters n square brackets, such as [a], dentfy endnotes whch wll gve detals of the calculatons and explanatons. The endnotes begn on 9. Output from Mntab sometmes wll be edted to reduce empty space or to mprove layout. Ths document was prepared wth Mntab 4. The data set used here can be found at the Web ste open the Other Data Sets folder M. The fle name s SWISS.MTP, and t can be found on the Stern Web ste as well. The data set concerns fertlty rates n 47 Swss cantons (provnces) n the year 888. The dependent varable wll be Fert, the fertlty rate, and all the other varables wll functon as ndependent varables. The data are found n Data Analyss and Regresson, by Mosteller and Tukey, s Ths document was prepared by the Statstcs Group of the I.O.M.S. Department. If you fnd ths document to be helpful, we d lke to know! If you have comments that mght mprove ths presentaton, please let us know also. Please send e-mal to gsmon@stern.nyu.edu. Revson date 4 NOV 005

2 {Data was brought nto the program through Fle Open Worksheet. Mntab s default for Fles of type: s (*.mtw; *.mpj), so you wll want to change ths to *.mtp to obtan the fle. On the Stern network, ths fle s n the folder X:\SOR\B0305\M, and the fle name s SWISS.MTP. The lstng below shows the data set, as coped drectly from Mntab s data wndow.} Fert Ag Army Ed Catholc Mort 0.80[a]

3 {The tem below s Mntab s Project Manager wndow. You can get ths to appear by clckng on the con on the toolbar.} [b] {The followng secton gves basc statstcal facts. It s obtaned by Stat Basc Statstcs Dsplay Descrptve Statstcs. All varables were requested. The request can be done by lstng each varable by name (Fert Ag Army Ed Catholc Mort) or by lstng the column numbers (C-C6) or by clckng on the names n the varable lstng.} Descrptve Statstcs: Fert, Ag, Army, Ed, Catholc, Mort [c][d] [e] [f] Varable N N* Mean SE Mean StDev Mnmum Q Medan Q3 Fert Ag Army Ed Catholc Mort Varable Maxmum Fert Ag Army Ed Catholc Mort {The next lstng shows the correlatons. It s obtaned through Stat Basc Statstcs Correlaton and then lstng all the varable names. For now, we have de-selected the feature Dsplay p-values.} 3

4 Correlatons: Fert, Ag, Army, Ed, Catholc, Mort Fert Ag Army Ed Catholc Ag Army [g] Ed Catholc Mort Cell Contents: Pearson correlaton {The lnear regresson of dependent varable Fert on the ndependent varables can be started through Stat Regresson Regresson Set up the panel to look lke ths: Observe that Fert was selected as the dependent varable (response) and all the others were used as ndependent varables (predctors). If you clck OK you wll see the basc regresson results. For the sake of llustraton, we ll show some addtonal features. Clck the Optons button and then select Varance nflaton factors. The choce Ft ntercept s the default and should already be selected; f t s not, please select t. The Ft ntercept opton should be de-selected only n extremely specal stuatons. We recommend that you routnely examne the varance nflaton factors f strong collnearty s suspected. The Durbn-Watson statstc was not used here because the data are not tmesequenced. 4

5 Clck the Graphs button and select the ndcated choces: Examnng the Resduals versus fts plot s now part of routne statstcal practce. The other selectons can show some nterestng clues as well. Here we wll use the Four n one opton, as t shows the resdual versus ftted plot, along wth the other three as well. The Resduals versus order plot wll not be useful, because the data are not tme-ordered. Some of the choces made here reflect features of ths data set or partcular desres of the analyst. Here the Regular form of the resduals was desred; other choces would be just as reasonable. Clck the Storage button and select H (leverages). Ths provdes a very thorough regresson job. } {The model correspondng to ths request s Fert = β 0 + β AG Ag + β Army Army + β ED ED + β CATH CATH + β MORT MORT + ε } 5

6 Regresson Analyss: Fert versus Ag, Army, Ed, Catholc, Mort The regresson equaton s [h] Fert = Ag Army Ed Catholc +.08 Mort Predctor Coef SE Coef T P VIF Constant[] 0.669[j] 0.07[k] 6.5[l] 0.000[m] [n] Ag -0.7[ø] [p] Army [q] 0.35[r] 3.7 Ed Catholc Mort S = [s] R-Sq = 70.7%[t] R-Sq(adj) = 67.% [u] Analyss of Varance [v] Source DF[w] SS[aa] MS[ee] F[] P[jj] Regresson 5[x] [bb] 0.046[ff] Resdual Error 4[y] 0.050[cc] [gg] Total 46[z] [dd] [hh] Source DF Seq SS[kk] Ag Army 0.04 Ed Catholc Mort Unusual Observatons[ll] Obs Ag Fert Ft SE Ft Resdual St Resd 6[mm] [nn] 0.039[øø]-0.440[pp] -.4R [qq] R X[rr] R R denotes an observaton wth a large standardzed resdual X denotes an observaton whose X value gves t large nfluence. {Many graphs were requested n ths run. The Four n one panel examnes the behavor of the resduals because they provde clues as to the approprateness of the assumptons made on the ε terms n the model. The most mportant of these s the resduals versus ftted plot, the plot at the upper rght on the next. The normal probablty plot and the hstogram of the resduals are used to assess whether or not the nose terms are approxmately normally dstrbuted. Snce the data ponts are not tme-ordered, we wll not use the plot of the resduals versus the order of the data.} 6

7 Resdual Plots for Fert 99 Normal Probablty Plot of the Resduals Resduals Versus the Ftted Values Percent 50 Resdual Resdual Ftted Value Hstogram of the Resduals Resduals Versus the Order of the Data Frequency Resdual Resdual Observaton Order [ss] {Many users choose also to examne the plots of the resduals aganst each of the predctor varables. These were requested for ths run, but ths document wll show only the plot of the resduals aganst the varable Mort.} 0.5 Resduals Versus Mort (response s Fert) Resdual Mort [tt] 7

8 {Fnally, recall that we had requested the hgh leverage ponts through Stat Regresson Regresson Storage and then selectng H (leverages). These wll show up n a new column, called HI, n the data wndow. Ths column can be used n plots, or t can smply be examned. What shows below s that column, coped out of the data wndow, and restacked to save space.} Case HI Case HI Case HI [uu] [vv] {There s a commonly-used threshold of concern, as dscussed n [uu]. Mntab wll automatcally mark ponts that exceed ths threshold; see [ll] and [rr]. It s therefore not crtcal that the leverage, or H, values be computed.} 8

9 ENDNOTES: [a] Ths s the frst lne of the data lstng. The lne numbers ( through 47) are not shown here, although they do appear n the Mntab data wndow. The numbers across ths row ndcate that ths frst canton had Fert = 0.80, Ag = 0.7, Army = 0.5, and so on. [b] Mntab s Project Manager wndow shows the varable names for the columns, and also some basc accountng. We see that each varable has 47 values, wth none mssng. Mntab data sets can also have Constants and Matrces, although ths set has none. Descrptons are saved only wth project (*.MPJ) fles. [c] The symbol N refers to the sample sze, after removng mssng data. In ths data set, there are 47 cantons and all nformaton s complete. We have 47 peces of nformaton for each varable. In some data sets, the column of N-values mght lst several dfferent numbers. [d] N * s the number of mssng values. In ths set of data, all varables are complete. [e] Ths s the standard error of the mean. It s computed for each varable as SD N, where N s the number of non-mssng values. Here you can confrm that for varable Fert, [f] Mntab computes the quartles Q and Q3 by an nterpolaton method. If the sample sze s n, then Q s the observaton at rank poston (n+)/4. If (n+)/4 s not an nteger, then Q s obtaned as a weghted average of the values at the surroundng nteger postons. For nstance, f (n+)/4 = 6.75, then Q s 3 4 of the dstance between the 6 th and 7 th values. The procedure for fndng Q3 works from rank poston 3(n+)/4. [g] The value s the correlaton between Army and Ag. It s also the correlaton between Ag and Army. Snce correlatons are symmetrc, t s not necessary to prnt the entre correlaton matrx. The correlaton between a varable and tself s.000 (also not prnted). Generally we lke to see strong correlatons (say above +0.9 or below -0.9) nvolvng the dependent varable, here Fert. We prefer not to have strong correlatons among the other varables. [h] Ths s the estmated model or ftted equaton. Some people lke to place the hat on the dependent varable Fert as Fêrt to denote estmaton or fttng. Note that the numbers are repeated n the Coef column below. The letter b s used for estmated values; thus b 0 = 0.669, b Ag = -0.7, and so on. [] The term Constant refers to the ncluson of β 0 n the regresson model. Ths s sometmes called the ntercept. 9

10 [j] Ths s the estmated value of β 0 and s often called b 0. [k] Ths s estmated standard devaton of the value n the Coef column; ths s also called the standard error. In ths nstance, we beleve that the estmated value, 0.669, s good to wthn a standard error of We re about 95% confdent that the true value of β 0 s n the nterval ± (0.07), whch s the nterval (0.4550, ). Coef [l] The value of T, also called Student s t, s ; here that arthmetc s SE Coef Ths s the number of estmated standard devatons that the estmate, , s away from zero. The phrase estmated standard devatons refers to the dstrbuton of the sample coeffcent, and not to the standard devaton n the regresson model. Ths T can be regarded as a test of H 0 : β 0 = 0 versus H : β 0 0. Here T s outsde the nterval (-,+) and we should certanly beleve that β 0 (the true-but-unknown populaton value) s dfferent from zero. Some users beleve that the ntercept should not be subjected to a statstcal test; ndeed some software does not provde a T value or a P (see the next tem) for the Constant lne. [m] The column P (for p-value) s the result of subjectng the data to a statstcal test as to whether β 0 = 0 or β 0 0. There s a precse techncal defnton, but crudely P s the smallest Type I error probablty that you mght make n decdng that β 0 0. Very small values of P suggest that β 0 0, whle larger values ndcate that you should mantan β 0 = 0. The typcal cutoff between these actons s 0.05; thus P 0.05 causes you to decde that the populaton parameter, here β 0, s really dfferent from zero. The p-value s never exactly zero, but t sometmes prnts as The p-value s drectly related to T; values of T far outsde the nterval (-,+) lead to small P. When P 0.05, we say that the estmated value s statstcally sgnfcant. As an mportant sde note, many people beleve that the Constant β 0 should never be subjected to statstcal tests. Accordng to ths pont of vew, we should not even ask whether β 0 = 0 or β 0 0; ndeed we should not even lst T and P n the Constant lne. Ths sde note apples only to the Constant. [n] The VIF, varance nflaton factor, comes as the result of a specal request. The VIF does not apply to the Constant. See also tem [p]. 0

11 [ø] The value -0.7 s b AG, the estmated value for β AG. In fact, the Coef column can be used to wrte the ftted equaton [h]. Some people lke to wrte the standard errors n parentheses under the estmated coeffcents n the ftted equaton n ths fashon: Fêrt = Ag Army Ed (0.07) (0.070) (0.54) (0.83) Catholc +.08 Mort ( ) (0.38) You may also see ths knd of dsplay usng n parentheses the values of T, so ndcate for your readers exactly what you are dong. Ths s precsely the relatonshp that s used to determne the n = 47 ftted values: Fêrt = Ag Army Ed Catholc +.08 Mort The dfferences between the observed and ftted values are the resduals. The th resdual, usually denoted e, s Fert - Fêrt. [p] The VIF, varance nflaton factor, measures how much of the standard error, SE Coef, can be accounted for by nter-relaton of one ndependent varable wth all of the other ndependent varables. The VIF can never be less than. If some VIF values are large (say 0 or more), then you have a collnearty problem. Plausble solutons to the collnearty are Stepwse Regresson and Best Subsets Regresson. A related concept s the tolerance; these are related through Tolerance = VIF. [q] The value of T for the varable Army tests the hypothess H 0 : β Army = 0 versus H : β Army 0. Ths T s n the nterval (-,+), suggestng that H 0 s correct, so you mght consder repeatng the problem wthout usng Army n the model. [r] The P for varable Army exceeds Ths s consstent wth the prevous comment, and t suggests repeatng the problem wthout usng Army n the model. The comparson of P wth 0.05 s a more precse standard than comparng T wth.0. [s] Ths s one of the most mportant numbers n the regresson output. It s called standard error of estmate or standard error of regresson. It s the estmate of σ, the standard devaton of the nose terms (the ε s). A common notaton s s ε. In ths data set, that value comes out to It s useful to compare ths to 0.49, whch was the standard devaton of Fert on the Descrptve Statstcs lst. The orgnal nose level n Fert was 0.49 (wthout dong any regresson); the nose left over after the regresson was Item [s] s the square root of tem [gg], the resdual mean square, whose value s ; observe that

12 [t] Ths s the heavly-cted R ; t s generally gven as a percent, here 70.7%, but t mght also be gven as decmal The formal statement used s The percent of the varaton n Fert that s explaned by the regresson s 70.7%. Techncally, R s the rato of two sums of squares; t s the rato of tem [bb], the regresson sum of squares, to tem [dd], the total sum of squares. Observe that = 70.67%. Large values of R are consdered to be good. [u] Ths s an adjustment made to R to account for the sample sze and the number of n ndependent varables and s gven by Radj = ( R ). In ths formula, n s n k the number of ponts (here 47) and k s the number of predctors used (here 5). Thus R s ε has the neat nterpretaton n [t], and we have Radj =. Here s ε s and sfert s Fert = 0.49 and gven n [s]. Most users prefer R over R. [v] Ths s the analyss of varance table. The work s based on the algebrac dentty n n n ( y ) ( ˆ ) ( ˆ y = y y + y y) = = = n whch y denotes the value of the dependent varable for pont, and $y denotes the ftted value for pont. Snce Fert s the dependent varable, we dentfy $y wth wth Fêrt as n [h] and [ø]. The three sums of squares n ths equaton are, respectvely, SS total, SS regresson, and SS resdual error. These have other names or abbrevatons. For nstance SS total s often wrtten as SS tot. SS regresson s often wrtten as SS reg and sometmes as SS ft or SS model. SS resdual error s often wrtten as SS resdual or SS resd or SS res or SS error or SS err. [w] The DF stands for degrees of freedom. Ths s an accountng of the dmensons of the problem, and the numbers n ths column add up to the ndcated total as = 46. See the next three notes. [x] The Regresson lne n the analyss of varance table refers to the sum adj n = ( yˆ y) whch appeared n [v]. The degrees of freedom for ths calculaton s k, the number of ndependent varables. Here k = 5.

13 [y] The Resdual Error lne n the analyss of varance table refers to the sum n = ( y y$ ) whch appeared n [v]. The degrees of freedom for ths calculaton s n - - k, where n s the number of data ponts and k s the number of ndependent varables. Here n = 47 and k = 5, so that = 4 appears n ths poston. [z] The Total lne n the analyss of varance table refers to the sum n = ( y y) whch appears n [v]. The degrees of freedom for ths calculaton s n -, where n s the number of data ponts. Here n = 47, so that 46 appears n ths poston. [aa] The SS stands for sum of squares. Ths column gves the numbers correspondng to the dentty descrbed n [v]. The values n ths column add up to the ndcated total as = n [bb] Ths s the sum ( yˆ y) whch appeared n [v]. Ths s SS regresson. = [cc] Ths s the sum ( y y$ ) [dd] Ths s the sum n = n = ( y y) whch appeared n [v]. Ths s SS resdual error. whch appears n [v]. Ths s SS total, the total sum of squares. It nvolves only the dependent varable, so t says nothng about the regresson. The regresson s successful f the regresson sum of squares s large relatve to the resdual error sum of squares. The F statstc, tem [], s the approprate measure of success. [ee] The MS stands for Mean Squares. Each value s obtaned by dvdng the correspondng Sum Squares by ts Degrees of Freedom. [ff] Ths s MS regresson = In a successful regresson, ths s large relatve to tem [gg], the resdual mean square. [gg] Ths s MS resdual error = [hh] There s no mean square entry n ths poston. The computaton SS total (n - ) would nonetheless be useful, snce t s the sample varance of the dependent varable. [] Ths s the F statstc. It s computed as MS regresson MS resdual error, meanng To test at sgnfcance level α, ths s to be compared to F α kn, k, the upper α pont from the F dstrbuton wth k and n - degrees of freedom, obtaned from statstcal tables or from Mntab. The F statstc s a formal test of the null hypothess that all the ndependent varable coeffcents are zero aganst the alternatve that they are not. In our example, we would wrte 3

14 H 0 : β Ag = 0, β Army = 0, β Ed = 0, β Catholc = 0, β Mort = 0 H : at least one of β Ag, β Army, β Ed, β Catholc, β Mort s not zero If the F statstc s larger than F α kn, k, then the null hypothess s rejected. Otherwse, we accept H 0 (or reserve judgment). In ths nstance, usng sgnfcance level α = 0.05, we fnd F α = F 005. kn, k 54, =.4434 from a statstcal table. Snce 9.76 >.4434, we would reject H 0 at the 0.05 level of sgnfcance; we would descrbe ths regresson as statstcally sgnfcant. Here s how to use Mntab to get the values of F α. kn, k Calc Probablty Dstrbutons F Select Inverse cumulatve probablty, choose Numerator degrees of freedom: 5 Denomnator degrees of freedom: 4 Input constant: 0.95 Then clck OK. [jj] Ths s the p-value assocated wth the F statstc. Ths s the result of subjectng the data to a statstcal test of H 0 versus H n tem []. The p-value noted n tem [m] s not cleanly related to ths. [kk] The Seq SS column s constructed by fttng the predctor varables n the order gven and notng the change n SS regresson. Logcally, ths means here fve regressons: Fert on Ag has SS reg = Fert on Ag, Army has SS reg = = Fert on Ag, Army, Ed has SS reg = = Fert on Ag, Army, Ed, Catholc has SS reg = = Fert on Ag, Army, Ed, Catholc, Mort has SS reg = = The fnal value s SS reg for the whole regresson, tem [bb]. Namng the predctor varables n another order would produce dfferent results. Ths arthmetc can only be nterestng f the order of the varables s nterestng. In ths case, there s no reason to have any nterest n these values. 4

15 [ll] Mntab wll lst for you data ponts whch are unusual observatons and are worthy (perhaps) of specal attenton. There are two concerns, unusual resduals and hgh nfluence. Unfortunately, Mntab has too low a threshold of concern regardng the resduals, as t wll lst any standardzed resdual below - or above +. Vrtually every data set has ponts wth ths property, so that nothng unusual s nvolved. A more reasonable concern would be for resduals below -.5 or above +.5. Indeed, n large data sets, one mght move the thresholds of concern to -3 and +3. The (-, ) thresholds cannot be reset, so you wll have to lve wth the output. Extreme resduals occur wth ponts for whch the regresson model does not ft well. It s always worth examnng these ponts. It s generally not approprate to remove these ponts from the regresson. The geometrc confguraton of the predctor varables mght ndcate that some ponts unduly nfluence the regresson results. These ponts are sad to have hgh nfluence or hgh leverage. Whether such ponts should be removed from the regresson s a dffcult queston. Ths secton of the Mntab lstng does not really provde enough gudance as to the degree of nfluence. You can get addtonal nformaton on hgh nfluence ponts through Storage and then askng for H (leverages) when the regresson s ntated. See also pont [uu]. [mm] The unusual observatons are dentfed by ther case numbers (here 6, 37, 45, and 47), by ther values on the frst-named predctor varable, and by ther values on the dependent varable. [nn] The ftted values refer to the calculaton suggested n tem [ø], and here s the value of Fêrt 6, the ftted value for pont 6. [øø] The SE Ft refers to the estmated standard devaton of the ftted value n the prevous column. Ths s only margnally nterestng. [pp] Ths gves the actual resdual; Here = Fert 6 - Fêrt 6 = [qq] Snce the actual resduals bear the unts of the dependent varable, they are hard to apprase. Thus, we use the standardzed resduals. These can be thought of as approxmate z-scores, so that about 5% of them should be outsde the nterval (-, ). [rr] Ths marks a large nfluence pont. See tem [ll]. 5

16 [ss] The purpose of the resdual versus ftted plot s to check for possble volatons of regresson assumptons, partcularly non-homogeneous resduals. Ths pathology reveals tself through a pattern n whch the spread of the resduals changes n movng from left to rght on the plot. The resdual versus ftted plot wll sometmes reveal curvature as well. Large postve and large negatve resduals wll be seen on ths plot. The detecton and relevng of these pathologes are subtle processes whch go beyond the content of ths document. The appearance of ths plot s not materally nfluenced by the partcular choce made for standardzng the resduals; ths was done here from the Stat Regresson Regresson Graphs panel by choosng Resduals for plots: as Regular. [tt] The purpose of plottng the resduals aganst the predctor varables (n turn) s to check for non-lnearty. Ths partcular plot shows no problems. [uu] The leverage value for the frst canton n the data set s Ths s computed as the (, ) entry of the matrx X(X X) - X where X s the n-by-(k + ) matrx whose n rows represent the n data ponts. The k + columns consst of the constant column (contanng a n each poston) and one column for each of the k predctor varables. The dependent varable Fert s not nvolved n ths arthmetc. The leverage value for the j th canton s the (j, j) entry of ths matrx. A commonly accepted cutoff 3( k + ) 35 markng off hgh leverage ponts s, whch s here ( + ) Only n 47 the leverage value for canton 45 s larger than ths; see [rr] and [vv]. [vv] The leverage value for the 45 th canton s Ths s clearly a hgh leverage pont. There s some cause for concern, because hgh leverage ponts can dstort the estmaton process. A quck look at the data set wll show that ths canton, the thrd from the bottom on the data lst, has very unusual values for Ag, Army, and Ed. A thorough analyss of ths data set would probably nclude another run n whch canton 45 s deleted. 6