Data Analysis Toolkit #10: Simple linear regression Page 1

Transcription

1 Data Aaly Toolkt #0: mple lear regreo Page mple lear regreo the mot commoly ued techque f determg how oe varable of teret the repoe varable affected by chage aother varable the explaaty varable. The term "repoe" ad "explaaty" mea the ame thg a "depedet" ad "depedet", but the fmer termology preferred becaue the "depedet" varable may actually be terdepedet wth may other varable a well. mple lear regreo ued f three ma purpoe:. To decrbe the lear depedece of oe varable o aother. To predct value of oe varable from value of aother, f whch me data are avalable 3. To crect f the lear depedece of oe varable o aother, der to clarfy other feature of t varablty. Ay le ftted through a cloud of data wll devate from each data pot to greater leer degree. The vertcal dtace betwee a data pot ad the ftted le termed a "redual". Th dtace a meaure of predcto err, the ee that t the dcrepacy betwee the actual value of the repoe varable ad the value predcted by the le. Lear regreo determe the bet-ft le through a catterplot of data, uch that the um of quared redual mmzed; equvaletly, t mmze the err varace. The ft "bet" precely that ee: the um of quared err a mall a poble. That why t alo termed "Ordary Leat quare" regreo. Dervato of lear regreo equato The mathematcal problem traghtfward: gve a et of pot, o a catterplot, fd the bet-ft le, Ŷ a + b uch that the um of quared err, Ŷ mmzed The dervato proceed a follow: f coveece, ame the um of quare "Q", Q Ŷ a b The, Q wll be mmzed at the value of a ad b f whch Q / a 0 ad Q / b 0. The frt of thee codto, Q a a b a + b 0 whch, f we dvde through by ad olve f a, become mply, a b 3 whch ay that the cotat a the y-tercept et uch that the le mut go through the mea of x ad y. Th make ee, becaue th pot the "ceter" of the data cloud. The ecod codto f mmzg Q, Q b a b a b 0 4 If we ubttute the expreo f a from 3 to 4, the we get, + b b 0 5 We ca eparate th to two um, b 0 6 whch become drectly, Copyrght 996, 00 Prof. Jame Krcher

2 Data Aaly Toolkt #0: mple lear regreo Page Copyrght 996, 00 Prof. Jame Krcher b 7 We ca tralate 7 to a me tutvely obvou fm, by otg that 0 ad 0 8 o that b ca be rewrtte a the rato of Covx,y to Varx:, Var Cov b The quatte that reult from regreo aalye ca be wrtte may dfferet fm that are mathematcally equvalet but uperfcally dtct. All of the followg fm of the regreo lope b are mathematcally equvalet:, x xy Var Cov b 0 A commo otatoal hthad to wrte the "um of quare of " that, the um of quared devato of the ' from ther mea, the "um of quare of ", ad the "um of cro product" a, Var x x Var y y Cov xy xy, 3 It mptat to recogze that Σx, Σy, ad Σxy, a ued Zar ad equato 0-3, are ot ummato; tead, they are ymbol f the um of quare ad cro product. Note alo that ad -3 are uppercae ' rather tha tadard devato. Bede the regreo lope b ad tercept a, the thrd parameter of fudametal mptace the crelato coeffcet r the coeffcet of determato r. r the rato betwee the varace that "explaed" by the regreo, equvaletly, the varace Ŷ, ad the total varace. Lke b, r ca be calculated may dfferet way: y x xy Var Ŷ Var Var Var Var y x, Cov Var Var b Var VarŶ r 4

3 Data Aaly Toolkt #0: mple lear regreo Page 3 Equato 4 mple the followg relatohp betwee the crelato coeffcet, r, the regreo lope, b, ad the tadard devato of ad ad : r b ad b r 5 The redual e are the devato of each repoe value from t etmate Ŷ. Thee redual ca be ummed the um of quared err E. The mea quare err ME jut what the ame mple, ad ca alo be codered the "err varace". The root-mea-quare-err RME, alo termed the "tadard err of the regreo" the tadard devato of the redual. The mea quare err ad RME are calculated by dvdg by -, becaue lear regreo remove two degree of freedom from the data by etmatg two parameter, a ad b. Ŷ E e ME Var e E r RME r E 6 where Var the ample, ot populato, varace of, ad the fact of -/- erve oly to crect f chage the umber of degree of freedom betwee the calculato of varace d.f.- ad d.f.-. Ucertaty regreo parameter The tadard err of the regreo lope b ca be expreed may dfferet way, cludg: b r b r b b 7 r r If all of the aumpto uderlyg lear regreo are true ee below, the regreo lope b wll be approxmately t-dtrbuted. Therefe, cofdece terval f b ca be calculated a, CI b ± tα,b 8 To determe whether the lope of the regreo le tattcally gfcat, oe ca traghtfwardly calculate t, the umber of tadard err that b dffer from a lope of zero: b t r 9 b r ad the ue the t-table to evaluate the α f th value of t ad - degree of freedom. The ucertaty the elevato of the regreo le at the mea that, the ucertaty Ŷ at the mea mply the tadard err of the regreo, dvded by the quare root of. Thu the tadard err the predcted value of Ŷ f ome the ucertaty the elevato at the mea, plu the ucertaty b tme the dtace from the mea to, added quadrature: Ŷ + b Var where Var the populato, ot ample varace of that, t calculated wth rather tha -. Ŷ alo t- dtrbuted, o a cofdece terval f Ŷ ca be etmated by multplyg the tadard err of Ŷ by t α,-. Note that th cofdece terval grow a move farther ad farther from the mea of. Extrapolato beyod the rage of the data aume that the uderlyg relatohp cotue to be lear beyod that rage. Equato 0 gve the tadard err of the Ŷ, that, the -value predcted by the regreo le. The ucertaty a ew dvdual value of that, the predcto terval rather tha the cofdece terval deped ot oly o the ucertaty where the regreo le, but alo the ucertaty where the dvdual data pot le relato to the regreo le. Th latter ucertaty mply the tadard devato of the redual,, whch added quadrature to the ucertaty Ŷ, a follow: Copyrght 996, 00 Prof. Jame Krcher

4 Data Aaly Toolkt #0: mple lear regreo Page Ŷ Ŷ The tadard err of the -tercept, a, jut a pecal cae of 0 f 0, a + b + The tadard err of the crelato coeffcet r, r r 3 We ca tet whether the crelato betwee ad tattcally gfcat by comparg r to t tadard err, r t r 4 r r ad lookg up th value a t-table. Note that tr/ r ha the ame value a tb/ b ; that, the tattcal gfcace of the crelato coeffcet r equvalet to the tattcal gfcace of the regreo lope b. Aumpto behd lear regreo The aumpto that mut be met f lear regreo to be vald deped o the purpoe f whch t wll be ued. Ay applcato of lear regreo make two aumpto: A The data ued fttg the model are repreetatve of the populato. B The true uderlyg relatohp betwee ad lear. All you eed to aume to predct from are A ad B. To etmate the tadard err of the predcto, you Ŷ alo mut aume that: C The varace of the redual cotat homocedatc, ot heterocedatc. F lear regreo to provde the bet lear ubaed etmat of the true, A through C mut be true, ad you mut alo aume that: D The redual mut be depedet. To make probabltc tatemet, uch a hypothe tet volvg b r, to cotruct cofdece terval, A through D mut be true, ad you mut alo aume that: E The redual are mally dtrbuted. Cotrary to commo mythology, lear regreo doe ot aume aythg about the dtrbuto of ether ; t oly make aumpto about the dtrbuto of the redual e. A wth may other tattcal techque, t ot eceary f the data themelve to be mally dtrbuted, oly f the err redual to be mally dtrbuted. Ad th oly requred f the tattcal gfcace tet ad other probabltc tatemet to be vald; regreo ca be appled f may other purpoe eve f the err are o-mally dtrbuted. tep cotructg good regreo model. Plot ad exame the data.. If eceary, trafm the ad/ varable o that: -the relatohp betwee ad lear, ad - homokedatc that, the catter cotat from oe ed of the data to the other Copyrght 996, 00 Prof. Jame Krcher

5 Data Aaly Toolkt #0: mple lear regreo Page 5 If a ofte the cae, the catter creae wth creag, the heterocedatcty ca be elmated by trafmg dowward o the "ladder of power" ee the toolkt o trafmg dtrbuto. Coverely, f the catter greater f maller, trafm upward o the ladder of power. Curvature the data ca be reduced by trafmg ad/ up dow the ladder of power accdg to the "bulgg rule" of Moteller ad Tukey 977, whch llutrated the followg dagram: Note that trafmg wll chage the curvature of the data wthout affectg the varace of, wherea trafmg wll affect both the hape of the data ad the heterocedatcty of the data. Note that vual aemet of the "catter" the data are vulerable to a optcal lluo: f the data dety chage wth, the pread the value wll look larger wherever there are me data, eve f the err varace cotat throughout the rage of. 3. Calculate the lear regreo tattc. Every tadard tattc package doe th, a do may preadheet, pocket calculat, etc. It ot dffcult to do by had va a cutom preadheet, a the example o page 6 llutrate. The tep are a follow: a f each data pot, calculate,, ad b calculate the um of the,,,, ad c calculate the um of quare x, y, ad xy alo wrtte Σx, Σy, ad Σxy va-3 d calculate a, b, ad r va 0, 3, ad 4 e calculate b ad r va 7 ad 3 4. a Exame the regreo lope ad tercept. Are they phycally plauble? Wth a plauble rage of value, doe the regreo equato predct reaoable value of? b Doe r dcate that the regreo expla eough varace to make t ueful? "Ueful" deped o your purpoe: f you eek to predct accurately, the you wat to be able to expla a ubtatal fracto of the varace. If, o the other had, you wat to mply clarfy how affect, a hgh r ot mptat deed, part of your tak of clarfcato cot determg how much of the varato explaable by varato. c Doe the tadard err of the lope dcate that b prece eough f your purpoe? If you wat to predct from, are the cofdece terval f adequate f your purpoe? Imptat ote: r ofte largely rrelevat to the tak at had, ad lavhly eekg to obta the hghet poble r ofte couterproductve. I polyomal regreo multple regreo, addg me adjutable coeffcet to the regreo equato wll alway creae r, eve though dog o may ot mprove the predctve valdty of the ftted equato. Ideed, t may uderme the uefule of the aaly, f oe beg fttg to the oe the data rather tha the gal. 5. Exame the redual, e Ŷ. The followg redual plot are partcularly ueful: 5a. Plot the redual veru. ee example o pp If the redual creae decreae wth, they are heterocedatc. Trafm to cure th. -If the redual are curved wth, the relatohp betwee ad olear. Ether trafm, ft a olear curve to the data. -If there are outler, check ther valdty, ad/ ue robut regreo techque. 5b Plot the redual veru Ŷ, aga to check f heterocedatcty th tep redudat wth 5a f mple oevarable lear regreo, ad ca be kpped. 5c Plot the redual agat every other poble explaaty varable the data et. -If the redual are crelated wth aother varable call t Z, the check to ee whether Z alo crelated wth. If both the redual ad are crelated wth Z, the the regreo lope wll ot Copyrght 996, 00 Prof. Jame Krcher

6 Data Aaly Toolkt #0: mple lear regreo Page 6 accurately reflect the depedece of o. ou mut ether: crect both ad f chage Z, befe regreg o, preferably ue multple regreo, aother fttg techque that ca accout f the teracto betwee ad Z ad ther combed effect o. If the redual are crelated wth Z, but ot, the multple regreo o both ad Z wll allow you to predct me accurately that, expla me of the varace, but gg Z wll ot ba the regreo lope of o. 5d Plot the redual agat tme. -Check f eaoal varato, log-term tred. Aga, they hould be accouted f, f they are preet. 5e Plot the redual agat ther lag that, plot e veru e-. -If the redual are trogly crelated wth ther lag, the redual are erally crelated. eral crelato alo called autocrelato mea the true ucertate the relatohp betwee ad wll be larger potetally much larger tha uggeted by the calculated tadard err. Dealg wth eral crelato requre pecal techque uch a the Hldreth-Lu procedure, whch wll be explaed a later toolkt. 5f Plot the dtrbuto of the redual ether a a htogram, a mal quatle plot -If the redual are ot mally dtrbuted, your etmate of tattcal gfcace ad cofdece terval wll ot be accurate. 6 Check f outler, both vually ad tattcally ee Helel ad Hrch, ecto 9.5 f me fmato. Oe of the mplet meaure of fluece Cook' "D", calculated f each data pot a e D + Pot wth D value greater tha F 0.,3,- are codered to have a large fluece o the regreo le; f greater tha about 30 th crepod to D.4. Hgh fluece doe ot automatcally make a pot a outler, but t doe mea that t make a ubtatal dfferece whether the pot cluded ot. If the pot ca be how to be a mtake, the t hould be ether crected deleted. If t ot a mtake, f you are uure, the a me robut regreo techque ofte advable. Commo ptfall mple lear regreo Mtakely attrbutg cauato. Regreo aume that caue ; t caot prove that caue. ad may be trogly crelated ether becaue caue, becaue caue, becaue ome other varable caue varato both ad. Whch brg u to: Overlookg hdde varable. A metoed above, hdde varable that are crelated wth both ad ca obcure, eve dtt, the depedece of o. Overlookg eral crelato. trog eral crelato ca caue you to erouly uderetmate the ucertate your regreo reult ce ucceve meauremet are ot depedet, the true umber of degree of freedom much maller tha ugget. I tme-ere data, t ca alo produce purou but mpreve-lookg tred. Overlookg artfactual crelato. Wheever ome part of the -ax varable alo appear o the -ax, there a artfactual crelato betwee ad addto to oppoto to the real crelato betwee ad. Note: all of ptfall lted above ca occur eve though r large; deed, all of them ca ometme erve to flate r. Th yet aother reao why r hould rarely be the "holy gral" of regreo aaly. Overlookg ucertaty. Lear regreo aume that kow precely, ad oly ucerta. If there are gfcat ucertate, the regreo lope wll be lower tha t would have bee otherwe. The regreo le wll tll be a ubaed etmat of the value of that lkely to accompay a gve meauremet, but t wll be a baed etmat of the value that would are f could be cotrolled precely. Copyrght 996, 00 Prof. Jame Krcher

7 Data Aaly Toolkt #0: mple lear regreo Page 7 Copyrght 996, 00 Prof. Jame Krcher

8 Data Aaly Toolkt #0: mple lear regreo Page 8 Referece: Chamber, J. M., W.. Clevelad, B. Kleer ad P. A. Tukey, Graphcal Method f Data Aaly, 395 pp., Wadwth & Brook/Cole Publhg Co., 983. Helel, D. R. ad R. M. Hrch, tattcal Method Water Reource, 5 pp., Elever, 99. Moteller, F. ad J. W. Tukey, Data Aaly ad Regreo, 588 pp., Addo-Weley, 977. Copyrght 996, 00 Prof. Jame Krcher