Data Analysis Toolkit #7: Hypothesis testing, significance, and power Page 1

Transcription

1 Daa Analyi Toolki #7: Hypohei eing, ignificance, and power Page 1 The ame baic logic underlie all aiical hypohei eing. Thi oolki illurae he baic concep uing he mo common e, - e for difference beween mean. Te concerning mean are common for a lea hree reaon. Fir, many queion of inere concern he average, or ypical, value of variable (I he average greaer or le han ome andard? I he average greaer or maller in one group han anoher?). Second, he mean i he mo able of all he claical meaure of a diribuion. Third, he diribuional properie of mean are well underood, and convenien: he cenral limi heorem how ha mean end oward he normal diribuion, which i imple (in he ene ha i depend only on he mean and andard error). Aumpion underlying -e: -The mean are normally diribued. Tha mean ha eiher he individual daa are hemelve normally diribued, or here are enough of hem combined ogeher in he mean o make he mean approimaely normal anyhow (2-3 meauremen are uually ufficien, depending on how non-normal heir diribuion are). Deviaion from normaliy become increaingly imporan a one rie o make inference abou more dian ail of he diribuion. -The only difference beween he hypohei and he null hypohei i ha he rue mean of he daa would be in a differen locaion; he diribuion would have he ame hape and widh under boh hypohee. Ouline of Hypohei Teing 1. Formulae he hypohei (H1) and he null hypohei (H). Eample: Eample: Thee hypohee mu be muually ecluive (hey canno be rue imulaneouly) and joinly ehauive (here are no oher poibiliie). Boh of hee condiion mu be me before you can legiimaely conclude ha if H i fale, hen H1 mu be rue. H1: average level of chromium in drinking waer are above he federal 1 ppb andard H: average Cr level in drinking waer are a or below he federal andard. Becaue he federal andard i fied, raher han deermined by meauremen, hi i a "ingle ample e". H1: average level of chromium in drinking waer in Ourown are higher han in Yourown. H: level in Ourown are he ame a, or lower han, in Yourown. Becaue you are comparing wo e of meauremen (raher han one e again a benchmark), hi i a "wo ample e". The only difference i ha one mu accoun for he uncerainie in boh ample (he benchmark in a one-ample e obviouly ha no uncerainy). Thi eample can be rephraed a if i were a one-ample e: "H1: The average Cr level in Ourown, minu he average in Yourown, i greaer han zero" and "H: The difference beween hee average i negaive or zero". Again, he only difference i ha, for a wo-ample e, you need o pool he variance from your wo ample. 2. Decide how much rik of "fale poiive" (Type I error raeα) and "fale negaive" (Type II error raeβ) you are willing o olerae. Eample: α i he chance ha, if he null hypohei i acually rue, you will miakenly rejec i. β i he chance ha, if he null hypohei i acually fale, and he rue diribuion differ from i by an amoun, you will miakenly accep i. If he average Cr level i really below he andard, I wan o be 9% ure I don' "cry wolf"; herefore α.1. If, on he oher hand, average Cr i above he andard by more han 2 ppb, I wan o be 95% ure I've deeced he problem; herefore, β.5 for 2. The choice of α and β hould be guided by he relaive conequence of making eiher miake. α.5 i convenional, bu may no be appropriae for your circumance. 3. Deermine how he propery of inere would be diribued, if he null hypohei were rue. Eample: Mean are ofen roughly normally diribued (ecep mean of mall ample of very non-normal daa). Normally diribued mean whoe andard error are eimaed by ampling (and hu alo uncerain) follow Suden' diribuion. 4. Deermine he accepance and rejecion region for he e aiic, and frame he deciion rule. The rejecion region i he ail (or ail) of he diribuion encompaing a oal probabiliy of α ha are lea compaible wih he null hypohei. Thu, if he null hypohei i rue, he chance i α ha he propery will fall wihin he rejecion region. The accepance region i he whole diribuion ecep for he rejecion Copyrigh 1995, 21 Prof. Jame Kirchner

2 Daa Analyi Toolki #7: Hypohei eing, ignificance, and power Page 2 region: he range ha encloe a probabiliy of 1-α ha i mo compaible wih he null hypohei. The deciion rule imply ae he condiion under which you'll accep or rejec he null hypohei. Uually, he rule i imply "rejec he null hypohei if he propery of inere fall wihin he rejecion region, and accep he null oherwie." Eample: If he rue mean (µ) i 1 ppb, hen he chance i α of geing value of ( 1 ) SE( ) ha are greaer han α, (where n-1 i he number of degree of freedom). The value α, i omeime referred o a he "criical value" for, ince hi i he value ha eparae value of ha are ignifican a α from hoe ha are no. Therefore, if >α, rejec H; oherwie accep H. Diribuion of poible ample mean i rue (µµ) p(bar) ail ecluded (probabiliyα ) α, bar Accepance region: accep If bar fall in hi region, H if bar fall in hi region rejec H Figure 1: Accepance and rejecion region in -e for aiical ignificance of mean. 5. Now--and no before--look a he daa, evaluae he e aiic, and follow he deciion rule. Eample: Five meauremen give a mean of 16 ppb, wih a andard deviaion of 1 ppb. Thu he andard error i 4.47 ppb, and (he number of andard error by which he mean deviae from he 1 ppb benchmark) i hu (16ppb-1ppb)/4.47ppb1.34. Thi i le han he criical value of, α,.1,4 1.53, o he null hypohei canno be rejeced (wihou eceeding our olerance for rik of fale poiive). However, fall beween.1,4 and.25,4 o he aiical ignificance of he daa lie beween p.1 and p If he null hypohei wa no rejeced, eamine he power of he e. Eample: In our eample, could he e reliably deec violaion of he andard by more han 2 ppb (ee power, below)? β α, 2 ppb ppb, The calculaed value of β, i lighly greaer han.25,42.78, o he probabiliy of a "fale negaive" if he rue mean violaed he andard by 2 ppb would be ju under 2.5%. Equivalenly, he power of he e i abou 1-β.975: he e hould be roughly 97.5% reliable in deecing deviaion of 2 ppb above he andard. 7. Repor he reul. Alway include he meaured magniude of he propery of inere (wih andard error or confidence inerval), no ju he ignificance level. Eample: "The average of he concenraion meauremen wa 16±4.5 ppb (mean±se). The andard i eceeded, bu he eceedance i no aiically ignifican a α.1; we canno conclude wih 9% confidence ha he rue mean i greaer han 1 ppb. The 9% confidence inerval for he mean i 96 o 116 ppb (16ppb±.5,4*4.47ppb). The e mee our requiremen for power; he e hould be abou 97% reliable in deecing level more han 2ppb above he andard." Copyrigh 1995, 21 Prof. Jame Kirchner

3 Daa Analyi Toolki #7: Hypohei eing, ignificance, and power Page 3 Common illegiimae inference 1. "The daa are conien wih he hypohei, herefore he hypohei i valid." The correc inference i: if he hypohei i valid, hen you hould oberve daa conien wih i. You canno urn hi inference around, becaue oher hypohee may alo be conien wih he oberved daa. The proper chain of inference i hi: conruc he hypohei (HA) and he null hypohei (H) uch ha eiher one or he oher mu be rue. Therefore i fale, hen HA mu be rue. Ne, deermine wha obervable daa are logically implied by H. If he daa you oberve are inconien wih H, hen H mu be fale, o HA mu be rue. In oher word: he only way o prove HA i rue i by howing ha he daa are inconien wih any oher hypohei. 2. "The null hypohei wa no rejeced, herefore he null hypohei i rue." Failing o how he null hypohei i fale i no he ame hing a howing he null hypohei i rue. 3. "The effec i no aiically ignifican, herefore he effec i zero." Failure o how ha he effec i no zero (i.e., he effec i no aiically ignifican) i differen from ucce in howing ha he effec i zero. The effec i whaever i i, plu or minu i aociaed uncerainy. Effec are aiically inignifican whenever heir confidence inerval include he null hypohei; ha i, whenever hey are mall compared o heir uncerainie. An effec can be aiically inignifican eiher becaue he effec i mall, or becaue he uncerainy i large. Paricularly when ample ize are mall, or he amoun of random noie in he yem i large, many imporan effec will be aiically inignifican. 4. "The effec i aiically ignifican, herefore he effec i imporan." Again, he effec i whaever i i, plu or minu i aociaed uncerainy. Effec are aiically ignifican whenever heir confidence inerval eclude he null hypohei; ha i, whenever hey are large compared o heir uncerainie. An effec can be aiically ignifican eiher becaue he effec i large, or becaue i uncerainy i mall. Many rivial effec are aiically ignifican, uually becaue he ample ize i o large ha he uncerainie are iny. 5. "Thi effec i more aiically ignifican han ha effec, herefore i i more inereing/imporan." A he rik of redundancy: he effec are whaever hey are, plu or minu heir aociaed uncerainie. Tha i why you need o alway repor he magniude of he effec, wih quanified uncerainie. If you only repor aiical ignificance, reader canno judge wheher he effec are large enough o be imporan o hem. Wheher an effec i imporan depend on wheher you care abou i. Wheher an effec i aiically ignifican depend on wheher he oberved daa would be improbable if he effec were aben (if he null hypohei were rue). Saiical ignificance and pracical imporance are no he ame hing; indeed here i no logical connecion beween hem. 6. "The null hypohei wa rejeced a α.5. Therefore, he probabiliy i <5% ha he null hypohei i rue, and herefore, he probabiliy i >95% ha he alernaive hypohei (HA) i rue." Saiical ignificance i a aemen abou your daa, no abou your hypohei. Saiical ignificance of α.5 mean ha if he null hypohei were rue, hen here i le han 5% chance ha your daa would deviae o far from i. You canno urn ha inference around; you canno logically conclude ha if he daa deviae o far from he null hypohei, hen here i a chance of hu-and-uch ha he null hypohei i rue or fale. You can validly evaluae he probabiliy ha your hypohei i rue or fale, bu you can only do o by uing Baye' heorem (Bayeian inference). You canno do i by claical hypohei eing. Copyrigh 1995, 21 Prof. Jame Kirchner

4 Daa Analyi Toolki #7: Hypohei eing, ignificance, and power Page 4 Si Queion ha Simple Saiic can Anwer 1. Saiical ignificance Anwer he queion: If H were acually rue, wha i he chance ha would deviae from µ (he rue mean prediced by he null hypohei) by a lea he oberved amoun? -Deermine by how many andard error deviae from µ: µ µ n -Given he number of degree of freedom (n-1), find (via a -able), he wo value of α correponding o value of α, ha bracke he calculaed value of o ha α, < < α,. -The ignificance level of (denoed p) hen lie beween α' and α'' uch ha α > p > α. If he e i wo-ailed (ha i, eiher value higher or lower han µ would be incompaible wih H), and you read he α' from a able of one-ailed value, hen he p value mu be doubled, o accoun for he equal probabiliy on he oher ail. 2. Lea Significan Difference (LSD) Anwer he queion: Wha i he malle deviaion from µ ha would be aiically ignifican? -Look up he criical value of for your deired ignificance level α and your degree of freedom. Then, given your andard error or andard deviaion, olve: LSD α, α, n If he e i wo-ailed, hen you need o ue α/2 raher han α in he epreion above. The LSD i imply he end poin of he 1-α confidence inerval for around µ. Imporan The LSD i he lea ignifican difference, given your curren ample ize. You can make maller difference ignifican by increaing he ample ize. The LSD i he malle difference ha, if i i deeced, will be aiically ignifican. However, he power o deec difference a mall a he LSD i very low (only 5%). Tha i, if he rue difference i equal o he LSD, you will fail o deec i half he ime (ee power, below; alo ee Figure 3). You can only deec deviaion from µ reliably if hey are much larger han he LSD. 3. Lea Significan Number (LSN) Anwer he queion: Wha number of meauremen would be required o make a given deviaion from µ aiically ignifican? -Look up he criical value of for your deired ignificance level α and your degree of freedom. Then, given your oberved andard deviaion, olve: α, n µ 2 Imporan If he e i wo-ailed, hen you need o ue α/2 raher han α in he epreion above. The LSN i he ample ize required o make he oberved deviaion from µ aiically ignifican. Bigger deviaion will be ignifican wih maller ample ize han he LSN. The LSN make he oberved deviaion from µ aiically ignifican, bu he power o deec deviaion ha large i very low (only 5%) wih ample ize e a he LSN. Tha i, if here were anoher deviaion ju a large a he one you've oberved, you would fail o deec i half he ime. In fac, if you increae your ample ize o he LSN, here i only a 5-5 chance ha he reuling meauremen will be aiically ignifican afer all (ee power, below; alo ee Figure 3). To reliably deec difference a large a he one you've oberved, your ample ize need o be much larger han he LSN. Copyrigh 1995, 21 Prof. Jame Kirchner

5 Daa Analyi Toolki #7: Hypohei eing, ignificance, and power Page 5 4. Power Anwer he queion: If he rue mean µ acually differed from µ by an amoun, wha i he chance ha he ha you meaure would le you rejec H a a ignificance level of α? In oher word, i acually fale by an amoun, wha i he probabiliy of correcly rejecing H? Power meaure how reliably you can deec an effec of a given ize (ee Figure 2). Imporan 5. Minimum deecable difference -Look up α, for your deired ignificance level, and hen calculae he correponding β,: β, α, α, n -Uing a -able, find he wo value ha bracke β, for degree of freedom. The "fale negaive" error rae, β, will be beween he ignificance level of he value ha bracke β,. The correponding power i 1-β. -If ielf i no aiically ignifican (ha i, < α, ), hen β i greaer han.5 (ee Figure 4). Eimae β, a above (which will yield a negaive value), hen calculae 1-β,-β,. Then, a above, look up he - value ha bracke 1-β, (ince β, i negaive, 1-β, i poiive). Now he power ielf will be beween he ignificance level of he value brackeing 1-β,. The "fale negaive" error rae i β1-power. If he e i wo-ailed, hen you need o ue α/2 raher han α in he epreion above. For ample ize above 1 or o, one can ue a Z-able inead; Z cloely approimae β,. value for eimaing power are alway one-ailed, wheher he ignificance e ielf i one-ailed or wo ailed. If he conequence of a fale negaive are dire, and H i no rejeced, he power of he e may be much more imporan han i aiical ignificance. Low power mean ha given he uncerainy in he daa, he e i unlikely o deec deviaion from he null hypohei of magniude. To increae he power of a e, you can do wo hing (ee he equaion above): you can raie α (reduce he ignificance level), or you can increae he ample ize (n). Alo, if he variabiliy in () i parly wihin your conrol (e.g., from analyical error), reducing hi variabiliy will increae he power of he e. Bu in mo environmenal meauremen, mo of he variabiliy in i caued by real-world variaion (from ime o ime, place o place, individual o individual, ec.). There i nohing you can do abou hi, unle you can find he ource of ha variabiliy and correc for i in he daa (uch a wih paired-ample eing). Anwer he queion: By how much mu he rue mean µ acually differ from µ, o ha he chance i 1-β or beer ha he ha you meaure will le you rejec H, a a ignificance level of α? In oher word, by how much mu H acually be fale, o make he probabiliy 1-β ha you will correcly rejec H? -Look up α, and β, correponding o your deired ignificance level and power. Then calculae (by rearranging he equaion above): ( + ) ( ) α, β, α, + β, n -The bai for hi epreion can be een direcly in he geomery hown in Figure 2. The relaionhip beween power and can alo be ummarized graphically in a "power curve" (Figure 5). -If he minimum deecable difference i uncomforably large, hen you can decide o olerae larger error rae α and β, or you can increae he ample ize (n). 6. Sample ize (for a pecified ignificance and power) Anwer he queion: How large a ample do you need, o ha if he rue mean µ acually differ from µ by, your chance of rejecing H (a ignificance level α) will be 1-β or beer? -Again, by rearranging he equaion above o olve for n, we obain direcly, n 2 2 ( α, + β, ) 2 -If he required ample ize i impracical, you mu decide o olerae larger error rae (α and β), or decide ha a bigger deviaion from H can go undeeced. Copyrigh 1995, 21 Prof. Jame Kirchner

6 Daa Analyi Toolki #7: Hypohei eing, ignificance, and power Page 6 Dir. of ample mean i rue (µµ ) Diribuion of ample mean i fale, and µµ + Dir. of ample mean i rue (µµ ) Diribuion of ample mean i fale, and µµ +LSD p(bar) True negaive (p1-α ) True poiive (power1-β ) p(bar) β.5 power 1-β.5 True poiive β α α α, bar β, bar Fale negaive (Type II error) µ µ + Fale poiive (Type I error) Figure 2. Tradeoff beween power (1-β), ignificance level (α), and minimum deecable difference (). Moving he ignificance hrehold righ or lef hrink β bu increae α, and vice vera. The only way o decreae α and β imulaneouly i by increaing (moving he wo diribuion farher apar) or by increaing n, hu decreaing (making he wo diribuion narrower) α, bar Fale negaive (Type II error) µ µ +LSD Accep H Rejec H Fale poiive (Type I error) Figure 3. If he rue mean µ differ from null hypohei mean µ by an amoun equal o he lea ignifican difference (LSD), hen half of all ample mean will be aiically ignifican (o he righ of µ) and he oher half will no. Thu, eing ample ize by LSN or LSD yield e wih only 5% power. p(bar) Diribuion of ample mean Dir. of ample mean i fale, and µµ + i rue (µµ ) True negaive (p1-α ) Fale negaive β α True poiive (power1-β ) power α.5 α Fale α, bar poiive 1 β, bar Accep H Rejec H Figure 4. Te wih very low power. If rue mean µ lie wihin he accepance region, mo of he ample mean (diribuion hown by dahed curve) will alo fall wihin he accepance region (fale negaive). Few of he ample mean will differ enough from µo o be aiically ignifican (rue poiive). High fale negaive rae make negaive reul inconcluive deviaion from µ o (# andard error) Figure 5. Power curve, howing he power o deec a given deviaion () from he null hypohei, meaured in andard error (ha i, he -ai i ). Noe ha making ignificance e more ringen widen he zone of low power. Curve are calculaed under he large-ample approimaion (ha i, uing normal diribuion o approimae he - diribuion). Copyrigh 1995, 21 Prof. Jame Kirchner