Power and Sample Size Concepts and Calculations

Transcription

1 Power ad Sample Size Cocepts ad Calculatios For a variety of statistical settigs, oe commoly approaches power ad sample size issues from oe of three startig poits: What sample size is required to achieve a cofidece iterval of specified width? What sample size is ecessary to achieve a specified power? What power is achieved give a specified sample size? I order to do ay of these calculatios, a estimate of the stadard deviatio i your sample (or possibly i each of several samples) is required. I may situatios, power ad sample size calculatios are doe before collectig data (i fact, it is usually a good idea to do so). Fortuately, it is possible to provide a estimate of the SD without havig see ay data. If you ca estimate the rage (largest mius smallest) of values you will likely see, that rage divided by three or four is a decet guess for the SD.

2 Sample Size Determied by Precisio Goals The width, which we will characterize by the margi of error, of cofidece itervals is liked directly to sample size. A smaller desired margi of error requires greater precisio (i.e. a smaller stadard error), which i tur requires a larger sample size. I these otes, we assume that the samplig distributio of the relevat statistic is ormal. We ll address the cocepts usig as example oe of the simplest of settigs, a cofidece iterval (CI) for a populatio mea, based o a sigle sample mea, assumig Normality for the samplig distributio of the mea. If you are t familiar with basic cofidece iterval costructio ad termiology,. There are three essetially equivalet ways you ca impose some chose precisio o the CI. The oe we ll use here is the margi of error, m.e.: me.. = t 1,.95 s. The margi of error is called by some the cofidece iterval halfwidth. Aother commo approach (techically equivalet) is to state the desired precisio i terms of the CI width w: ( w 2( me..)) =.

3 A third way is as a percet, as i, We wat to be withi plus or mius 10%. Here the desired precisio (we ll call it relative margi of error, ad label it with rme) is relative to the size of the mea; the coectio to m.e. is.. rme = 100% me. y The take-home poit: cofidece iterval precisio i terms of CI margi of error, width, or relative margi of error are all equivalet. It is easy to traslate from ay oe style to either of the other two. Give a choice of margi of error, the coectio to sample size is straightforward coceptually, ad, at first glace, looks like it leads to a easy computatio: me.. = t 1,.975 s ( square bothsides;swap ad me..) 2 s = t 1,.975 me... The problem with this formula is that oe eeds to kow i order to kow t (sice the correct t-distributio depeds o the degrees of freedom) i order to calculate i order to compute t i order headache material. A way out of the problem: it is a fact that as icreases, the multiplier t b g gets closer ad closer to z b1 α 2g (the correspodig 1, 1 α 2

4 value from a stadard Normal distributio). I fact, t b 1, 1 α 2 g is always goig to be larger tha zb 1 α 2 g sample size estimate usig zb 1 α 2 g for ay sample size. If we do a iitial i the formula (which we ca do easily), we will have a sample size that is smaller (likely ot by much) tha actually required. We the iteratively icrease our estimated sample size util we reach the desired precisio. A example may make this clearer. Example. A biologist wishes to make a 95% CI for a populatio mea, ad wats the iterval to be plus or mius 10% of the target mea. She estimates, after coferrig with colleagues, that the mea will be about 50. These together mea that her desired CI margi of error is 5. She ad her colleagues also estimate that the stadard deviatio will be about 10. Steps Formula Calculatio Use basic formula to get a (slight uder-) estimate z s me = 2 = 2 F I HG K J = Use 16 (rouded up from 15.37) to see what m.e. is attaied. me.. = t 16 1,.975 s me.. = = That s bigger tha our target. Add 1 to the sample size; try agai me.. = t 1,.975 s me.. = =

5 That s still bigger tha our target. Add 1 to the sample size; try agai. This time it works. me.. = t 1,.975 s me.. = = Clearly this would be pretty tedious to do routiely. To do these calculatios easily ad effortlessly,. Sample Size to Achieve Specified Power The power of a statistical test is the probability of correctly detectig a existig effect (we ll defie effect shortly). Power is a fuctio of 1. the effect size (power icreases as effect size icreases); ad 2. the chose alpha level (power icreases with larger alpha levels); 3. the sample size (power icreases as sample size icreases). Two type of errors ca occur whe doig a hypothesis test. Oe, a socalled false sigificace (or false rejectios of the ull hypothesis), has a specified chace, alpha, of occurrig. The other, a false retetio of the

6 ull, occurs whe oe fails to reject the ull hypothesis, but i fact some alterate is true. This probability goes by the ame beta. Power is 1 beta. to visually study the iterplay betwee these two probabililities. Power of a Statistical Test I this discussio, we will assume that the samplig distributio of the relevat statistic is ormal. The power of a test is the probability of correctly rejectig the ull hypothesis whe a specified alterate is true. Swift Fox Example: (adapted from a study by Olso (1999)). Oe questio of iterest i a study of swift foxes was whether home rage size icreased from summer to fall. A sample of 10 summer home rages yielded a mea of 1000ha, with a SD of 320ha. The samplig distributio of the mea is assumed to be ormal, with stadard deviatio estimated from the stadard error of the summer sample to be se( x ) s 320 = = This is a oe-tailed hypothesis test (they are oly iterested i icreases), for which they chose to use the covetioal α = 0.05.

7 Effect Size I order to do a power aalysis, oe eeds to establish effect sizes for which to do the calculatios. Effect size is the differece betwee the value of a parameter uder the ull hypothesis ad its value uder a alterate. Effect size is a commoly used geeric term, the cotextdepedet meaig of which will hopefully be made clear through the followig examples. Example 1. I a fisheries study, 15 radomly chose et sites were chose. A oe-sided, oe-sample t-test was used to test whether the mea umber of trout per et was 6 (the historical average), or if, i fact, the mea umber appears to be decliig. The biologist wated to be sure to have high power to detect a declie of 2 fish per et, if ideed the declie was that large. Here, the effect size of iterest was a chage of 2 i the mea umber of trout caught per et. Example 2. A study was doe to compare male ad female salaries amog biologists of similar rak ad seiority i a federal agecy. They performed a two-sided, two-sample t-test for the ull hypothesis of o differece. I discussio, it was decided that if the true differece was more tha $2,000 per year, they wated to be sure to detect it. Here, the effect size, was a differece of $2000 i the mea salaries.

8 I a regressio study, the effect size might be a differece i slopes (that is, some chose differece from the value beig tested i the ull hypothesis). Effect size, is a geeric term that truly gais its meaig from the particular cotext. Oe way to approach the questio of effect size is to choose a rage of effect sizes. Pick a effect that is sufficietly small so that ay effects (should they exist) that are ay smaller are ot of much iterest. Pick a effect that is sufficietly large so that ay effects that are ay larger are quite importat to detect. Aalyze power/sample sizes for that rage of effects. Swift Fox Example. For thi study, a icrease of 10% (100ha) was chose for the smaller value, ad 50% (500ha) for the larger. Statistical Decisio Rule For the swift fox example, the researchers used α = 0.05, which dictates thatp-values less tha 0.05 would lead to rejectio of the ull hypothesis, while for p-values larger tha 0.05, we would fail to do so (at least, this is the formal, mechaistic way to use p-values ad alpha). I a Normal distributio, 5% of the values are larger tha 1.65 stadard deviatios above the mea. Give our SE of 100, we would, expect

9 meas larger tha 1165 about 5% of the time if the ull hypothesis (o icrease i home rage size) were true. The key poit to keep i mid is that a mea home rage size larger tha 1165 will have a p-value less tha α = 0.05, ad will lead to rejectio of the ull hypothesis. Let s cosider the cosequeces of this decisio rule uder the assumptio that, i fact, the average home rage i the fall is 1100ha (for a effect size of 100); ad let s examie the cosequeces for for 1500ha (effect size is 500). The other pieces we eed for the calculatios are: SD = 320; = 10; α =.05; ad the fact that the test is oe-tailed. Write those figures o a scrap of paper, ad. What power is achieved give a specified sample size? The power of a statistical test is a fuctio of the effect size (power icreases as effect size icreases); the alpha level (power icreases with larger alpha levels); ad the sample size (power icreases as sample size icreases). This questio is the complemet of calculatig the sample size required to achieve a give power. Usually, this questio is asked for existig studies, to seek a better uderstadig of their limitatios ad for plaig for future studies. The swift fox example had home rage size estimates from = 10 fox pairs. What size of chage (if ay) i home rage size ca they detect with that sample size?

10 Calculatios for Paired data A paired t-aalysis proceeds as a aalysis of a sigle sample, after havig subtracted the values i oe sample from those i the other. If you have existig data to work with, the use the sample of differeces to directly estimate the stadard deviatio of the differeces SD d ad proceed followig the sigle sample istructios. The difficulty i workig through power ad sample size calculatios without existig data is that the stadard deviatio of the differeces is a fuctio of the stadard deviatio i each sample ad the correlatio betwee the samples, which is really hard to estimate without data. That said, if you are blessed with eough isight ito your data, you ca use as a estimate SD = S 2 2r, where S is the (assumed to be the same for both) d stadard deviatio withi each sample, ad r is the estimated correlatio betwee the samples. The followig is a tool that icorporates several desigs (two idepedet samples, paired, ad a mixture of the two) alog with flexible variace structures (assumed equal variaces, arbitrary (your choice) variaces, SD proportioal to meas, etcetera):.

11