Power & Effect Size power Effect Size

Transcription

1 Power & Effect Size Until recently, researchers were primarily concerned with controlling Type I errors (i.e. finding a difference when one does not truly exist). Although it is important to make sure that the Type I error is controlled, there is also always a chance that a researcher will not find a difference when one truly does exist (i.e. make a Type II error). The power of a statistical test is technically the probability of correctly rejecting a false null hypothesis. The more power a researcher has, the more likely s/he is to detect true differences, if they exist. Power is affected by the true alternative hypothesis the greater the difference between the distributions under the null and alternative hypothesis the higher the power. Power increases with:. Larger sample sizes as the sample size increases so does the power because the standard error of sampling distribution decreases as the sample size increases.. Smaller standard deviations as the standard deviation decreases so does the standard error of the sampling distribution. 3. Greater differences between the population means assumed under the null and alternative hypotheses - the greater the difference between the distributions under the null and alternative hypothesis the higher the power. 4. Increasing α - Increasing the probability of making a Type I error will also decrease the probability of making a Type II error, but at a price. If we find differences we can never be sure if they are true differences or Type I errors. The only factors that a researcher has control of in the above list are sample size and α. Since sample size is the only thing that affects power without any undesirable results discussions around power typically center around discussions around sample size. Effect Size Any test statistic is a function of both the standardized difference between the sample mean and the population mean assumed under the null hypothesis and sample size. Having a large sample size can result in a researcher rejecting a null hypothesis even when there are very minimal differences between the sample mean and the population mean assumed under the null hypothesis. Effect size is a measure of the degree to which 0 and differ in terms of standard deviation units from the parent population, rather than standard error units which rely on sample size. Many journals now require researchers to report effect sizes in addition to the results of statistical tests. Symbolically, effect size is expressed by: d = σ 0

2 Determining the power of a statistical test is dependent on the effect size. However, the effect size cannot be calculated based on the data collected from a research study because that would require the use of the sample mean to estimate which is a very big assumption and an approximation at best. Rather, the effect size must be estimated from:. Past research obtain sample means and standard deviations from past studies and make an informed guess at what the effect size might be.. Personal opinion as to what difference is important to detect for example, you might decide that a 0 point difference is test scores is important. In this case you only need to determine the population standard deviation from some other source. 3. Use of special conventions There are arbitrary cut-points for what is considered a small, medium or large effect, d = 0.0, 0.50, or 0.80, respectively. However, these values should only be used when there is no other way to estimate the required parameters for d. Power Calculations Since we removed the influence of sample size to obtain an effect size measure to deal with the true differences between 0 and independent of sample size we now need to combine our measure of effect size with some function of sample size to determine power. Delta, δ, combines the effect size with a function of the sample size that reflects the test statistic. It is known as a noncentrality parameter which is simply the mean of the sampling distribution of the test statistic if the null hypothesis is false. If the null hypothesis is true then δ = 0. When a test statistic is calculated based on a null hypothesis that is false the sampling distribution of the test statistic will not be centered around the population mean assumed under the null. Rather it will be centered around δ. The difference between the mean of the sampling distribution of the test statistic when the null hypothesis is true and when it is false is known as the degree of noncentrality and δ is the noncentrality parameter. It reflects how much the distribution has shifted from the distribution assumed under the null. The power of a statistical test depends on the likelihood of obtaining a test statistic from the shifted (or non-central) distribution that is greater than the critical value obtained assuming the null hypothesis is true. Although oftentimes we will obtain test statistics that are larger than the critical value because the null hypothesis is false, occasionally we will obtain test statistics that are smaller than the critical value by chance alone. The One-Sample t-test Recall that δ, combines the effect size with a function of the sample size that reflects the test statistic For a one-sample t-test the function of the sample size is the square root of n. So, δ = d n

3 3 Suppose a school psychologist believes that children with ADD tend to have a higher IQ than the general population. She knows that in the population the mean IQ is 00 with a standard deviation of 5 and she wants to know the power of detecting a difference of at least 3 points if she randomly selects a sample of 5 of students in her school that have been diagnosed with ADD. In this case 0 = 00, = 03, and σ = 5 so d = = = = 0. 0, and σ 5 5 δ = d n = 0.0( 5) =. 00 If she were to conduct a two-tailed test, with α = 0.05, then the power of her test would only be 0.7, meaning that if the null hypothesis is false and really is 03 for children with ADD then she will only detect these difference 7% of the time. What would the power be if she were to conduct a one tailed test with α = 0.05? How large of a sample size would she need to obtain power = 0.95? The t-test for Two Independent Samples When calculating the power for testing whether the means from two independent samples are equivalent the function of the sample size that we use to calculate δ depends on whether the two samples are equivalent in size. If the samples sizes for the two groups are equal then δ n = d, where n = the number of cases in one of the groups If the sample sizes are not equal then we need to take into consideration the fact that the larger sample size should be weighted more heavily. Therefore, rather than taking the arithmetic mean of the two sample sizes, which would weight them equally, we take the harmonic mean of the two sample sizes. If n = the sample size of the first group, and n = the sample size of the second group then the harmonic mean of n and n is: nn n h = = n + n + n n We use the harmonic mean of the two sample sizes to calculate δ, and hence find power so that n h δ = d,

4 4 Suppose a researcher believes that attitude towards school decreases as children get older. So he plans to select a random sample of second grade students and a random sample of sixth grade students and administer a survey measuring attitude towards school. Scores on the measure can range from 0 to 50, with lower scores reflecting less interest and enthusiasm in school, and in previous research he has found the standard deviation of this measure to be 7.3. If he can get 0 second grade students and 0 fifth grade students to participate in his study what will the power of conducting a two tailed test with α = 0.05 be if he wants to be able to detect a difference of at least 5 points on the attitudinal measure? In this case 0 = 5 and σ = d = = 5 = 0. 68, and σ 7.3 n 0 δ = d = 0.68 = 0.68(3.6) =. 5 So the power of his test is about 0.58, meaning that if the null hypothesis is false and the difference between and 0 really is 5 then he will only detect these differences about 58% of the time. How many students would be needed in each group to obtain power = 0.90? What would the power be if he were only able to get 0 second grade students to participate in his study? The Matched Sample t-test To conduct a power analysis for a dependent t-test one must also know the population standard deviation of the difference scores, which is often not available in practice. Although it might be possible to obtain some reasonable estimates for the population standard deviations for the two groups, and using the laws of variance we can use these to obtain the standard deviation of the difference scores we also need an estimate of the population correlation between the two scores because: Var(X - X ) = Var(X ) + Var(X ) - ρ(σ x ) (σ x ) We can assume that the variances in the two matched samples are equivalent so that Var(X - X ) = σ ρσ = σ ( ρ) = σ ( ρ) However, we still need to find an estimate of the correlation between X and X in the population. Assuming we can do so then d = where σ ( ) σ X = σ ρ X X X and then δ = d n, where n = the number of matched pairs

5 5 Suppose a researcher wants to conduct a study to see if reading achievement test scores improve for at risk students after participating in an after school tutoring program. Assume that the standard deviation of test scores is 5. and the correlation between test scores in the population (which is actually the test-retest reliability of the measure) is What would the power be of detecting a 5 point increase in test scores if he could only obtain 0 students for his study? How does this compare to the power of conducting an independent sample t-test with 0 students participating in the tutoring program and a control group of 0 students that did not? What if the reliability of the measure was only.6?