Effect Size and Power

Size: px

Start display at page:

Download "Effect Size and Power"

Constance Smith
10 years ago
Views:

1 Università di Trento DiSCoF Dipartimento di Scienze della Cognizione e Formazione Effect Size and Power [email protected] Methodological course (COBRAS-DiSCoF)

2 A simple example of one-way anova. Factor A with four different levels.

3 Data patterns (subjects)

4 Cluster Analysis

5 Cluster Analysis Between-Groups Sum of Squares Within-Groups Sum of Squares

6 Cluster Analysis

7 Errors in hypothesis testing

8 Effect size A researcher should attempt to design a sensitive experiment, that is to say one that is sufficiently powerful to detect any differences of interest. It would be useful to have an index of the efficiency of experimental treatments. Many researchers use p-value to measure this efficiency. However, this is simply not appropriate! For example, many use F ratio in ANOVA to conclude about treat. This is a problem since a large F (either large treatment effects or large sample size or a combination of both) Power and sample size are positively related!

Many researchers use p-value to measure this efficiency. However, this is simply not appropriate!

9 A measure of Relative Treatment Magnitude: omega squared

11 Cohen s (1977) criteria for omega squared A small effect is an experiment that produces an ω 2 of A medium effect is an experiment that produces an ω 2 of A large effect is an experiment that produces an ω 2 of 0.15 or greater. In sum, the presence of a significant F gives us some assurance that a statistical association exists. The size of the F itself (or the p-value) does not reflect the degree of this statistical association unambiguously. Omega squared provides this additional information. Both statistics, F and ω 2, contribute to a complete understanding of the statistical outcome and both should be included in any research report.

In sum, the presence of a significant F gives us some assurance that a statistical association exists.

12 Reasonable power α = 0.05 α ω 2 =0.01 ω 2 n Power ω 2 =0.06 (1 - β) ω 2 =0.15

13 ω 2 =0.01 ω 2 =0.06 ω 2 =0.15 α = 0.01 α = 0.01 α = 0.01 α = 0.05 α = 0.05 α = 0.05

14 Power-sample size facts of life (Kraemer, 1985) Increasingly larger sample sizes are needed to increase power a fixed amount. Relatively small expected effect sizes require substantial sample sizes to achieve a reasonable power. Adopting a more stringent significance level leads to a hefty increase in sample size to maintain power at the same level with a less stringent criterion.

Relatively small expected effect sizes require substantial sample sizes to achieve a reasonable

15 pwr an R package for power analysis

16 pwr an R package for power analysis R code > library(pwr) > pwr.anova.test(f=0.543,k=4,n=4,sig.level=0.05) R output Balanced one-way analysis of variance power calculation k = 4 n = 4 f = sig.level = 0.05 power = NOTE: n is number in each group

05) R output Balanced one-way analysis of variance power

17 pwr an R package for power n analysis f Effect R code > library(pwr) size groups > pwr.anova.test(f=0.543,k=4,n=4,sig.level=0.05) k n n subjs. per group R output Balanced one-way analysis of variance power calculation k = 4 n = 4 f = sig.level = 0.05 power = NOTE: n is number in each group

per group R output Balanced one-way analysis of variance power calculation k

1.5 Oneway Analysis of Variance

Statistics: Rosie Cornish. 200. 1.5 Oneway Analysis of Variance 1 Introduction Oneway analysis of variance (ANOVA) is used to compare several means. This method is often used in scientific or medical experiments