Università di Trento DiSCoF Dipartimento di Scienze della Cognizione e Formazione Effect Size and Power luigi.lombardi@unitn.it Methodological course (COBRAS-DiSCoF)
A simple example of one-way anova. Factor A with four different levels.
Data patterns (subjects)
Cluster Analysis
Cluster Analysis Between-Groups Sum of Squares Within-Groups Sum of Squares
Cluster Analysis
Errors in hypothesis testing
Effect size A researcher should attempt to design a sensitive experiment, that is to say one that is sufficiently powerful to detect any differences of interest. It would be useful to have an index of the efficiency of experimental treatments. Many researchers use p-value to measure this efficiency. However, this is simply not appropriate! For example, many use F ratio in ANOVA to conclude about treat. This is a problem since a large F (either large treatment effects or large sample size or a combination of both) Power and sample size are positively related!
A measure of Relative Treatment Magnitude: omega squared
Cohen s (1977) criteria for omega squared A small effect is an experiment that produces an ω 2 of 0.01. A medium effect is an experiment that produces an ω 2 of 0.06. A large effect is an experiment that produces an ω 2 of 0.15 or greater. In sum, the presence of a significant F gives us some assurance that a statistical association exists. The size of the F itself (or the p-value) does not reflect the degree of this statistical association unambiguously. Omega squared provides this additional information. Both statistics, F and ω 2, contribute to a complete understanding of the statistical outcome and both should be included in any research report.
Reasonable power α = 0.05 α ω 2 =0.01 ω 2 n Power ω 2 =0.06 (1 - β) ω 2 =0.15
ω 2 =0.01 ω 2 =0.06 ω 2 =0.15 α = 0.01 α = 0.01 α = 0.01 α = 0.05 α = 0.05 α = 0.05
Power-sample size facts of life (Kraemer, 1985) Increasingly larger sample sizes are needed to increase power a fixed amount. Relatively small expected effect sizes require substantial sample sizes to achieve a reasonable power. Adopting a more stringent significance level leads to a hefty increase in sample size to maintain power at the same level with a less stringent criterion.
pwr an R package for power analysis
pwr an R package for power analysis R code > library(pwr) > pwr.anova.test(f=0.543,k=4,n=4,sig.level=0.05) R output Balanced one-way analysis of variance power calculation k = 4 n = 4 f = 0.543 sig.level = 0.05 power = 0.3121873 NOTE: n is number in each group
pwr an R package for power n analysis f Effect R code > library(pwr) size groups > pwr.anova.test(f=0.543,k=4,n=4,sig.level=0.05) k n n subjs. per group R output Balanced one-way analysis of variance power calculation k = 4 n = 4 f = 0.543 sig.level = 0.05 power = 0.3121873 NOTE: n is number in each group