Chapter 0. Introduction to sample size calculations

Transcription

1 Chapter 0. Introduction to sample size calculations C.O.S. Sorzano National Center of Biotechnology (CSIC) June 29, Introduction to sample size calculations June 29, / 85

2 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

3 Sample size N= Introduction to sample size calculations June 29, / 85

4 References Paul Mathews. Sample size calculations. Practical methods for engineers and scientists. Mathews Malnar and Bailey, Inc. (2010) 0. Introduction to sample size calculations June 29, / 85

5 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

6 Why this course? How many mice do I need to put in each group to show that a new vaccine is effective? Too few is a waste of time (=money) and money, it is unethical Too many is a waste of time (=money) and money, it is unethical I need just enough 0. Introduction to sample size calculations June 29, / 85

7 Why this course? 0. Introduction to sample size calculations June 29, / 85

8 Why this course? 0. Introduction to sample size calculations June 29, / 85

9 Why this course? 0. Introduction to sample size calculations June 29, / 85

10 Why this course? 0. Introduction to sample size calculations June 29, / 85

11 Why this course? 0. Introduction to sample size calculations June 29, / 85

12 Why this course? 0. Introduction to sample size calculations June 29, / 85

13 Why this course? 0. Introduction to sample size calculations June 29, / 85

14 Why this course? 0. Introduction to sample size calculations June 29, / 85

15 Why this course? 0. Introduction to sample size calculations June 29, / 85

16 Why this course? 0. Introduction to sample size calculations June 29, / 85

17 Why this course? 0. Introduction to sample size calculations June 29, / 85

18 Why this course? 0. Introduction to sample size calculations June 29, / 85

19 Why this course? 0. Introduction to sample size calculations June 29, / 85

20 Why this course? 0. Introduction to sample size calculations June 29, / 85

21 Why this course? 0. Introduction to sample size calculations June 29, / 85

22 Why this course? 0. Introduction to sample size calculations June 29, / 85

23 Why this course? 0. Introduction to sample size calculations June 29, / 85

24 Why this course? 0. Introduction to sample size calculations June 29, / 85

25 Why this course? 0. Introduction to sample size calculations June 29, / 85

26 Why this course? 0. Introduction to sample size calculations June 29, / 85

27 Why this course? 0. Introduction to sample size calculations June 29, / 85

28 Why this course? 0. Introduction to sample size calculations June 29, / 85

29 Why this course? 0. Introduction to sample size calculations June 29, / 85

30 Why this course? 0. Introduction to sample size calculations June 29, / 85

31 Why this course? 0. Introduction to sample size calculations June 29, / 85

32 Why this course? 0. Introduction to sample size calculations June 29, / 85

33 Why this course? 0. Introduction to sample size calculations June 29, / 85

34 Why this course? 0. Introduction to sample size calculations June 29, / 85

35 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

36 Basics of statistical inference Research hypothesis: The new vaccine reduces the number of infected animals in a population. H 0 : π π 0 One-tail test H 1 : π < π 0 Research hypothesis: The new drug increases survival for patients with this disease in the next 5 years. H 0 : S S 0 One-tail test H 1 : S > S 0 Research hypothesis: The new machine does not produce tablets with the prescribed concentration H 0 : c = c 0 Two-tail test H 1 : c c 0 0. Introduction to sample size calculations June 29, / 85

37 Basics of statistical inference Research hypoteheses never use All, Some, None or Not all. Research hypothesis: All hypertense patients benefit from a new drug. No hypertense patient benefits from a new drug. Problem: We would have to measure absolutely ALL hypertense patients Research hypothesis: Not all hypertense patients benefit from a new drug. Some hypertense patients benefit from a new drug. Problem: Too imprecise, being true does not provide much information 0. Introduction to sample size calculations June 29, / 85

38 Basics of statistical inference Post-mortem analysis: 1 Design hypothesis research 2 Collect data 3 Hypothesis test Safe analysis: 1 Design hypothesis research 2 Calculate number of samples 3 Collect data 4 Hypothesis test 0. Introduction to sample size calculations June 29, / 85

39 Basics of statistical inference You CAN reject the null hypothesis and accept the alternative hypothesis You CAN fail to reject the null hypothesis because, there is not sufficient evidence to reject it You CANNOT accept the null hypothesis and reject the alternative because you would need to measure absolutely all elements (for instance, all hypertense patients). It s like in legal trials: The null hypothesis is the innocence of the defendant. You CAN reject his innocence based on proofs (always with a certain risk). You CAN fail to reject his innocence. You CANNOT prove his innocence (you would need absolutely all facts) 0. Introduction to sample size calculations June 29, / 85

40 Basics of statistical inference The goal of hypothesis testing is to disprove the null hypothesis! We do this by proving that if the null hypothesis were true, then there would be a very low probability of observing the sample we have actually observed. However, there is always the risk that we have been unlucky with our sample, this is our confidence level (the p-value is also related to this risk: the lower the p-value, the lower the risk). 0. Introduction to sample size calculations June 29, / 85

41 Basics of statistical inference: Significance and Statistical power 0. Introduction to sample size calculations June 29, / 85

42 Basics of statistical inference Step 1: Define the hypotheses An engineer works for MyPharma. He knows that the manufacture of each tablet has a standard deviation of 1 mg. (the manufacturing process can be approximated by a Gaussian). Knowing this, he sets the machine to a target amount of 250 mg. In a routine check with 20 tablets, he measures an average of mg. Is it possible that the machine is malfunctioning? H 0 : µ = 250 H 1 : µ Introduction to sample size calculations June 29, / 85

43 Basics of statistical inference E{x 1 } = µ, Var{x 1 } = σ 2 ˆµ = x 1 E{ˆµ} = µ, Var{ˆµ} = σ 2 0. Introduction to sample size calculations June 29, / 85

44 Basics of statistical inference ˆµ = x 1 + x 2 2 E{x 2 } = µ, Var{x 2 } = σ 2 E{ˆµ} = µ, Var{ˆµ} = σ Introduction to sample size calculations June 29, / 85

45 Basics of statistical inference ˆµ = x 1 + x 2 + x 3 3 E{x 3 } = µ, Var{x 3 } = σ 2 E{ˆµ} = µ, Var{ˆµ} = σ Introduction to sample size calculations June 29, / 85

46 Basics of statistical inference ˆµ = x 1 + x 2 + x 3 + x 4 4 E{x 4 } = µ, Var{x 4 } = σ 2 E{ˆµ} = µ, Var{ˆµ} = σ Introduction to sample size calculations June 29, / 85

47 Basics of statistical inference ˆµ = i=1 x i E{ˆµ} = µ, Var{ˆµ} = σ Introduction to sample size calculations June 29, / 85

48 Basics of statistical inference ˆµ = i=1 x i E{ˆµ} = µ, Var{ˆµ} = σ Introduction to sample size calculations June 29, / 85

49 Basics of statistical inference ˆµ = i=1 x i E{ˆµ} = µ, Var{ˆµ} = σ Introduction to sample size calculations June 29, / 85

50 Basics of statistical inference Step 2: Find the distribution of a suitable statistic if H 0 is true x i N(µ, σ 2 ) ˆµ N ) (µ, σ2 N Step 3: Plug-in the observed data if H 0 is true z = = Z = ˆµ µ σ N N(0, 1) 0. Introduction to sample size calculations June 29, / 85

51 Basics of statistical inference Step 4: Calculate the p-value Probability of observing a value as extreme as this one if H 0 is true. p value = Pr{ Z > } = Pr{Z < } + Pr{Z > } = = 0.3% 0. Introduction to sample size calculations June 29, / 85

52 Basics of statistical inference Step 5: Reject or not reject H 0 p value = 0.003( ) < 0.05 Reject H 0 p<0.05 * p<0.01 ** p<0.001 *** 0. Introduction to sample size calculations June 29, / 85

53 Types of tests Significance tests: Equivalence tests: Superiority tests: Non-inferiority tests: H 0 : µ = µ 0 H A : µ µ 0 H 0 : µ µ 0 H A : µ = µ 0 H 0 : µ µ 0 H A : µ > µ 0 H 0 : µ < µ 0 H A : µ µ 0 0. Introduction to sample size calculations June 29, / 85

54 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

55 Sample size determination: Confidence level 1 Step 1: Define the null hypothesis H 0 : µ = Step 2: Distribution under the null hypothesis Z = µ µ σ N = σ N N(0, 1) 3 Step 3: Plug-in observed data z = Step 4: Calculate p-value Pr{ Z > } = 0.3% 5 Step 5: Decide on H 0 0.3% < 0.5% Reject 1 Step 1: Define the minimum meanigful difference = 0.5(mg) 2 Step 2: Determine population variance σ 2 = 1 2 (mg 2 ) 3 Step 3: Determine significance and statistic threshold α = 0.05 Pr{ Z > 1.96} = Step 4: Solve for N σ N > 1.96 N > ( ) 1.96σ 2 = Introduction to sample size calculations June 29, / 85

56 Sample size determination: Confidence level Factors that affect sample size: N > ( z1 α 2 σ ) 2 (1) 1 Confidence level: 1 α z 1 α 2 N More confidence requires more samples. 2 Sample variance: σ 2 N If the sample variance increases, it is more difficult to detect the diference. 3 Effect size: N If we want to detect more subtle differences, we need more samples. 4 One- or Two-sided test: Two-sided N If the test is one-sided, z 1 α should be replaced by z 2 1 α, which is smaller. 0. Introduction to sample size calculations June 29, / 85

57 Sample size determination: Test power (right) µ 0 +z 1 α 2 σ σ < µ 1 z 1 β N N > N ( (z1 α 2 + z 1 β)σ µ 1 µ 0 ) 2 = ( (z1 α 2 + z 1 β)σ ) 2 0. Introduction to sample size calculations June 29, / 85

58 Sample size determination: Test power (left) µ 1 +z 1 β σ N < µ 0 z 1 α 2 ( σ (z1 α N > N 2 + z 1 β)σ µ 1 µ 0 ) 2 = ( (z1 α 2 + z 1 β)σ ) 2 0. Introduction to sample size calculations June 29, / 85

59 Sample size determination: Test power (two-sided) H 0 : µ = µ 0 H 1 : µ < µ 0 µ > µ 0 } N > ( (z1 α 2 + z 1 β)σ ) 2 (2) 0. Introduction to sample size calculations June 29, / 85

60 Sample size determination: Test power (one-sided) } H 0 : µ < µ 0 N > H 1 : µ > µ 0 ( ) 2 (z1 α + z 1 β )σ (3) 0. Introduction to sample size calculations June 29, / 85

61 Sample size determination: Confidence level+test power Factors that affect sample size: N > ( (z1 α 2 + z 1 β)σ ) 2 1 Confidence level: 1 α z 1 α 2 N More confidence requires more samples. 2 Population variance: σ 2 N If the population variance increases, it is more difficult to detect the diference. 3 Effect size: N If we want to detect more subtle differences, we need more samples. 4 One- or Two-sided test: Two-sided N If the test is one-sided, z 1 α should be replaced by z 2 1 α, which is smaller. 5 Test power: 1 β N If we want to increase the power of the test, we need more samples. 0. Introduction to sample size calculations June 29, / 85

62 Test power calculation: Confidence level+sample size If we fix the sample size, then z 1 β = N σ z 1 α Power π = Pr{z > z 2 1 β } = 1 β (4) 1 Confidence level: 1 α z 1 α 2 z 1 β π More statistical confidence implies less statistical power. 2 Population variance: σ 2 z 1 β π If the population variance increases, the statistical power decreases. 3 Effect size: z 1 β π If we want to detect more subtle differences, the statistical power decreases. 4 One- or Two-sided test: Two-sided z 1 α 2 > z 1 α z 1 β π Two-sided tests have less power than one-sided tests. 5 Sample size: N z 1 β π If we use fewer samples, the statistical power decreases. 0. Introduction to sample size calculations June 29, / 85

63 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

64 Population variance sample variance N > ( (z1 α Population mean µ 2 + z 1 β)σ Sample mean ˆµ = 1 N Population variance σ 2 N x i i=1 ) 2 Sample variance ˆσ 2 = s 2 = 1 N 1 µ ˆµ σ 2 s 2 N (x i ˆµ) 2 i=1 0. Introduction to sample size calculations June 29, / 85

65 Population variance mean variance Population variance σ 2 = Var{x i } Mean variance σ2 σ2ˆµ = Var{ˆµ} = N Sample mean variance s 2ˆµ σ 2 σ 2ˆµ s 2ˆµ 0. Introduction to sample size calculations June 29, / 85

66 Components of the population variance: repeated measures x im = bp i + ɛ im σ 2 = σ 2 BP + σ 2 ɛ If we repeat the measurement process M times and average the results M x i = 1 M x im = 1 M m=1 M = bp i + 1 M ɛ im m=1 M (bp i + ɛ im ) m=1 σ 2 = σ 2 BP + 1 M σ2 ɛ (5) 0. Introduction to sample size calculations June 29, / 85

67 Finite population effects Very large population: Blood pressure in Spain: We study a group of N = 30 people: Small population: Blood pressure in this class: µ, σ 2 ˆµ, σ 2ˆµ = σ2 N µ, σ 2 We study a group of N = 30 people (out of 32!!): ˆµ, σ 2ˆµ = σ2 N ( 1 N N population ) (6) This formula is only valid for the mean. In this example, ( σ 2ˆµ = σ ) = σ Introduction to sample size calculations June 29, / 85

68 Repeated measures Replication Repeated measures: σ 2 = σ 2 BP + 1 M σ2 ɛ Replication: Allows estimating σ 2ˆµ 0. Introduction to sample size calculations June 29, / 85

69 Components of the population variance: blocking variables No blocking: σ 2 = σ 2 gender + σ2 BP + σ2 treatment + σ 2 ɛ Blocking: σ 2 = σ gender 2 + σbp 2 + σtreatment 2 + σɛ 2 0. Introduction to sample size calculations June 29, / 85

70 Replication Replicates Replication: Allows estimating σ 2ˆµ x ij = µ + α j + ɛ ij Replicates: Allows estimating σ 2 α j x ij = µ + α j + ɛ ij 0. Introduction to sample size calculations June 29, / 85

71 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

72 Sample size determination fails if... 1 The estimate of the sample variance is wrong. 2 The distribution of the samples does not follow the hypothesis. 3 Two populations are assumed to have the same variance, when they do not. 4 Two populations are assumed to have the same distribution, when they do not. 5 Variable transformation does not fully solve a distribution problem. 6 Large-sample approximation does not apply. 0. Introduction to sample size calculations June 29, / 85

73 If the sample size is too large... 1 Improve the measurement repeatability and reproducibility. 2 Introduce a blocking variable to reduce the precision error. 3 Use a variable with a better precision. 4 Replace a categorical response (yes/no; pass/fail;...) by an ordinal or continuous variable. 5 Identify covariates that can help to reduce the uncertainty in the model. 6 Reduce the confidence level. 7 Reduce the test power. 8 Repeat measurements on the same subject as a way to reduce measurement variance. 9 Use paired-sample methods instead of two-independent-sample methods. 0. Introduction to sample size calculations June 29, / 85

74 Non-parametric tests In many ocasions we do not know the distribution of the underlying data and non-parametric tests are used 0. Introduction to sample size calculations June 29, / 85

75 Non-parametric tests The number of samples needed for a non-parametric test is larger than for a parametric one (because it throws away information, e.g., the sign test only uses the sign). The sample size must be increased by a factor that is inversely proportional to the Asymptotic Relative Efficiency : N non parametric = N parametric ARE (7) Mann-Whitney U test 3/π = Wilcoxon signed-rank test 3/π = Spearman correlation test 0.91 Kruskal-Wallis test Friedman ANOVA 0.955J/(J + 1) If not in this table, use a conservative value 0.85 where J is the number of repeated measures. 0. Introduction to sample size calculations June 29, / 85

76 Bad practices I 1 There is one magic sample size (say, N = 10) for all situations. 2 Use N = 30 because Student s t distribution is approximately normal for that size. 3 Use N = N total + 1 in a single sampling from a population with N total individuals. 4 Use a table based on Cohen s d = s. Reason: it assumes normality in the data. 5 Sample size and power calculations are exact. Reason: they are calculated in a conext with high uncertainty (the experiment has not been performed yet). 6 The sample variance is unknown. Reason: look for previously published results or perform a pilot study. 0. Introduction to sample size calculations June 29, / 85

77 Bad practices II 7 Zero acceptance number sampling plans are superior to other samplings. Reason: they are often poorly understood, and they maybe appropriate or not. 8 Postexperiment power is a useful indicator of the value of an experiment. Reason: postexperiment power calculation assumes that the parameter estimates are the true value of the parameters. Post-experiment high power is a necessary condition for supporting the goal of the experiment, but it is not a sufficient condition. 9 Special software is required to calculate sample size and test power. Reason: accurate values are better calculated by software, but approximate values can be calculated on paper. 10 We do not need to know how the collected data will be analyzed. Reason: Every sample size calculation is matched to an analysis method and decision criterion. 11 An experiment may be practically significant but not statistically significant. Reason: to be practically significant, the experiment must be first statistically significant. The converse is not true: an experiment may be statistically significant, but not practically significant. 0. Introduction to sample size calculations June 29, / 85

78 Good practices 1 Calculate the sample size, power and/or effect size before collecting data. Reason: Make sure you will have enough (and not too much) data to meet the goal of the experiment. 2 If necessary, perform a pilot study to estimate variance. 3 Increase the sample size to compensate for anticipated losses at random. 4 Before collecting your data, make sure that the experiment design meet the goals. You may even simulate the data collection and analysis. 5 Use power calculation to design the experiment and confidence intervals to report the results. 6 Use at least two methods (two softwares, or manual and software) to make sure there is no mistake. 7 Write up a summary describing all the steps taken to design the experiment. It will be useful for future designs and to review the outcome of the experiment if it fails. 0. Introduction to sample size calculations June 29, / 85

79 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

80 Further reading Introductory tests: G. van Belle. Statistical rules of thumb: Chapter 2. Wiley, 2008 A. Gelman, J. Hill. Data analysis using regression and multilevel/hierarchical models: Chapter 20. Cambridge Univ. Press, 2007 Fox N., Hunn A., and Mathers N. Sampling and sample size calculation The NIHR RDS for the East Midlands / Yorkshire & the Humber, 2007 Statistical software: PASS manual G* Power manual Stata power and sample size manual MLPowSim manual 0. Introduction to sample size calculations June 29, / 85

81 Further reading Web calculators: Introduction to sample size calculations June 29, / 85

82 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

83 Summary 1 Sample size calculations are particularized to the way the data will be analyzed. 2 The goal of hypothesis testing is to prove that the null hypothesis is false. Our research hypothesis should be in the alternative hypothesis. 3 There are five variables strongly related: sample size, population variance, confidence level, test power, effect size (relevance) 4 We can measure the variance of many different, but related, variables (population, sample, sample mean,...) 0. Introduction to sample size calculations June 29, / 85

84 Summary 0. Introduction to sample size calculations June 29, / 85

85 Outline 1 Introduction to sample size calculations Why this course? Basics of statistical inference Sample size determination Thoughts on variance Guidelines Further reading Summary 0. Introduction to sample size calculations June 29, / 85

86 Chapter 1. Sample size for the mean C.O.S. Sorzano National Center of Biotechnology (CSIC) June 23, Sample size for the mean June 23, / 57

87 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

88 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

89 A single sample with known variance µ 0 +z 1 α 2 σ σ < µ 1 z 1 β N N > N ( (z1 α 2 + z 1 β)σ µ 1 µ 0 ) 2 = ( (z1 α 2 + z 1 β)σ ) 2 1. Sample size for the mean June 23, / 57

90 A single sample with known variance Let us focus on the limit ˆµ value, the one that separates the values within the confidence interval and values outside: ˆµ = µ 0 + z 1 α σ 2 N z 1 α = ˆµ µ0 σ 2 N σ ˆµ = µ 1 z 1 β N z 1 β = µ1 ˆµ σ If we add now both equations, we reach the same result as in the previous lecture N z 1 α + z 2 1 β = µ 1 µ 0 σ N = N ( z1 α 2 + z 1 β ) 2 (1) where = σ is the normalized effect size. However, note that the first ratio z = ˆµ µ 0 σ N is distributed as a N(0, 1) and that z 1 α is the z value of this distribution below 2 which there is a probability 1 α 2. The same applies to the second ratio. 1. Sample size for the mean June 23, / 57

91 A single sample with known variance Example 1 Solution: Let us assume that we are manufacturing syrup with 3 mg/ml of a drug. The standard deviation of the manufacturing process is 0.1 mg/ml, and the deviations from the target amount follows a Gaussian distribution. How many samples do we have to screen if we want to detect a deviation from target of = 0.03 mg/ml, with a power of 90% and a confidence level of 95%? Power 1 β = 0.9 β = 0.1 z 1 β = z 0.9 = Significance 1 α = 0.95 α = 0.05 z 1 α = z = Effect size = 0.03 Population variance σ 2 = = 0.01 ( z1 α +z 1 β ) 2 ( ) 2 N > 2 = = N = 117 /σ / Sample size for the mean June 23, / 57

92 A single sample with known variance Example Solution: Let us assume that the measurement process (determination of the concentration of drug in each sample) has a coefficient of variation of 15%. How does this measurement error increase the variance of the samples? CV = σ ɛ µ 0 σ 2 ɛ = (CV µ 0 ) 2 = (0.15 3) 2 = The variance of the measurements is given by the variance of the manufacturing and the variance of the measurement process σ 2 = σ 2 manufacturing + σ 2 ɛ = σ = σ 2 = Sample size for the mean June 23, / 57

93 A single sample with known variance Example 2 Solution: N > ( z1 α What is the sample size now if we want to be as precise as before detecting the malfunctioning of the manufacturing process? 2 + z 1 β /σ ) 2 = ( ) = / N = Sample size for the mean June 23, / 57

94 A single sample with known variance Example 3 Solution: The variance of the samples now reduces to We wonder if we can reduce the cost of the experiment by dividing each syrup sample in 4 aliquotes, and determining the concentration of the sample by averaging the estimation of the concetration in the 4 aliquotes? σ 2 = σmanufacturing 2 + σ2 ɛ = = σ = N > ( z1 α 2 + z 1 β /σ ) 2 = ( ) = N = / However, the total number of concentration determinations is 4N = = 2832 > That is, it is cheaper to perform independent concentration determinations than 4 concentration determinations from the same sample. 1. Sample size for the mean June 23, / 57

95 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

96 A single sample with unknown variance Example 4 1-month old babies awake by night every 3 hours with a standard deviation of 0.5 h (the sleeping period is supposed to be normally distributed). We hypothesize that babies in a orphanage adapt already at this age and sleep longer (the mean shifts at least 1 hour). We do not know the standard deviation of the sleeping time since this may have also changed with respect to the general population, but it cannot be too far from 0.5. We plan to estimate the standard deviation of the sleeping time of babies in an orphanage from the data itself. How many children do I have to examine in order to prove my hypothesis? Solution: We cannot apply the calculations above because we do not know the population variance, but make a new theoretical development. 1. Sample size for the mean June 23, / 57

97 A single sample with unknown variance We may substitute σ 2 (the true population variance) by s 2 (the sample variance) in the design equations above z 1 α = ˆµ µ0 σ 2 N t 1 α 2,0,N 1 = ˆµ µ0 s N z 1 β = µ1 ˆµ σ N If we add now both equations t 1 β, s t 1 α 2,0,N 1 +t 1 β, s,n 1 = µ 1 µ 0 s N = N N,N 1 = µ1 ˆµ s N N t 1 α 2,0,N 1 + t 1 β, s N,N 1 2 where the normalized effect size is = s. (2) 1. Sample size for the mean June 23, / 57

98 Central Student s t distributions t ν : ν degrees of freedom 1. Sample size for the mean June 23, / 57

99 Noncentral Student s t distributions t µ,ν : ν degrees of freedom, µ non-centrality parameter 1. Sample size for the mean June 23, / 57

100 A single sample with unknown variance Example (continued) Solution: Our hypothesis test is one-sided: H 0 : µ 3 H A : µ > 3 We need to solve the equation t 1 α,0,n 1 + t 1 β, s N = N,N 1 We will use α = 0.05, β = 0.1, and = 1. We do not know s yet, we will use s guess = σ = 0.5 instead. This gives = = s guess σ = = Sample size for the mean June 23, / 57

101 A single sample with unknown variance Example (continued) Substituting N = ( t0.95,0,n 1 + t 0.9, 2 2 ) 2,N 1 N This is a non-linear equation with no analytical form. We can solve it iteratively. For the first iteration we will use the Gaussian parameters: Iter. t 0.95,0,N 1 t 0.9, 2,N 1 N N 1 z 0.95 = 1.65 z 0.9 = N = 6 1. Sample size for the mean June 23, / 57

102 A single sample with unknown variance An approximate formula for this case is given by N = ( t1 α 2,0,N 1 + t 1 β,0,n 1 ) 2 (3) where the non-centrality parameter is disregarded. Example (continued) The iterative procedure for this approximate method converges to 3.99, which would make N = 4. As seen in the exact formula (with non-centrality parameter), N = 4 would result in an insufficient level of confidence and/or test power. 1. Sample size for the mean June 23, / 57

103 Test power improvement Example (continued) Due to the increase from N = 5.63 to N = 6, we increase a little bit the test power (if we keep fixed the confidence level in our hypothesis test). To calculate the new test power we need to find 1 β such that the following equation is satisfied t 1 α,0,n 1 + t 1 β, s N = N,N = ( t0.95,0,5 + t 1 β, 2 2 ) 2,5 6 We find 1 β = , that is the power has slightly increased from 90% to 92.82%. 1. Sample size for the mean June 23, / 57

104 Test power improvement Example (continued) 1. Sample size for the mean June 23, / 57

105 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

106 Paired samples Example 5 We are developing a cough mixture and we would like to know its effectiveness. For doing so, we will take N different people with cough. Measure the frequency of coughing (coughs/min) before taking the mixture and 1 hour after taking the mixture. For a severe allergic response, this value is about 5 coughs/min. We would like to detect a reduction to at most 3.5 coughs/min. The standard deviation in the population of people with severe allergic response is 0.4 coughs/min. Let us assume that due to the limited time of observation, we may have a standard deviation due to measurements of 0.1 coughs/min. Confidence level=95%, Power=90%. Solution: We will see that with a small transformation of the data, we can use the case of one sample with known or unknown means. 1. Sample size for the mean June 23, / 57

107 Paired samples Let us denote as x i1 and x i2 the two measurements for the i-th individual. Each measurement alone has a variance Var{X 1 } = Var{X 2 } = σ 2 total = σ 2 population + σ 2 measurement Let us define now the difference between the two measurements x i = x i1 x i2 If the two measurements are independent from each other, then the variance of the difference is given by Var{ x} = 2σ 2 total σ x = 2σ total (4) If there is no difference in the treatment, then µ x = 0 1. Sample size for the mean June 23, / 57

108 Paired samples Example (continued) The hypotheses in our case are H 0 : x 0 H A : x > 0 For simplicity, let us assume a design with known variance. The design values are = 1.5, σtotal 2 = 2 ( ) = 0.34, = σ = 2.57, α = 0.05 (z 1 α = 1.65), β = 0.1 (z 1 β = 1.28). With these values we have ( ) 2 z1 α + z 1 β N = = 1.30 N = 2 1. Sample size for the mean June 23, / 57

109 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

110 Two-samples with known variance Example 6 We are interested in knowing if two competitive drugs taken by pregnant women has an effect on the birthweight of their babies. We presume that the standard deviation of the birthweight is σ = 800 g, and the mean of the whole population µ = 3 kg. We are interested in detecting differences larger than 300 g. Confidence level=95%, Power=90%. For this experiment we plan to observe a group of N 1 women taking drug 1 and N 2 women taking drug 2. Then, we will calculate the mean of each group and check if their difference is significant or not. Solution: We need to derive a new theoretical framework, although it is very similar the one-sample case. 1. Sample size for the mean June 23, / 57

111 Two-samples with known variance Measurements in group 1: x 1,1, x 1,2,..., x 1,N1 ˆµ 1 = 1 N 1 N 1 x 1,i i=1 Measurements in group 2: x 2,1, x 2,2,..., x 2,N2 ˆµ 2 = 1 N 2 N 2 x 1,i i=1 Optimal sampling: µ = ˆµ 1 ˆµ 2 Var{ µ} (5) = σ2 1 N 1 + σ2 2 N 2 argmin N 1,N 2 Var{ µ} s.t. N 1 + N 2 = ct N 2 = N 1 σ 2 σ 1 (6) Hypothesis test: H 0 : µ = 0 H A : µ 0 1. Sample size for the mean June 23, / 57

112 Two-samples with known variance We can solve this problem as we did for the one-sample case through the use of statistics with known distributions Adding both equations we have z 1 α = µ 2 σ 1 2 N + σ2 2 1 N 2 z 1 β = z 1 α 2 + z 1 β = µ σ 1 2 N + σ2 2 1 N 2 σ 2 1 N 1 + σ2 2 N 2 Using the relationship given by the optimal sampling we reach to N 1 = z 1 α + z 2 1 β σ1(σ 1+σ 2) 2 N 2 = z 1 α + z 2 1 β σ2(σ 1+σ 2) 2 (7) 1. Sample size for the mean June 23, / 57

113 Two-samples with known variance Example (continued) We note that σ 1 = σ 2, so N 1 = N 2 = N. The data provided yields α = 0.05 z 1 α = β = 0.1 z 1 β = 1.28 = 300 σ 1 = σ 2 = 800 = = σ2(σ 1+σ 2) N = ( ) 2 = N = Sample size for the mean June 23, / 57

114 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

115 Two-samples with unknown variance: σ 1 = σ 2 Since the variance of both groups is presumed to be the same, we can estimate the variance from both at the same time as s 2 12 = s2 1 + s2 2 2 The sample variance of the difference sought is The statistics defined as s 2 µ = s s2 12 = s N 1 N 2 N t 1 α 2,0,df = µ s 12 2 N t 1 β, s µ,df = µ 2 s 12 N follow Student s t distributions (central and noncentral, respectively) with df = 2(N 1) degrees of freedom. 1. Sample size for the mean June 23, / 57

116 Two-samples with unknown variance: σ 1 = σ 2 Adding both equations and solving for N we get t 1 α 2,0,df + t 1 β, s,df µ N = 2 2 (8) where the normalized effect size is = s 12. Remind that for sample size calculation s 12 is normally unknown (yet) and it is substituted by s guess, the standard deviation of the two populations (assumed to be the same in both). 1. Sample size for the mean June 23, / 57

117 Two-samples with unknown variance: σ 1 σ 2 The sample variance of the difference sought is The statistics defined as s 2 µ = s2 1 N 1 + s2 2 N 2 t 1 α 2,0,df = µ s 2 1 N 1 + s2 2 N 2 t 1 β, s,df = µ s 2 µ 1 N 1 + s2 2 N 2 follow Student s t distributions (central and noncentral, respectively) with (Welch Satterthwaite) df = ( ) s N 1 + s2 2 N 2 ( ) 1 s 2 2 ( ) 1 N 1 1 N s N 2 1 N 2 1. Sample size for the mean June 23, / 57

118 Two-samples with unknown variance: σ 1 σ 2 We exploit now the optimal allocation for unequal variances, which states that to obtain σ 2 σ 1 = N 2 N 1 t 1 α 2,0,df + t 1 β, s,df µ N 1 = 1 2 t 1 α 2,0,df + t 1 β, s,df µ N 2 = 2 2 where 1 = and 2 =. s1(s 1+s 2) s2(s 1+s 2) (9) 1. Sample size for the mean June 23, / 57

119 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

120 Equivalence test for one mean Example 7 We are manufacturing tablets with a target amount of drug of 250 mg. We have introduced a new manufacturing process, and we want to show that the new system is equivalent to the old system. How many tablets we need to analyze in order to show that both system perform equally? Presume that the variance of the new manufacturing process has to be estimated from the data itself. We want to detect departures from the mean of at least 2.5mg (δ = 2.5mg), and the standard deviation of the old manufacturing process is 3mg. We want to have a power of 90% if the difference is larger than = 5 mg. Confidence level=95%. Solution: This is an equivalence test with hypotheses H 0 : µ 250 H A : µ = Sample size for the mean June 23, / 57

121 Equivalence test for one mean Actually, the hypotheses tested are H 0 : µ µ 0 > δ H A : µ µ 0 δ where δ is a deviation from µ 0. This is equivalent to H 0 : µ µ 0 < δ or µ µ 0 > δ H A : µ µ 0 δ and µ µ 0 δ Applying the Two One-Sided Tests (TOST) methodology, we decompose the hypotheses above into two subproblems H 01 : µ µ 0 < δ H 02 : µ µ 0 > δ H A1 : µ µ 0 δ H A2 : µ µ 0 δ 1. Sample size for the mean June 23, / 57

122 Equivalence test for one mean 1. Sample size for the mean June 23, / 57

123 Equivalence test for one mean Known variance: ( ) 2 z1 α + z 1 β N = (10) Unknown variance: t 1 α,0,n 1 + t 1 β, +δ s N = N,N 1 2 (11) where = δ σ is the normalized effect size. 1. Sample size for the mean June 23, / 57

124 Equivalence test for one mean Example (continued) Substituting in our example: s guess = 3, = = 0.83 ( t0.95,0,n 1 + t 0.9, 0.83 N = 0.83 N,N 1 ) 2 N = Sample size for the mean June 23, / 57

125 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

126 Equivalence test for two means Example 8 We want to show the bioequivalence of two different drugs, i.e., the mean effect of the two drugs are similar. The diastolic blood pressure taking a reference drug is about 96 mmhg, and with an experimental drug, it is presumed to be similar. The population variation is σ = 18 mmhg. The two drugs are supposed to be similar if their difference is smaller than = 19.2 = 20% 96. α = 0.05, β = 0.1. The minimum detectable difference should be δ = 10 mmhg. Solution: This is an equivalence test with hypotheses H 0 : µ 1 µ 2 0 H A : µ 1 µ 2 = 0 1. Sample size for the mean June 23, / 57

127 Equivalence test for two means Sample size formulas are the same as in the case of the significance tests (Eq. 7, Eq. 8, and Eq. 9) but substituting by δ and α 2 by α. Example (continued) α = 0.05 β = 0.1 s guess = 18 = δ s guess = = = s s µ guess N df = 2(N 1). Finally we need to solve for N the equation t 1 α,0,df + t 1 β, s,df µ N = 2 2 N = Sample size for the mean June 23, / 57

128 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

129 1-way ANOVA contrasts Example 9 We are studying the effect of two different drugs on the blood pressure of patients. We have three study groups: placebo (drug 0), drug 1 and drug 2. We wonder which is the sample size for each group if we want to test: 1 There is no difference between placebo and the other two treatments. 2 There is no difference between the two treatments Let us assume that the variance of the population for each of the treatments is σ = 18 mmhg, and that we want to be able to detect effect sizes of = 10 mmhg. Solution: The two problems above can be addressed through an ANOVA contrast given by 1 µ 0 µ1+µ2 2 = 0 2 µ 1 µ 2 = 0 1. Sample size for the mean June 23, / 57

130 1-way ANOVA contrasts The following table shows an example of the expected measurements observed for the balanced (all cells have the same size) experiment. Placebo Drug 1 Drug 2 x 01, x 02,..., x 0N x 11, x 12,..., x 1N x 21, x 22,..., x 2N N N N T N ˆµ 0 = 1 N x 0j ˆµ 1 = 1 N x 1j ˆµ 2 = 1 N x 2j ˆµ = 1 TN j=1 Each observation is modelled as j=1 x ij = ˆµ + α i + ɛ ij The α s are the effect of each of the treatments and it is calculated as Note that the sum of α i s is 0 and that α i = ˆµ i ˆµ ˆµ i = ˆµ + α i j=1 x ij i=1 j=1 1. Sample size for the mean June 23, / 57

131 1-way ANOVA contrasts The ANOVA table accounts for the contributions of the different sources of variation to the total variation. The total variation is measured by the variance of all observations. This variance is decomposed as the sum of different sources SS tot = SS α + SS ɛ T N (x ij ˆµ ) 2 = i=1 j=1 T T N (ˆµ i ˆµ ) 2 + (x ij ˆµ i ) 2 i=1 i=1 j=1 Source SS MS F df Total SS tot df tot = NT 1 ɛ SS ɛ MS ɛ = SSɛ df ɛ df ɛ = T (N 1) α SS α MS α = SSα df α F = MSα MS ɛ df α = T 1 Example 45 = Treatments are significant 45 = Treatments are not significant 1. Sample size for the mean June 23, / 57

132 1-way ANOVA contrasts The following formulas show the true underlying contrast and how it can be estimated T T µ c = c i µ i ˆµ c = c i µ ˆ i i=1 The variance of the estimate is given by i=1 Var{ˆµ c } = T i=1 c 2 i σ 2 ɛ N i If all cells have the same number of observations N i = N Var{ˆµ c } = σ2 ɛ N T i=1 c 2 i 1. Sample size for the mean June 23, / 57

133 1-way ANOVA contrasts We may design the sample size through a one-sample mean design with the hypotheses: H 0 : µ c = 0 H A : µ c 0 This is equivalent to the derivation of Eqs. 1 and 2 to give N = ( z1 α 2 + z 1 β ) 2 (12) or t 1 α + t 2,0,dfɛ 1 β, s,df ɛ N = N 2 (13) with = σ ɛ T i=1 c 2 i or = s ɛ T i=1 c 2 i. 1. Sample size for the mean June 23, / 57

134 1-way ANOVA contrasts Example (continued) Design for µ 0 µ1+µ2 2 = 0 10 = ( 12) 2 +( 12) = N = ( z z 0.9 ) = N = 52 Design for µ 1 µ 2 = 0 10 = = ( 1) 2 N = ( z z 0.9 ) = N = 69 The most limiting comparison is the second one, so we use N = Sample size for the mean June 23, / 57

135 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

136 Multiple testing correction Example 10 Let us suppose that we are screening 1000 compounds vs a control, and that we have a confidence level of 95% (α = 0.05) in each test. Let us assume that none of the compounds is effective for our disease. However, on average, we will reject the null hypothesis for 50 compounds, and we will incorrectly assume that these 50 compounds make a difference with respect to the control. This problem is known as multiple testing. For K comparisons: Bonferroni correction: Dunn-Sidak correction: α = α family K (14) α = 1 (1 α family ) (1/K) (15) 1. Sample size for the mean June 23, / 57

137 Multiple testing correction When analyzing ANOVA data: ( ) 1 T If we are to compare all effects vs all others: K = = 2 2 If we are to compare all effects vs control: K = T 1 ANOVA actually tests H 0 : µ 1 = µ 2... = µ T H A : At least one µ i is different from the rest T (T 1) 2 Once the ANOVA test fails (not all treatments are the same), post-hoc comparisons are used. Remind to use a method that compensates for the multiple comparisons All vs all: Tukey s Honestly Significant Difference All vs control: Dunnett s test All vs best: Hsu s test Unplanned constrasts: Scheffé s test 1. Sample size for the mean June 23, / 57

138 All vs control Suppose we have T treatments (Groups 1, 2,..., T ) that will be compared to a control group (Group 0). These are contrasts of the form µ 0 µ 1 = 0 µ 0 µ 2 = 0... µ 0 µ T = 0 The variance of the contrast estimate is Var{ˆµ c } = σ 2 ɛ ( 1 N N i ) 1. Sample size for the mean June 23, / 57

139 All vs control This variance is minimized (subject to N 0 + TN i = ct) for N 0 = N i T (16) and N i given by the design formulas for ANOVA contrasts (Eqs. 12 and 13) with = ) or = ). 1 1 T T ( σ ɛ N 1+ 1 i ( s ɛ N 1+ 1 i Remind to correct α in some way (Bonferroni, Sidak,...) to account for the multiple (T ) comparisons, and that the number of degrees of freedom for ɛ of this design is df ɛ = (N 0 1) + T (N i 1) 1. Sample size for the mean June 23, / 57

140 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

141 Summary N = ( z1 α 2 + z 1 β ) 2 = σ (17) 1 One-sided or two-sided alternative test? 2 1 or 2 independent samples? 3 2 samples that can be reduced to 1 independent sample? (paired, ANOVA contrasts,...) 4 Known or unknown variance? 5 Equivalence tests significance tests 6 Increase group size if more variance N 2 = N 1 σ 2 σ 1. 7 Decrease α for multiple comparisons. 8 Increase control size N 0 = N i T. 1. Sample size for the mean June 23, / 57

142 Outline 1 Sample size for the mean A single sample with known variance A single sample with unknown variance Paired samples Two-samples with known variance Two-samples with unknown variance Equivalence test for one mean Equivalence test for two means ANOVA contrasts Multiple testing correction Summary 1. Sample size for the mean June 23, / 57

143 Chapter 2. Sample size for the variance C.O.S. Sorzano National Center of Biotechnology (CSIC) November 24, Sample size for the variance November 24, / 22

144 Outline 1 Sample size for the variance Variance: One sample Variance: Two samples Coefficient of variation: One sample Summary 2. Sample size for the variance November 24, / 22

145 Outline 1 Sample size for the variance Variance: One sample Variance: Two samples Coefficient of variation: One sample Summary 2. Sample size for the variance November 24, / 22

146 Variance: One sample Example 11 Solution: Let us assume that we are manufacturing syrup with 3 mg/ml of a drug. The standard deviation of the manufacturing process is meant to be 0.1 mg/ml, and the deviations from the target amount follows a Gaussian distribution. How many samples do we have to screen if we want to detect an increase from the target standard deviation of R = 50%, with a power of 80% and a confidence level of 95%? H 0 : σ 2 = H A : σ 2 > Sample size for the variance November 24, / 22

147 Variance: One sample This test is of the form If H 0 is true, then the statistic H 0 : σ 2 = σ 2 0 H A : σ 2 > σ 2 0 χ 2 = (N 1)s2 σ 2 0 is distributed as a χ 2 distribution with N 1 degrees of freedom. 2. Sample size for the variance November 24, / 22

148 Variance: One sample 2. Sample size for the variance November 24, / 22

149 Variance: One sample The sample size is the smallest N satisfying 1 N 1 σ2 0χ 2 1 α,n 1 1 N 1 σ2 1χ 2 β,n 1 χ 2 1 α,n 1 χ 2 β,n 1 σ2 1 σ 2 0 (1) when N is large, an approximation (delta method) is N = 1 2 ( z 1 α + z 1 β ln σ1 σ 0 ) 2 (2) 2. Sample size for the variance November 24, / 22

150 Variance: One sample Example (continued) The exact design gives χ ,N 1 χ 2 0.2,N 1 (1.5) 2 N = 19 The approximate one N = 1 2 ( ) 2 z z 0.8 = 18.8 N = 19 ln(1.5) 2. Sample size for the variance November 24, / 22

151 Variance: One sample Variance increases: One-sided H 0 : σ 2 = σ 2 0 H A : σ 2 > σ 2 0 χ 2 1 α,n 1 χ 2 β,n 1 σ2 1 σ 2 0 (3) Variance decreases: One-sided H 0 : σ 2 = σ 2 0 H A : σ 2 < σ 2 0 χ 2 α,n 1 χ 2 1 β,n 1 σ2 1 σ 2 0 (4) Variance changes: Two-sided H 0 : σ 2 = σ 2 0 H A : σ 2 σ 2 0 max { χ 2 1 α 2,N 1 χ 2 β,n 1 σ2 1 σ0 2, } χ 2 α 2,N 1 χ 2 σ2 1 1 β,n 1 σ0 2 (5) 2. Sample size for the variance November 24, / 22

152 Outline 1 Sample size for the variance Variance: One sample Variance: Two samples Coefficient of variation: One sample Summary 2. Sample size for the variance November 24, / 22

153 Variance: Two samples Example 12 We are considering buying a new analytical machine. We wonder if the new machine reduces the variance of the measurements by at least 50%. How many samples should we evaluate with each one of the machines? α = 0.05, β = 0.1. Solution: The statistic F = s2 1 /σ2 1 s 2 2 /σ2 2 H 0 : σ 2 1 = σ2 2 H A : σ 2 1 > σ2 2 is distributed as a Snedercor s F with ν 1 = N 1 1 and ν 2 = N 2 1 degrees of freedom. 2. Sample size for the variance November 24, / 22

154 Variance: Two samples 2. Sample size for the variance November 24, / 22

155 Variance: Two samples If the two groups have the same size, N 1 = N 2 = N, then the sample size is the smallest N satisfying F 1 α,n 1,N 1 F β,n 1,N 1 σ 2 1 σ 2 2 F 1 α,n 1,N 1 F β,n 1,N 1 σ2 1 σ 2 2 (6) when N is large, an approximation (delta method) is N = ( z 1 α + z 1 β ln σ1 σ 2 ) 2 (7) 2. Sample size for the variance November 24, / 22

156 Variance: Two samples Example (continued) The exact design gives F 0.95,N 1,N 1 F 0.1,N 1,N 1 2 N = 74 The approximate design gives ( ) 2 z z 0.9 N = ln = 71.3 N = 72 2 We need to measure N = 74 samples from each analytical machine to determine if the new one reduces the variance by at least a factor 0.5. Note that it is assumed that all measurements within each machine have the same mean. 2. Sample size for the variance November 24, / 22

157 Outline 1 Sample size for the variance Variance: One sample Variance: Two samples Coefficient of variation: One sample Summary 2. Sample size for the variance November 24, / 22

158 Coefficient of variation CV = σ µ 2. Sample size for the variance November 24, / 22

159 Coefficient of variation: One sample Example 13 Guideline on bioanalytical method validation. European Medicines Agency (2011). The Committee for Medicinal Products for Human Use (CHMP) allows an analytical error of 15%. How many samples do I need to take to check if our measurements are compliant? We would like to have a power of 90% at CV=20%. Solution: H 0 : CV 0.15 H A : CV > Sample size for the variance November 24, / 22

160 Coefficient of variation: One sample The statistic ĈV = sˆµ is, for large N, approximately distributed as a Gaussian with mean σ µ and standard deviation (delta method) σĉv = ĈV ĈV 2 + 1/2 N 2. Sample size for the variance November 24, / 22

161 Coefficient of variation: One sample The condition implies CV 0 + z 1 α σ CV0 CV 1 z 1 β σ CV1 N ( ) 2 z 1 α CV 0 CV /2 + z 1 β CV 1 CV /2 (8) CV 1 CV 0 Example 14 CV 0 = 0.15 CV 1 = 0.2 α = 0.05 β = 0.1 N = 53.8 N = Sample size for the variance November 24, / 22

162 Outline 1 Sample size for the variance Variance: One sample Variance: Two samples Coefficient of variation: One sample Summary 2. Sample size for the variance November 24, / 22

163 Summary Exact formulas are non-trivial Approximations exist, e.g., two samples: N = ( z 1 α + z 1 β ln σ1 σ 2 ) 2 If σ 1 and σ 2 are close to each other (σ 1 /σ 2 1), then N rapidly increases. 2. Sample size for the variance November 24, / 22

164 Outline 1 Sample size for the variance Variance: One sample Variance: Two samples Coefficient of variation: One sample Summary 2. Sample size for the variance November 24, / 22

165 Chapter 3. Sample size for proportions C.O.S. Sorzano National Center of Biotechnology (CSIC) September 2, Sample size for proportions September 2, / 78

166 Outline 1 Sample size for the proportion One sample: Proportion interval (from a large population) One sample: Test on proportion (from a large population) One sample: Proportion interval (from a small population) One sample: Test on proportion (from a small population) Two samples: Confidence interval on the difference Two samples: Confidence interval on the Risk Ratio Two samples: Confidence interval on the Odds Ratio Two samples: Test on two proportions (Fisher s exact test) Two samples: Test on two proportions (Normal approximation) Two samples: Test on two proportions (Arc sine approximation) Two samples: Test on two proportions (Risk Ratio method) Two samples: Test on two proportions (Odds Ratio method) Two samples: Test on two paired proportions (McNemar s method) One sample: Equivalence tests Two samples: Equivalence tests C samples: Two way R C contingency tables (χ 2 tests) Summary 3. Sample size for proportions September 2, / 78

167 Outline 1 Sample size for the proportion One sample: Proportion interval (from a large population) One sample: Test on proportion (from a large population) One sample: Proportion interval (from a small population) One sample: Test on proportion (from a small population) Two samples: Confidence interval on the difference Two samples: Confidence interval on the Risk Ratio Two samples: Confidence interval on the Odds Ratio Two samples: Test on two proportions (Fisher s exact test) Two samples: Test on two proportions (Normal approximation) Two samples: Test on two proportions (Arc sine approximation) Two samples: Test on two proportions (Risk Ratio method) Two samples: Test on two proportions (Odds Ratio method) Two samples: Test on two paired proportions (McNemar s method) One sample: Equivalence tests Two samples: Equivalence tests C samples: Two way R C contingency tables (χ 2 tests) Summary 3. Sample size for proportions September 2, / 78

168 One sample: Proportion interval (from a large population) Example 15 How many mice do we need to prove that if they are vaccinated against a certain disease with a new vaccine we are developing, the proportion of them that get infected when exposed to the microbial agent is less than 1%? (α = 0.05). Solution: H 0 : p > 0.01 H A : p 0.01 Let s say we study N mice and X of them get infected. If the probability of infection is p U, then this occurs according to a Binomial distribution ( ) N Pr{infected = X} = p X X U (1 p U ) N X 3. Sample size for proportions September 2, / 78

169 One sample: Proportion interval (from a large population) Example (continued) The following figure shows the binomial distribution for N = 300 and p U = If the number of observed infected animals were X, we would reject H 0 if X ( ) N 1 p x x U(1 p U ) N x 1 α (1) x=0 3. Sample size for proportions September 2, / 78

170 One sample: Proportion interval (from a large population) In these designs, we have to fix the number of allowed infected observations, X. The smallest N is obtained for X = 0. In this case, the equation above reduces to 1 (1 p U ) N 1 α N log(α) log(1 p U ) (2) That is, if we test N individuals and we observe X = 0 infected ones, then we reject H 0. If we observe X > 0, we cannot reject H 0. Eq. 1 is valid for any X. However, it is not easy to solve for N if X > 0. If X N, then we can make use of the fact that p = X N 1 2N χ2 2(X+1) That is, p related to a χ 2 distribution with 2(X + 1) degrees of freedom. For X = 0, it gives the smallest N that is N χ2 1 α,2 2p U (3) 3. Sample size for proportions September 2, / 78

171 One sample: Proportion interval (from a large population) Example (continued) In our example, the exact method gives N log(0.05) = N = 299 log(1 0.01) and the approximate one N χ2 0.95,2 2p U = p U = N = 300 This latter equation is the famous rule of 3 used in clinical trials. Remind that to reject H 0 with this N = 300, we cannot see any infected animal amongst the Sample size for proportions September 2, / 78

172 One sample: Proportion interval (from a large population) Example 16 We would like to determine the proportion of heavy smokers (>5 cigarretes per day) developing lung cancer between an age of with a precision of = 0.02 and a confidence level of 95% (we expect it to be about p 0 = 15%). How many samples we need for that? Solution: We should find an interval of the form [p L, p U ]. 3. Sample size for proportions September 2, / 78

173 One sample: Proportion interval (from a large population) Example (continued) Every time we randomly draw an individual (a smoker between 40 and 50 years old), the probability of having developed a lung cancer is about p 0 = 0.32M 2.1M 0.15 If we study N individuals, which is the probability of observing X lung cancers ( N Pr{cancer = X} = p X) 0 X (1 p 0 ) N X 3. Sample size for proportions September 2, / 78

174 One sample: Proportion interval (from a large population) Example (continued) If we study N = 20, the probability of observing X lung cancers (p = 0.15) is 3. Sample size for proportions September 2, / 78

175 One sample: Proportion interval (from a large population) Example (continued) But our problem is just the opposite, we have observed, let s say, X = 3 in N = 20 ˆp = 3 20 = 0.15 and we wonder what is the confidence interval for p. In this case p [0.03, 0.38] (see below why). Exact solution: Binomial, Two-sided We need to find p L and p U such that ( ) N N p x=x x L x(1 p L) N x = α 2 ( ) X N p x=0 x U x (1 p U) N x = α 2 (4) 3. Sample size for proportions September 2, / 78

176 One sample: Proportion interval (from a large population) Exact solution: Binomial, One-sided If p is very low, we may want to construct an interval of the form [0, p U ] X x=0 ( ) N p x U x (1 p U) N x = α (5) Alternatively, if p is very high, we may want to construct it of the form [p L, 1] N x=x ( ) N p x L x(1 p L) N x = α (6) 3. Sample size for proportions September 2, / 78

177 One sample: Proportion interval (from a large population) To solve for the sample size, which is actually our problem, we would need to solve the equations above for N, which is not an easy task because they are non-linear and because p L and p U are unknown. Approximate solution: Normal approximation If Np > 5 and N(1 p) > 5 (this normally happens for middle range p ( )), then we may use a normal approximation to the binomial distribution This translates into x Np Np(1 p) N(0, 1) ( p N ˆp, ) ˆp(1 ˆp) N Our problem (estimating p within a range of ± with a confidence level of 1 α) would be = z 1 α 2 ˆp(1 ˆp) N N z 1 α 2 ˆp(1 ˆp) 2 (7) 3. Sample size for proportions September 2, / 78

178 One sample: Proportion interval (from a large population) Example (continued) In our example: N ( z ) 2 = N = Sample size for proportions September 2, / 78

179 Correction for a not so large population The methods in this section assume that the population is so large, that the probability p does not change when we sample without replacement. If N > 10%N population, then this assumption does not hold. By this large sample, we are significantly reducing the uncertainty about the population, and the sample size should be corrected to = z 1 α 2 ˆp(1 ˆp) N = N N 1 + N 1 N population (8) Example (continued) Let s imagine that our whole population of smokers between 40 and 50 years old is only of 10,000 people. Our sample N = 1, 225 is larger than 10% of the whole population, so we correct N to be N = = N = Sample size for proportions September 2, / 78

180 Outline 1 Sample size for the proportion One sample: Proportion interval (from a large population) One sample: Test on proportion (from a large population) One sample: Proportion interval (from a small population) One sample: Test on proportion (from a small population) Two samples: Confidence interval on the difference Two samples: Confidence interval on the Risk Ratio Two samples: Confidence interval on the Odds Ratio Two samples: Test on two proportions (Fisher s exact test) Two samples: Test on two proportions (Normal approximation) Two samples: Test on two proportions (Arc sine approximation) Two samples: Test on two proportions (Risk Ratio method) Two samples: Test on two proportions (Odds Ratio method) Two samples: Test on two paired proportions (McNemar s method) One sample: Equivalence tests Two samples: Equivalence tests C samples: Two way R C contingency tables (χ 2 tests) Summary 3. Sample size for proportions September 2, / 78

181 One sample: Test on proportion (from a large population) Example 17 The probability of suffering from De Quervain tenosynovitis (Blackberry finger) in the general population is p 0 = It is suspected that amongst heavy smartphone users, this probability is larger. How many subjects (heavy smartphone users) do we need to study to determine if this is true? We want a power of 0.9 when p is at least α = Solution: Our test is of the form H 0 : p = p 0 h A : p > p 0 3. Sample size for proportions September 2, / 78

182 One sample: Test on proportion (from a large population) We need to find N and X such that x=x+1 ( ) N p x x 0 (1 p 0 ) N x = α and X x=0 ( ) N p x x 1 (1 p 1 ) N x = β (9) 3. Sample size for proportions September 2, / 78

183 One sample: Test on proportion (from a large population) Example (continued) In our example p 0 = 0.01 and p 1 = The solution is N = 390, X = 7 That is, we will evaluate 390 individuals. If the number of individuals with De Quervain syndrome is 7 or less, we cannot reject the null hypothesis (H 0 : p = p 0 ). 3. Sample size for proportions September 2, / 78

184 One sample: Test on proportion (from a large population) If the Gaussian approximation is valid, we can approximate N by N ( ) 2 z 1 α p0 (1 p 0 ) + z 1 β p1 (1 p 1 ) (10) p 1 p 0 If p 0 p 1, then we can further simplify to N z 1 α + z 1 β p(1 p) 2 (11) with p = p0+p Sample size for proportions September 2, / 78

185 One sample: Test on proportion (from a large population) Example (continued) In the example or N N ( ) 2 z z = N = 366 z z ) 2 = N = 420 Note that we are not meeting the conditions for the Gaussian approximation since Np 0 = = 3.66 < 5 or Np 0 = = 4.20 < Sample size for proportions September 2, / 78

186 Outline 1 Sample size for the proportion One sample: Proportion interval (from a large population) One sample: Test on proportion (from a large population) One sample: Proportion interval (from a small population) One sample: Test on proportion (from a small population) Two samples: Confidence interval on the difference Two samples: Confidence interval on the Risk Ratio Two samples: Confidence interval on the Odds Ratio Two samples: Test on two proportions (Fisher s exact test) Two samples: Test on two proportions (Normal approximation) Two samples: Test on two proportions (Arc sine approximation) Two samples: Test on two proportions (Risk Ratio method) Two samples: Test on two proportions (Odds Ratio method) Two samples: Test on two paired proportions (McNemar s method) One sample: Equivalence tests Two samples: Equivalence tests C samples: Two way R C contingency tables (χ 2 tests) Summary 3. Sample size for proportions September 2, / 78

187 One sample: Proportion interval (from a small population) Example 18 We have received in the lab a batch of 50 bottles of the same reagent. How many bottles do we have to test if we want to show that the total number of defective bottles is at most 2 with a confidence level of 95%? Solution: We want to show that Pr{0 S 2} 0.95 Let s say we receive N T bottles, out of which S are defective. We study N, the probability of observing X defective follows an hypergeometric distribution ( ) ( ) S NT S X Pr{defective = X} = ( NT N N X ) 3. Sample size for proportions September 2, / 78

188 One sample: Proportion interval (from a small population) This problem is of the form Pr{0 S S U } α Exact solution: In a zero-acceptance plan, we need to find N such that in the worse case (S = S U ) ( ) NT S U Pr{X = 0} = N ( ) NT α (12) N This equation is trascendental and not easy to solve manually. 3. Sample size for proportions September 2, / 78

189 One sample: Proportion interval (from a small population) Binomial approximation for small samples: If N < 10%N T, then we can apply the small sample approximation of the hypergeometric distribution by a binomial ( Hypergeometric(N T, S U, N) Binomial N, p = S ) U N T Then ( ) ( N SU Pr{X = 0} = 0 N T ) 0 ( 1 S ) N U N N T log(α) ( ) (13) log 1 S U N T 3. Sample size for proportions September 2, / 78

190 One sample: Proportion interval (from a small population) Binomial approximation for rare events: If S U < 10%N T, then we can apply the small sample approximation of the hypergeometric distribution by a binomial ) Hypergeometric(N T, S U, N) Binomial (S U, p = NNT Then ( ) ( SU N Pr{X = 0} = 0 N T ) 0 ) (1 NNT SU ( ) N N T 1 α 1/S U (14) 3. Sample size for proportions September 2, / 78

191 One sample: Proportion interval (from a small population) Example (continued) Solution: Small sample: N log(0.05) log(1 50) = 74 2 But we violate the hypothesis of small sample!! Rare events: N 50( /2 ) = N = 39 The hypothesis of rare event is fulfilled because 2 10%50. Exact: N = Sample size for proportions September 2, / 78