Study Design Sample Size Calculation & Power Analysis. RCMAR/CHIME April 21, 2014 Honghu Liu, PhD Professor University of California Los Angeles

Size: px

Start display at page:

Download "Study Design Sample Size Calculation & Power Analysis. RCMAR/CHIME April 21, 2014 Honghu Liu, PhD Professor University of California Los Angeles"

Stewart Holmes
8 years ago
Views:

1 Study Design Sample Size Calculation & Power Analysis RCMAR/CHIME April 21, 2014 Honghu Liu, PhD Professor University of California Los Angeles

2 Contents 1. Background 2. Common Designs 3. Examples 4. Computer Software 5. Summary & Discussion

3 Background Flow of Study Design & Its Process Research Question Study Aims Hypotheses Big topic Sample Data Collection Select sample and gather data Specific goals Sample Size Calculation & Power Analysis Determine the sample size Theories/Speculation about aims Target Population Inference to Analysis Conclusion Obtain results 3

goals Sample Size Calculation & Power Analysis Determine the sample size

4 Background Power of a statistical test The probability that it will yield statistically significant results Sample size The minimum sample size required to detect a certain difference between parameters Sample size and statistical power are linked with study aims and hypothesis Need to collect sample data to test the hypothesis Hypotheses are tested with certain power

difference between parameters Sample size and statistical power are linked with study aims

5 Key Concepts (a) Hypothesis Null: H 0 : µ = µ 1 2 Alternative: H a: 2 Background (cont.) µ ( 2 1 µ µ > or µ < µ 1 2 ) 1 µ (b) Type I error (α ) ---Reject a null hypothesis when it is true: α = Prob( rejecting H 0 H0 is true)

6 Background (cont.) (c) Type II error (β ) --Accept a null hypothesis when it is false: β = Prob( accepting H 0 H0 is false) (d) Power --The probability of rejecting a null hypothesis when it is false: Power= Pr( rejecting H 0 H is false)=1 β 0

7 (e) Effect size Background (cont.) --The difference between parameters to be tested (e.g., = µ µ 1 2). --can be expressed as per standard deviation (e.g., ES= / σ = ( )/ ) ) µ µ σ 1 2 (f) Critical value --The deviate of a distribution that reaches statistical significance under the null hypothesis for a given type I error (e.g., z 1 α =1.645 and 1 α / 2 z =1.96)

, ES= / σ = ( )/ ) ) µ µ σ 1 2 (f) Critical value --The deviate of a distribution that

8 Background (cont.) (g) One-sided vs. Two-sided test --One-sided test: a null hypothesis can only be rejected in one direction (directional test.) (e.g., reject if z > z 1 α ) --Two-sided test: a null hypothesis can be rejected in either direction. (e.g., reject if z > z1 α / 2 )

9 Background (cont.) (h) Acceptance region & rejection region --Acceptance region: the null hypothesis will be accepted for all values that fall into this region (e.g., z <= z<= z 1 α / 2 1 α / 2 ) --Rejection region: the null hypothesis will be rejected for all values that fall into this region z < z or z > z (e.g., 1 α / 2 1 α / 2 )

will be accepted for all values that fall into this regi

10 Normal Distribution with One-sided Test and Type I Error 0.05 Power Acceptance region Rejection region

11 Five Key Factors 1. Sample size 2. Effect size 3. Significance level 4. Power of the test 5. Variability

12 I. Continuous measure a) One sample normal H µ 0 n 0 : Common Designs µ = : = + β 1 α / 2 H µ 1 1 µ = (( z z )/( / s)) 1 Where = µ µ 1 0 is the difference to be detected; S is the standard deviation; z 1 β and z 1 α / 2 are the normal deviates for desired power and significance level. Note: this is also the sample size for the case of paired observations. 2

standard deviation; z 1 β and z 1 α / 2 are the normal deviates for desired power and

13 b) Two sample normal H µ : Common Designs (cont.) µ = : H µ µ n (( z + z )/( / s)) 2/(1 1/ r) 1 1 β + = 1 α / 2 n = r*n 1 with 0<r 1, Where 2 is the difference to be detected; S is the common standard deviation; z 1 β and z 1 α / 2 are the normal deviates for desired power and significance level.

14 c) Two group repeated measures (time-averaged means) (Diggle, et al, 1994) H µ : µ =, : µ H µ m = 2( z z ) 2 σ 2{1 ( n 1) ρ} / nd 2 α β = 2( z + ) 2 {1 + ( 1) } / α z n ρ n β Where z α and z β are the normal deviates; σ 2 is the common variance; ρ =Corr( y ij, y ) ik is the intra-patient correlation; d is the difference between the average response of two groups. Note: Unbalanced design: Liu, et al Journal of Modern Applied Statistics; PASS 2008, NCSS.

is the common variance; ρ =Corr( y ij, y ) ik is the intra-patient correlation; d is the difference between the average

15 II. Binomial distribution n a) One sample binomial H p0 0 : p = : H p= p1 1 = [{ z + z * sqrt( p *(1 p )/ p /(1 p ))}/( p p ) 1 β 1 α / * p *(1 p ) 1 1 Where p 0 is the null value of the probability; p 1 is the alternative value of the probability; z and z 1 α / 2 are the normal deviates for desired 1 β power and significance level.

0 is the null value of the probability; p 1 is the alternative value of the

16 b) Two sample binomial H : p n p = : H p p = p 1 1 α / 2 r+ {[ z + z * sqrt( p*(1 p)*(1/ r+ 1)/( p *(1 )/ 1 β 1 1 p *(1 p ))]/( p p )} 2*[ p *(1 p )/ r+ p *(1 )] p n = r*n 1 with 0<r 1 Where 2 p = ( p )/2 1 + p2 p 1 and p z 2 are the probability of groups 1 and 2; 1 β and z 1 α / 2 are the normal deviates for desired power and significance level.

2 p n = r*n 1 with 0<r 1 Where 2 p = ( p )/2 1 + p2 p 1 and p z 2 are the probability of

17 c) Two sample binomial repeated measures (Diggle, et al. 1994) H : p p = : (time-averaged proportions) H p p m = [ z {2 pq(1+ ( n 1) ρ)} / 2+ z {(1+ ( n α ( p + nd q p q )} 1/ 2] 2/ Where p = ( p + p 2 2 )/ 1 q =1 p 2 1) 1 ρ β ρ =Corr y ij, y ) is the intra-patient correlation ( ik d is the difference between the average response for the two groups. )

2+ z {(1+ ( n α ( p + nd q p q )} 1/ 2] 2/ 1 1 2 2 Where p = ( p + p 2 2 )/ 1 q =1 p 2 1) 1 ρ

18 III. Other Designs Matched case-control McNemar s test Analysis of variance (ANOVA) Correlation coefficient Logistic regression Multiple regression

19 III. Other Designs (cont.) Survey Design Methods» Stratification» Clustering» Complex Survey (stratification and clustering) Estimate Design Effect: deff = vav surveydesd vav ( ign) / ( srs) Sample Size=deff*(traditional sample size calculation formula)

Survey (stratification and clustering) Estimate Design Effect:

20 Examples Study A: Study Aim: To study the impact of a new physical therapy on quality of life of patients with chronic back pain Hypothesis: The new physical therapy can significantly improve the quality of life of patients with chronic back pain Step 1 Outcome measure: SF-12 physical component summary (PCS) score Step 2 Design: two group comparison between treatment and control Step 3-Statistical model: two group comparison with a continuous outcome measure

with chronic back pain Step 1 Outcome measure: SF-12 physical component summary (PCS) score Step 2 Design: two

21 Study A (cont.) Step 4 Obtain the required statistics for the statistical model: 1) type I error: ) type II error: 0.15 (power 85%) 3)mean of PCS : 44 4) SD of PCS: 22 5) effect size: 5 (minimally clinical meaningful difference) Step 5 Find a sample size calculation software and plug in the statistics and get the results: n=348 for each arm

22 Study B Study Aim: To study the difference in rate of participation in novel clinical trial of gene therapy/stem cell research between the Innovative Health Research Intervention (IHRI) and the Standard HIV Attention Control (AC) Hypothesis: IHRI has a higher participation rate than AC Step 1: Outcome measure willingness to participate (binary yes/no variable) Step 2: Design randomized two group comparison between IHRI and AC arms Step 3: Statistical Model two group comparison with binomial distribution

23 Study B (cont.) Step 4 Obtain the required statistics for the statistical model 1) Type I error: ) Type II error: 0.20 (power 80%) 3) The estimated rate of participation (60% for AC) 4) Sample size capacity: 180 in each arm 5) Effect size: to be determined Step 5 find a sample size and power analyses software and plug in the statistics and get the effect size: 15%.

24 Sample Size Calculation & Power Analyses Software General purpose statistical software (e.g., STATA, SPSS, SAS, GLIM, Sigmastat and XLISP_STAT) Special purpose statistical software (e.g., EpiInfo) Stand-alone sample size & power analysis software (e.g., NCSS-PASS, nquery and SYSTAT Design) Stand-alone sample size & power analysis software for specialized applications (PRESISION for survival studies) Software on Internet (e.g.,

25 Related key factors Summary & Discussion Min-Max rule Minimum required sample size for each main hypothesis Maximum sample size among the multiple minimums Practical factors that influence sample size determination Budget/sample limitation Backward estimation

26 Summary & Discussion (cont.) Find the necessary and right statistics e.g., mean, SD & ES Get multiple solutions and select the best design

27 References Jacob Cohen (1988). Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, Publishers. Hillsdale, New Jersey. Diggle, PJ, Liang, KY and Zeger, SL (1996). Analysis of Longitudinal Data. Oxford University Press Inc., New York. R. Barker Bausell and Yu-Fang Li (2002). Power analysis for experimental research. Cambridge University Press. Pass 2008 Power Analysis and Sample Size for Windows. NCSS, Kaysville, Utah. Liu HH and Wu TT. Sample size calculation and power analysis for Time-averaged difference. Journal of Modern Applied Statistical Methods. 2005;4(2): Helena Chmura Kraemer and Sue Thiemann (1987). How many subjects? Sage Publications, London. Liu HH & Wu TT. Sample Size Calculation and Power Analysis of Changes in Mean Response Over Time. Journal of communication in Statistics (in press) Sharon L. Lohr (1999). Sampling: Design and Analysis. Duxbury Press.

28 Questions? Honghu Liu, PhD

Consider a study in which. How many subjects? The importance of sample size calculations. An insignificant effect: two possibilities.

Consider a study in which. How many subjects? The importance of sample size calculations. An insignificant effect: two possibilities. Consider a study in which How many subjects? The importance of sample size calculations Office of Research Protections Brown Bag Series KB Boomer, Ph.D. Director, boomer@stat.psu.edu A researcher conducts