Bayesian trial monitoring: Separating the baby from the bathwater. John M. Kittelson, PhD University of Colorado Denver

Transcription

1 Bayesian trial monitoring: Separating the baby from the bathwater John M. Kittelson, PhD University of Colorado Denver 1

2 Outline 1. Motivation 2. PPV as the scientific objective 3. Bayesian family for clinical trial design 4. Trial monitoring 5. Flexible implementation 2

3 Outline 1. Motivation 2. PPV as the scientific objective 3. Bayesian family for clinical trial design 4. Trial monitoring 5. Flexible implementation Three foundational objectives: 1. Develop PPV as the scientific goal 2. Evaluate its use as a measure of the scientific impact of new statistical methods 3. Argue that statistical practice should be evaluated by its impact on PPV 3

4 1. Motivation Example: New treatment for hypertension in adolescents Setting: prevalence of hypertension and pre-hypertension is increasing. Better understanding and new therapeutic approaches are needed. Treatment: Allopurinol to lower uric acid Phase II/III design: - RCT: allopurinol vs placebo - Endpoint: success (normotensive at 12-months) - Treatment effect measure: θ P = probability of normotensive (placebo) θ A = probability of normotensive (allopurinol) θ = θ A θ P 4

5 1. Motivation - Structuring parameter space: Clinically Important Harm θ No Difference θ 0 Clinically Important Benefit θ + Inferiority Superiority Clinical Inferiority Clinical Superiority θ < θ θ > θ + - Mapping scientific question to parameter space: H 0 : θ θ 0 = 0.00 H + : θ θ + =

6 1. Motivation Frequentist design Statistical standard for scientific decision: - Based on sampling distribution: P r(ˆθ θ) - Rule out the hypothesis θ = θ ref if either P r(ˆθ ˆθ obs θ = θ ref ) < α P r(ˆθ ˆθ obs θ = θ ref ) < α with α = % CI denotes hypotheses than cannot be ruled out. Example: - If 2N = 1050 and θ P = 0.4, then under H + : ˆθ N ( θ, ) - 95% CI has width = 0.12; thus the design will discrminate between θ 0 and θ

7 1. Motivation Trial monitoring (scientific/clinical aspects) Rationale/need: - Ethical responsibility to trial participants - Ethical responsibility to participants who have yet to enroll - Responsibility to larger patient population - Scientific/financial efficiency Approach (frequentist): - Choose stopping boundaries to satisfy need for early conservatism - Search for critical values to maintain fixed-sample α-level - Flexible implementation while maintaining operating characteristics - Bias-adjustment of resulting point estimates 7

8 1. Motivation Trial monitoring (scientific/clinical aspects) Issues stemming from monitoring - Early termination means forgoing information on other endpoints - Multiple analyses alters the sampling density * Depends on stopping rules * Jump discontinuities * Not shift invariant * Altered operating characteristics - Biased point estimates E(ˆθ) θ OBF Pocock

9 Sampling density for OBF and Pocock 1-sided designs. OBF (theta = 0) OBF (theta = 1.96) Probability Density Probability Density X Pocock (theta = 0) X Pocock (theta = 1.96) Probability Density Probability Density X X 9

10 1. Motivation Bayesian monitoring Approach: - Likelihood principle: Information in the data about a parameter is specified in the likelihood - Posterior density does not depend on stopping rules - Posterior distribution can be computed at any interim analysis Some equivalences with frequentist designs: - Decisions criteria based on posterior induce critical values in the sample space - Bayesian decision criteria have frequentist operating characteristics - Frequentist decision criteria have Bayesian properties (which are a function of the prior) 10

11 1. Motivation Example Consider the following stopping rules (as z-scores): Pocock Design Bayesian Design* Analysis (j) a j d j a j d j Decide for benefit if ˆθ j > d j Decide for lack of benefit if ˆθ j a j * (non-informative prior) Operating characteristics: Pocock Bayes Size (α) Power (β)

12 1. Motivation Question What are the scientific implications of maintaining statistical operating characteristics when monitoring trials? 12

13 2. Positive predictive value in clinical trials: Characterizing the scientific objective There is no common objective: Pharmaceutical sponsor: * Trials are needed to find marketable drugs * Trial portfolio with high probability of positive result Academic investigator: * Novel hypotheses generate funding for trials * Need statistical significance to generate more novel hypotheses Patients with disease (or future patients): * Need the cure for their disease Regulatory Agencies (FDA) * Approve efficacious therapies * (Do not approve ineffective therapies.) 13

14 2. Positive predictive value in clinical trials: Characterizing the scientific objective The FDA s perspective best reflects the scientific objectives: * Maximize the chance of identifying efficacious therapies. * Minimize the risk of incorrectly identifying bad therapies as good. These objectives are implemented in the trial design, conduct, and analysis. 14

15 2. PPV and the scientific objective Measuring the objective The scientific objective of trial design, conduct, and analysis is to achieve a high prevalence of truly efficacious therapies among all that labelled as beneficial; i.e., a high positive predictive value. 15

16 More formally: 2. PPV and the scientific objective Measuring the objective * P P V = P r(truly effective positive trial) * Using Bayes rule: P P V = = θ>θ 0 P r(θ ˆθ > ˆθ c ) θ>θ 0 P r(ˆθ > ˆθ c θ)dp(θ) u P r(ˆθ > ˆθ c u)dp(u) where ˆθ > ˆθ c denotes a positive trial. * In this setting p(θ) represents the (unknowable) distribution of effects of all therapies that are might be evaluated in clinical trials. 16

17 2. PPV and the scientific objective Measuring the objective P P V is the p-weighted area under the power curve. Probability Power curve Distribution of Treatment effects θ 0 θ^c θ + θ 17

18 2. PPV and the scientific objective Measuring the objective It is not the posterior distribution: Probability Prior Likelihood Posterior θ 0 θ^c θ 18

19 2. PPV and the scientific objective Quantifying PPV Problem: p(θ) is impossible to know. Can we find an alternative quantification? A lower limit: * The power curve is non-decreasing; therefore: β(θ a)π ab where: Z θ a<θ θ b β(θ)p(θ) β(θ b )π ab β(θ) = P r(ˆθ ˆθ c θ) (power curve) π ab = Z b a p(θ) 19

20 * It follows that: 2. PPV and the scientific objective Quantifying PPV P P V = = β(θ)p(θ) β(θ)p(θ) θ 0<θ θ β(θ)p(θ) + β(θ)p(θ) θ 0<θ θ + θ +<θ β(θ)p(θ) + β(θ)p(θ) + β(θ)p(θ) θ θ 0 θ 0<θ θ + θ +<θ β(θ 0 )π 0+ + β(θ + )π + β(θ 0 )π 0 + β(θ 0 )π 0+ + β(θ + )π + β(θ + )π + β(θ + )π + + β(θ 0 )(1 π + ) 20

21 2. PPV and the scientific objective Relationship with diagnostic testing Recall diagnostic testing: * PPV represents P r(disease test postiive): π = disease prevalence Sn = P (test positive disease) = Sensitivity Sp = P (test negative no disease) = Specificity P P V = Sn π Sn π + (1 Sp)(1 π) 21

22 2. PPV and the scientific objective Relationship with diagnostic testing Recall diagnostic testing: * PPV represents P r(disease test postiive): π = disease prevalence Sn = P (test positive disease) = Sensitivity Sp = P (test negative no disease) = Specificity P P V = Sn π Sn π + (1 Sp)(1 π) * Lower limit for PPV as a scientific objective: π π + = prevalence of treatments with important benefit β β(θ +) = Power at θ + = Sensitivity α β(θ 0) = Power at θ 0 = 1 Specificity βπ P P V = βπ + α(1 π) 22

23 2. PPV and the scientific objective Properties of PPV (π 0 = 0.1) β = 0.8 α = P(p < 0.025) False positive fraction True positive fraction P(p < 0.025) False positive fraction True positive fraction α β 23

24 Increasing PPV: 2. PPV and the scientific objective Trial planning and PPV βπ P P V = βπ + α(1 π) * Increase π: - Hypothesis-driven research (avoid science by hunch ) - Careful planning of preliminary studies * Increase β: - Good practice (no missing data, low variation in outcome assessment, good adherence, etc.) - Increase sample size. * Reduce α - Pre-specify, pre-specify, pre-specify - Avoid subgroups * Staged evaluation (phase II as screening trials) 24

25 2. PPV and the scientific objective Example: 2-stage evaluation paradigms A sequence of trials produces higher PPV: * New therapies are screened in small initial studies * Second stage studies test therapies that are positive in stage 1 testing. * Set of therapies for second stage is enriched with efficacious treatments. Overall PPV: Stage 1 (P P V 1 ) : π 1 = Stage 2 (P P V 2 ) : π 2 = π 0 β 1 π 0 β 1 + (1 π 0 )α 1 π 1 β 2 π 1 β 2 + (1 π 1 )α 2 25

26 2. PPV and the scientific objective Motivating example (revisited) Consider the following stopping rules (as z-scores): Pocock Design Bayesian Design* Analysis (j) a j d j a j d j * (non-informative prior) Operating characteristics: Pocock Bayes Size (α) Power (β)

27 2. PPV and the scientific objective Motivating example (revisited) If π = 0.1: P P V = βπ βπ + α(1 π) Pocock P P V = Bayes P P V = = =

28 Motivating example (revisited) PPV is improved by controlling operating characteristics: π = 0.05 π = 0.1 Positive Predictive Value π = π = 0.4 Type I error controlled Not controlled Number of interim analyses 28

29 3. Bayesian design family for trial monitoring Purpose: * Allow Bayesian interpretation of stopping decisions. * Indexed by reference priors: - Decision for benefit must convince the reference pessimist - Decision against benefit must convince the reference optimist * Adequate information: P r(inconclusive result) = 0. P r(ambiguous result) = 0. * Utility of Bayesian interpretation: If a trial is supposed to inform practice, then it is useful to consider whether the trial design will provide enough information to inform the reference pessimist and optimist. * Optionally: Allow and facilitate control of β(θ 0 ) and β(θ + ). 29

30 3. Bayesian design family Review of Gaussian conjugate family Likelihood: Prior: ˆθ N ) (θ, σ2 N ) θ N (ξ, σ2 N 0 Posterior: θ ˆθ N (E, V ) with: E = N ˆθ + N 0 ξ N 0 + N σ 2 V = N 0 + N 30

31 3. Bayesian design family Reference prior distributions Optimist: * Prior: * Posterior mean: «θ N θ + + O, σ2 N 0 Pessimist: * Prior: * Posterior mean: E O = N ˆθ + N 0(θ + + O) N 0 + N «θ N θ 0 P, σ2 N 0 E P = N ˆθ + N 0(θ 0 P) N 0 + N 31

32 3. Bayesian design family Decision criteria Optimist: * Decide against benefit if P O(θ θ + ˆθ) > γ (think: γ 0.975). which implies: Pessimist: * Decide for benefit if ˆθ a J N 0 = θ + O N zγσ N0 + N N P P(θ θ 0 ˆθ) > γ (think: γ 0.975). which implies: ˆθ > d J N 0 = θ 0 + P N + zγσ N0 + N N 32

33 3. Bayesian design family Choosing trial information To reach a conclusive result we require that: * Upon completion the reference optimist and pessimist agree: - Impossible for the pessimist to conclude benefit and the optimist to conclude lack of benefit. - Impossible that neither reaches a conclusion. * As a consequence we require a J = d J : N 0 θ + O N z N0 + N γσ N so that the following equality must hold: N 0 = θ 0 + P N + z σ N 0 + N γ N (θ + θ 0 )N = 2z γ σ N 0 + N + ( O + P )N 0 33

34 3. Bayesian design family Choosing trial information (Repeating the equality:) (θ + θ 0 )N = 2z γ σ N 0 + N + ( O + P )N 0 * Fix z γ = 1.96 and σ; choose three of the following; calculate the fourth: (θ + θ 0 ) = Design resolution N = Trial sample size O + P = Reference optimist and pessimist N 0 = Degree of dogmatism in reference priors 34

35 3. Bayesian design family Example: Fixed-sample allopurinol trial Parameters: N 0 = 50; θ + θ 0 = 0.12, γ = 0.975;N = 525; O = P = Prior distributions: Note: Choosing O = P = 0 requires N =

36 3. Bayesian design family Example: Fixed-sample allopurinol trial Prior and likelihood (at critical value):

37 3. Bayesian design family Example: Fixed-sample allopurinol trial Prior and posteriors (at critical value):

38 4. Trial monitoring How should we monitor clinical trials to maintain operating characteristics (and PPV)? Classic development: * J interim analyses occur at N 1 < N 2 <... < N J subjects have completed. * Decision criteria and the jth interim analysis (j = 1,..., J): ˆθ > d j Decide for benefit ˆθ < a j Decide for lack of benefit with a j < d j and a J = d J. 38

39 Frequentist families: 4. Trial monitoring Design families * Emerson-Fleming one-sided symmetric designs: d j = θ 0 + G σ N J Π P j a j = θ + G σ N J Π P j where Π j = Nj N J ; P controls early conservatism; find G to control operating characteristics. * Others Whitehead Christmas tree designs Stochastic curtailment Conditional futility α-spending Unified family 39

40 A Bayesian family: 4. Trial monitoring Design families * Recall above fixed-sample decision criteria: N 0 d J = θ 0 + P N + z N0 + N γσ N N 0 a J = θ + O N z N0 + N γσ N * Rewriting and extending to interim analyses: ) d j = θ 0 + ( P Π 0 + G σnj Π0 + Π j Π P j ) a j = θ + ( O Π 0 + G σnj Π0 + Π j Π P j where Π 0 = N0 N J 40

41 4. Trial monitoring Design families Note similarites: * Emerson-Fleming one-sided symmetric designs: d j = θ 0 + G σ N J Π P j a j = θ + G σ N J Π P j * Bayesian design family: ) d j = θ 0 + ( P Π 0 + G σnj Π0 + Π j Π P j ) a j = θ + ( O Π 0 + G σnj Π0 + Π j Π P j 41

42 4. Trial monitoring Example: Allopurinol trial Mapping science to parameter space: Inferior θ < θ 0 = 0 Important benefit θ θ + = 0.12 Frequentist design: * With 2N = 1050 subjects a fixed-sample design discriminates between the above hypotheses. * Trial monitoring (P = 0.5: Pocock boundary shape): Analysis (j) a j d j Power at θ = 0.0: Power at θ = 0.12:

43 4. Trial monitoring Bayes monitoring * Recall decision criteria: «d j = θ 0 + PΠ 0 + G σnj p Π0 + Π j Π P j «a j = θ + OΠ 0 + G σnj p Π0 + Π j Π P j * Bayes decision criteria (P = 1, G = ): Pocock Bayes Analysis (j) a j d j a j d j P (θ > 0 d j) β(0.0) β(0.12)

44 4. Trial monitoring Bayes monitoring Comparing Bayes and Pocock designs Bayes (P=1) Pocock Difference in Proportions Sample size 44

45 4. Trial monitoring Bayes monitoring * Increasing early conservatism (P = 1.5, G = 2.001): O Brien-Fleming Bayes Analysis (j) a j d j a j d j P (θ > 0 d j) > β(0.0) β(0.12)

46 4. Trial monitoring Bayes monitoring Effect of early conservatism in Bayes family Bayes (P=1) Bayes (P=1.5) Difference in Proportions Sample size 46

47 5. Flexible implementation How to control operating characteristics when interim analyses differ from pre-trial plan: * α-spending functions (Lan & DeMets) * Operating characteristics are robust to relatively large changes (Emerson & Fleming) * Constrained boundaries (Burrington and Emerson) 47

48 5. Flexible implementation How to control operating characteristics when interim analyses differ from pre-trial plan: * α-spending functions (Lan & DeMets) * Operating characteristics are robust to relatively large changes (Emerson & Fleming) * Constrained boundaries (Burrington and Emerson) Example (Bayesian design for Allopurinol trial): * Pre-trial plan: Bayes Analysis (j) a j d j 1 (N = 105) (N = 210) (N = 315) (N = 420) (N = 525) β(0.0) β(0.12)

49 5. Flexible implementation Example (constrained boundaries) Suppose that the first interim analysis occurs at 158 instead of 105 subjects per group. * Calculate the decision rule that would have been used at Π 1 = 158/525 = 0.3 information time: d 1 = θ 0 + PΠ σ «Π = N J a 1 = θ + OΠ σ «Π = N J 49

50 5. Flexible implementation Example (constrained boundaries) Suppose that the first interim analysis occurs at 158 instead of 105 subjects per group. * Calculate the decision rule that would have been used at Π 1 = 158/525 = 0.3 information time: d 1 = θ 0 + PΠ σ «Π = N J a 1 = θ + OΠ σ «Π = N J * Assume there will then be one more analysis at the full sample size. Fix d 1 and a 1 as above. Find the new value for G to satisfy the operating characteristics: d J = θ 0 + PΠ σ «Π = N J a J = θ + OΠ σ «Π = N J 50

51 5. Flexible implementation Example (constrained boundaries) Suppose trial did not stop at first analysis and that the second analysis occurs at N 2 = 370 subjects. * Constrain d 1 = and a 1 = * Calculate the decision rule that would have been used at Π 2 = 370/525 = 0.7 information time: d 2 = θ 0 + PΠ σ «Π = N J a 2 = θ + OΠ σ «Π = N J 51

52 5. Flexible implementation Example (constrained boundaries) Suppose trial did not stop at first analysis and that the second analysis occurs at N 2 = 370 subjects. * Constrain d 1 = and a 1 = * Calculate the decision rule that would have been used at Π 2 = 370/525 = 0.7 information time: d 2 = θ 0 + PΠ σ «Π = N J a 2 = θ + OΠ σ «Π = N J * Assume there will then be one more analysis at the full sample size. Fix d 1, d 2, a 1, a 2 as above. Find the new value for G to satisfy the operating characteristics: d J = θ 0 + PΠ σ «Π = N J a J = θ + OΠ σ «Π = N J 52

53 5. Flexible implementation Example (constrained boundaries) Actual design: Bayes Analysis (j) a j d j 1 (N = 158) (N = 370) (N = 525) β(0.0) β(0.12) Notes on constrained boundaries: * Also addresses changing estimates of information-time. * Also allows increase/decrease the sample size to maintain power. * Implemented in S+ SeqTrial function seqmonitor. 53

54 Summary/Conclusions The positive predictive value of a clinical trial is a meaningful framework for measuring success in development of therapeutic or prevention interventions. PPV can be used to assess the scientific implications of new approaches to statistical practice. Bayesian approaches to trial design and monitoring: * PPV can be low unless operating characteristics are maintained * A more formal approach to Bayesian designs is warranted * Flexible implementation can be achieved with any design 54

55 Summary/Conclusions I am enthusiastic about the application of Bayesian constructions to capture the likelihood that a trial will inform practice. However, I advocate for careful consideration of the scientific implications (PPV) before introducing any new statistical method. 55