Testing Multiple Secondary Endpoints in Confirmatory Comparative Studies -- A Regulatory Perspective

Transcription

1 Testing Multiple Secondary Endpoints in Confirmatory Comparative Studies -- A Regulatory Perspective Lilly Yue, Ph.D. Chief, Cardiovascular and Ophthalmic Devices Branch Division of Biostatistics CDRH/FDA April 29, 2009 FDA/AdvaMed Statistical Conference *No official support or endorsement by the Food and Drug Administration of this presentation is intended or should be inferred.

2 Some Clarifications CODB recommendation, not guidance Recommended for cardiovascular device trials; may not be appropriate for others. Two popular statistical procedures are recommended, consistent with the practice in CDER cardiovascular drug area. But, there are many other statistical methods available, each with advantages and limitations. 2

3 Study Success Decision Both safety and effectiveness are evaluated. Generally, study success decision rule regarding both primary safety and effectiveness endpoints needs to be met for a study success claim. Study success decision rule examples: S1, E1: Require success in both S1 and E1 S1; E1, E2: Require success in S1 and (E1 and E2) S1 and (E1 or E2) 3

4 Study-Wise Significance Level Control Primary Endpoint Family Effectiv. Safety Study Success Decision Rule Study-Wise Overall Sig. Level Allocation AND

5 Endpoint Family-Wise Significance Level Control Primary Effectiveness (or Safety) Endpoint Family Endpoint Success Decision Rule Sig. Level Family-Wise Allocation Overall E1 E E1, E2 E1 and E2 E1 or E2 0.05, 0.05* 0.025, *: 0.64 Study Power 0.8 5

6 Making Statistical significance Claim on Secondary Endpoints? Multiple primary and secondary endpoints are often simultaneously considered in pivotal device trials. After the primary objectives of the trial are met, one may wish to test a large number of prespecified secondary endpoints for any potential claim of statistical significance. However, the multiple assessments lead to multiple opportunities for positive findings purely due to chance (false positive findings), and so need control. 6

7 Chance of False Positive Findings (Overall Type I Error) Suppose K endpoints are independent None of them has true treatment effect Test each of them at level 0.05 Then, the chance of finding at least one sig. endpoint (a false positive finding): K Chance (%) Generally, with 10 endpoints and testing each at 0.05, the chance of having at least one false positive finding is as large as 40%. 7

8 General Multiplicity Control in Significance Level Effective. Safety Endpoint-Wise Overall Study-Wise Overall Primary Endpoint 0.05 AND Family Secondary Endpoint Family IF Primary Objectives are met, Additional 0.05* 0.05* *Conditional on successful primary endpoints 8

9 What if Primary Endpoints didn t Meet the Pre-Specified Study Objective? Secondary endpoints cannot be validly analyzed if the primary endpoint does not demonstrate clear statistical significance Robert O Neill, Controlled Clinical Trials 18: (1997) Secondary endpoints can be validly analyzed, even if the primary endpoint does not provide clear statistical significance C.E. Davis, Controlled Clinical Trials 18: (1997) 9

10 When to Test Secondary Endpoints for Statistical Significance (CODB Recommendation) If and only if a pre-specified study success rule based on primary endpoints is met; And If in protocol, 1). The secondary endpoints were pre-specified 2). Associated hypothesis tests were pre-specified 3). Multiplicity adjustment strategy was pre-specified Otherwise, no statistical significance claim or p-value should be presented in labeling. 10

11 When to Test Secondary Endpoints for Statistical Significance (cont.) Post-hoc selected secondary endpoint analysis is exploratory or hypothesis-generating finding, and is not able to provide valid statistical evidence of safety and effectiveness in the current confirmatory pivotal trial. The P-value cannot be presented for descriptive purpose. 11

12 How to Test Secondary Endpoints Endpoint-specific hypothesis testing is needed. Multiple statistical methods are available, including Bonferroni Test each of k endpoints at the same level 0.05/k e.g. if k =10, test each endpoint at May be too conservative! Two sequential test procedures: Hierarchical closed test procedure Holm s step-down procedure (improvement of Bonferroni) 12

13 Hierarchical Closed Test Procedure Order k (>1) secondary endpoints based on clinical importance: E (1), E (2),, E (k). Test E (1), E (2),, E (k) sequentially at the same level 0.05 as follows. Test E (1) first. If and only if it is significant, test E (2). If and only if E (2) has a significant result, test E (3). The process continues until the first time the test is failed. Then the process stops and the remaining endpoints won t be tested. Claim statistical significance for those endpoints that passed the test, if allowed by clinical judgment. 13

14 Note: The order needs to be pre-specified in protocol. Sponsor could propose the order, but it is subject to FDA clinical review and agreement. A difficult case: sequence breaks with extreme subsequent p-values, e.g., Proposed E (1) = exercise tolerance E (2) = mortality rate But p 1 > 0.05, p 2 =

15 A modification: Suppose that there are two secondary endpoints ordered as E (1) and E (2) Predefine α andα such that α = α + e.g., 1 α 1 = 0.04 and α 2 = Test E (1) first at level (= 0.04) 2 α (a) If the test is significant,, test E (2) at level (= 0.05 not 0.01) (b) if it is not significant, test E (2) at level (= 0.01) This test strategy controls the overall error rate at level (0.05) But, what if there are 10 secondary endpoints? 1 α 2 α α 2 α 15

16 Comparison of the Two Procedures Hierarchical closed test procedure All endpoints are tested at the same level 0.05 if there is a chance to be tested. Easy to handle with a large number of endpoints. An endpoint as well as the following endpoints won t have a chance to be tested if its pervious endpoint fails. The modification All endpoints have a chance to be tested even if the previous endpoint fails. All endpoint are tested at level < 0.05 (could be much smaller than 0.05 later on, making the test very difficult to pass) unless the test of its previous endpoint is significant. The alpha spending has to be pre-specified. May not be easy to handle with a large number of endpoints 16

17 Holm s Step-down Procedure No need to pre-order the secondary endpoints Order observed univariate p values: p p... p k (1) (2) ( ) Corresponding endpoints: E 1, E 2,, E k Sequentially, test E 1, E 2,.., E k at the following level, α / k, α/(k - 1 ),..., α 17

18 Test E 1 first. Holm s Step-down Procedure (cont.) p(1) < 0.05/ k If, claim stat sig. for E 1, and then test E 2 If p(2) < 0.05/( k 1), then claim stat sig. for E 2, and then test E 3 at level 0.05/(k-2). Continue the procedure until the first time the test is failed, and then stop. Claim statistical significance for those endpoints that passed the test. 18

19 Pros and Cons of the Three Procedures Bonferroni No need to order the multiple endpoints beforehand. All endpoints have a chance to be tested. But, they are tested at the same level 0.05/k, e.g., if k =10, -- may be too conservative. Hierarchical Closed Test Procedure All endpoints are tested at the same level 0.05 if there is a chance to be tested. An endpoint as well as the following endpoints won t have a chance to be tested if its previous endpoint fails. The order of tests has to be pre-specified. 19

20 Pros and Cons of the Three Procedures (cont.) Holm s Step-down Procedure No need to order the multiple endpoints beforehand. It is much harder to pass the test, e.g., if k = 10, test E 1 at 0.005; if significant, then test E 2 at 0.006, An endpoint as well as the following endpoints won t have a chance to be tested if its previous endpoint fails. Understanding of relationship between endpoints helps in selecting an efficient test strategy for the multiple endpoints. Clinical inputs are crucial. 20

21 Where the Significance Claim Appears A claim of statistical significance based on primary endpoints goes into the indications for use. A claim of statistical significance based on secondary endpoints appears in the clinical study section in labeling, but not in the indication for use. However, it is more clinical and regulatory issue, so may vary. 21

22 CODB Current Practice -- the presentation of p-values in labeling For new IDE trials in development, or for which the protocol has not been finalized, it is strongly suggested to incorporate the recommendation in protocol. For a study that is already ongoing or completed, p- value (or confidence interval) might be presented with footnote indicating that p-values and/or confidence intervals are unadjusted for multiplicity. 22

23 Concluding Remarks In order to claim statistical significance w.r.t. multiple secondary endpoints after successful primary endpoints, it is suggested to pre-specify the following in protocol Secondary endpoints Hypothesis testing associated with the secondary endpoints (Endpoint-specific hypothesis testing is needed) Multiplicity adjustment strategy Post-hoc selected secondary endpoints or hypothesis testing may not be able to provide valid scientific evidence. 23

24 Thanks! 24