Evaluating the generalizability of an RCT using electronic health records data

Transcription

1 Evaluatng the generalzablty of an RCT usng electronc health records data

2 3 nterestng questons Is our RCT representatve? How can we generalze RCT results? Can we use EHR* data as a control group? *) Electronc Health Records

3

4 Dfference n data collecton The RCT data The EHR data Patents ncluded by crteras Regular vsts Measurements accordng to protocol Extenste data cleanng Usually randomzed Mssng could be non-nformatve e.g. LDL mssng due to lost sample Patents ncluded f they go to the doctor Vsts as needed Measurements as needed Data cleanng? confoundng Mssng s almost never non-nformatve e.g. LDL not needed and thus mssng Placebo effect

5 An llustratve(?) example One large global CV outcome study n prmary preventon wth lberal ncluson/excluson crtera Three CPRD cohorts ndcated for prmary preventon of CV dsease were created from a 6 year study wndow ( ): Cohort 1: patents meetng the tral s ncluson crtera and excludng those wth pror CV hstory or CRP value ndcatng severe bacteral nfecton Cohort 2: patents meetng NICE gudelnes for recommended statn therapy n prmary preventon of CV dsease

6 Sample szes and varables Sampleszes Data set N N complete cases N after mputaton RCT CPRD CPRD Realstc cohort Very broad cohort The analyss was run on two dfferent sets of varables: Framngham varables: AGE, BMI, TC, sex, smokng All* varables: AGE, ANTIHT_USE, ASA_USE, BMI, CRP, DBP, FPG, HDL, LDL, SBP, TC, TG, sex, smokng, weght

7 7 RWE Introducton February 2013

8 8 RWE Introducton February 2013

9 If we have the RCT results, what can we say about the effect n dfferent populaton? effect

10 Is our tral representatve? -compare patent characterstcs One varable at a tme All varables at the same tme Convex hull Cross matchng (Lnear) dscrmnant analyss

11 Descrptve statstcs per varable RCT mean RCT std EHR mean EHR std GENDER_MALE AGE WEIGHT BMI SBP DBP CIGS/DAY SMOKER FPG GLUC LDL HDL TG TC HBA1C CRP ASA USE ANTIHT USE MEDHIST DM FAMHIST CHD

12 Convex hull Idea: Construct the convex hull for the RCT patents and see how many RWE patents fall nto that Defnton: the convex hull for a set S s the smallest convex set that contans s 1 d convex hull: The range 2 d convex hull: K d convex hull: Wrap t n (stff) paper Example: matcht(formula = tral ~ AGE + BMI+ CRP + SBP + DBP + sex + LDL + FPG + CR_CL + TC + TG + HDL, data=anadata, method="nearest", dscard="hull.control") Sample szes: EHR patents All 3933 In the RCT convex hull 254 Dscarded 3679 Almost all of the EHR patents les outsde of the convex hull for the RCT patents. It s enough to be extreme on one varable (or n one drecton)

13 Cross matchng Cross matchng as a test comparng multvarate dstrbutons Rosenbaum (2005) Idea: Merge all the data Create match pars usng the Mahalanobs dstance Count the number of cross matches (A matched to B) The number of cross matches has a know dstrbuton under H 0 A B A B A B A B B A A B A B A A A B B B A A B B Observed number of cross matches: 907 Expected number of cross matches under H 0 : 1183 P-value: 10**(-35) Indcates that the RCT and RWE populatons dffer

14 Lnear dscrmnant analyss Try lnear dscrmnant analyss to fnd a lnear functon that separates RCT and EHR patents Coeffcents of lnear dscrmnants: LD1 AGE_I BMI CRP SBP DBP LDL FPG CR_CL TC TG HDL Percent of Total Comparng dstrbuton of posteror predcton probabltes RCT RWE Mean=0.58 Mean= posteror predcton probabltes

15 Next steps Propensty score (Stuart &Cole 2010, 2011) Cross desgn synthess (Kazar 2009) Herarccal models (Prevost et al 2000)

16 The nce thng about propensty score e X = P T X e X X

17 For us S ndcates membershp n the RCT sample T ndcates treatment assgnment T = {1,0} covarates X potental outcomes: Y (1) would be observed under treatment Y (0) would be observed under control The sample average treatment effect SATE = 1 n s =1 Y 1 Y 0 The populaton average treatment effect N PATE = 1 Y N 1 Y 0 =1

18 Key assumptons All patents n the populaton have some probablty of beng n the tral and no patents are always n the tral 0 < P S = 1 X < 1 Incluson n the tral does not depend on the potental outcomes except though the covarates X S Y 0, Y 1 X Treatment assgnment does not depend on the ncluson nto the tral or the potental outcomes except through X T S, Y 0, Y 1 X

19 Propensty score as a dstance meaure S S p x e n N x e n So, what constues a bg dffernece? Suggeston: bg f p p c ˆ C=0.25 or 0.1 has been suggested

20 Propensty score, all varbles Comparng dstrbuton of propensty scores, PLS-glm Cohort1 RCT Jupter 20 Comparng dstrbuton of propensty scores, PLS-glm Cohort3 Jupter RCT Percent of Total 20 CPRD CPRD 1 30 CPRD 3 0 Percent of Total Propensty score Propensty score Δ=0.165 Δ=0.478

21 Propensty scores based on rsk factors Comparng dstrbuton of propensty scores, PLS-glm Cohort1 FH RCT Jupter 25 Comparng dstrbuton of propensty scores, PLS-glm Cohort3 FH RCT Jupter Percent of Total 25 CPRD 1 5 CPRD 1 CPRD 3 0 Percent of Total 30 CPRD Propensty score Propensty score Δ=0.06 Δ=0.38

22 Predct the results n the EHR populaton (PATE) IPSW : Inverse probablty of selecton weght Weght: w P P S S 1 X In our case the endpont s a tme to event so we ll ft a weghted Cox proportonal hazards model wth the partal lkelhood L n n 1 k 1 expx w expx w k 1 R t k k Y Stuart & Cole 2011

23 Predcted treatment effect n the example Cohort Varables used Hazard Lower Upper p-value rato RCT < Cohort1 All < Cohort1 Framngham < Cohort3 All < Cohort3 Framngham < The predcted treatment effect s slghtly closer to 1 whch ndcates an under representaton of low rsk patents n the Jupter populaton The dfference s smallest for cohort 1 and largest for cohort 3 Rsk factors don t account for all confoundng(?)

24 What s next? pacehr Tralo (n=453) EHR(n=38,000) N=662 At least 2 exacerpaton 1 year pre ndex Age Sprometry data N=367 No COPD & 1 year follow up data Only Gna 4 and 5 placebo (n=151) N=193 ptw

25 EHR data as a control group Weght the EHR data by: P S = 1 X P S = 0 X Weght the HER to estmate the treatment outcome that would have been observed f the HER data had the same dstrbuton of patents characterstcs as the RCT Sort of lke estmatng the ATT n an observatonal study.

26 Dong wthout patent level RCT data Evaluate the generalzablty usng Presslee s method Use weghts from the method of moments (Sgnorovch 2012)

27 Dong wthout tyhe RCT patent data 1 Idea: Use the RCT ncluson/excluson crtera to splt a regstry cohort nto RCT elgble and RCT non elgble patents. No acctual RCT data needed! E Defne the sub populaton average treatment effect n each sub populaton PATE SPATE(I) SPATE(E) 1 NI 1 N I N 1 0 N 1 N I 1 Y 1 Y 0 1 N I E 1 Y 1 Y 0 E 1 E I Y 1 Y Measure the generalzaton error by comparng outcomes for I and the whole populaton ˆ ˆ I I E E E ˆ I I ˆ E E

28 Dong wthout patent level data 2 R Regstry Actve Placebo Patent level data Aggregate data Weght the regstry on Gender Age BMI SBI LDL TG Smokng Outcome Y x, y x, y C C Idea: Use ˆ y w w y C where w P P S S 1 x 0 x T Estmate the weghts usng logstc regresson: w exp x T ˆ x exp x Method of moments, solve xc 0 T expx ˆ

29 References Ima K, Kng G, Stuart EA. Msunderstandngs between expermentalsts and observatonalsts about causal nference. Journal of the Royal Statstcal Socety A. 2008; 171: Stuart EA, Cole SR, Bradshaw CP, Leaf PJ. The use of propensty scores to assess the generalzablty of results from randomzed trals. Journal of the Royal Statstcal Socety A. 2011; 174: Cole SR, Stuart EA. Generalzng evdence from randomzed clncal trals to target populatons. Amercan Journal of Epdemology. 2010; 172: Rosenbaum PR. An exact dstrbuton-free test comparng two multvarate dstrbutons based on adjacency. Journal of the Royal Statstcal Socety B. 2005; 67: Prevost TC, Abrams KR, Jones DR. Herarccal models n generalzed synthess of evdence: an example based on studes of breast cancer screenng. Statstcs n Medcne. 2000; 19: Sgnorovch et al Matchng-Adjusted Indrect Comparsons: A New Tool for Tmely Comparatve Effectveness Research. Value n Health. 2012; 12: Pressler T. R. Kazar E. E. The use of propensty scores and observatonal data to estmate randomzed controlled tral generalzablty bas. Statstcs n Medcne. 2013; 32: