Other Analytical Techniques. Nick Salkowski SRTR February 13, 2012

Transcription

1 Other Analytical Techniques Nick Salkowski SRTR February 13, 2012

2 Control Charts and Control Limits 1 Control Charts: Routinely monitor quality Distinguish between in-control and out-of-control processes Distinguish between normal variation and assignable cause variation Run until there is an out-of-control signal Exceeding Control Limits or thresholds trigger a response 1 NIST/SEMATECH e-handbook of Statistical Methods, Section 6.3 September 13,

3 Thresholds and Responses Control thresholds and response plans need to be developed together Lower thresholds will produce more false-positive signals, and are appropriate if the response is minor Higher thresholds will produce fewer false-positive signals, and are appropriate if the response is intensive Of course, higher thresholds make it more difficult to signal when the process is out-of-control, too! 3

4 Statistical Hypothesis Tests Theoretically distinct from Control Charts Test a specific null hypothesis against an alternative Type I errors Rejecting a true null hypothesis Type II errors Failing to reject a false null hypothesis Adjustments are needed for multiple testing Statistical Hypothesis Tests produce a decision 4

5 CUSUM Strengths Tracks process continuously using current data Produces a signal after a center has a sufficiently bad run of outcomes Chart provides a visual summary of center performance over time "When retrospectively compared to currently available data reporting, the CUSUM method was found to detect clinically significant changes in center performance more rapidly, which has the potential to inform center leadership and enhance quality improvement efforts." Axelrod, et al American Journal of Transplantation 9(part 2):

6 CUSUM Limitations Data doesn't always appear instantly It can take months for a death to appear in the data set! CUSUM charts are intended to run until there is a signal In-control processes will all signal eventually Calculating the CUSUM can be computationally challenging When the in-control and out-of-control rates are based on survival models, the daily hazard for every person at risk must be calculated every day CUSUM is a tool for constant quality monitoring: it is best if it is calculated whenever there is new data Daily computation is probably sufficient Much less useful if the CUSUM is calculated every 6 months CUSUM doesn't provide a statistic to compare programs 6

7 Threshold Difficulties Thresholds need to be uniquely determined for each program Simulations are needed Predictions about future rates are needed Thresholds will only perform well under a steady state If a program changes over time, the thresholds need to change too! If the number of transplants performed increase, the expected graft failure rate per day probably increases If a program performs more transplants with high expected graft failure rates, the expected graft failure rate per day increases What does an "out-of-control" program look like? Double the risk of an "in-control" program? 50% more risk than an "in-control" program? 7

8 Funnel Plots Scatterplot of an estimate against a measure of the estimate's precision Tend to form a funnel shape since low-precision estimates tend to spread out more than high-precision estimates Good for comparing different centers Good for identifying programs with unusually good or bad outcomes 8

9 Funnel Plot Examples: O/E 9

10 Funnel Plot Examples: (O+1)/(E+1) 10

11 Period Analysis Cohorts Use different cohorts to estimate different segments of the survival curve, so that the most recent outcomes are used For example: Use 2011 transplants to estimate survival during year 1 Use 2010 transplants to estimate survival during year 2 Use 2009 transplants to estimate survival during year 3 Use (2012-Y) transplants to estimate survival during year Y Long-term survival can be estimated without using old data to estimate initial survival Odd behavior at boundaries: 12/31/2010 transplant is used only for 2 nd year survival, but 1/1/2011 transplant is used only for 1 st year survival 11

12 Period Analysis Cohort Example 12

13 Alternative Period Analysis Cohorts Possible to define cohorts as all persons at-risk for a particular event during a specific period of time For example, all persons at-risk for graft failure during the first 3 years post-transplant between January 1, 2011 and December 31, 2011 Includes all transplants during 2011 Includes all transplants during without a graft failure before 1/1/2011 Only failures during 2011 count! Left-truncated / Right-censored analysis Compatible with longer follow-up outcomes (e.g., 5-year, 10-year) Compatible with O/E Hypothesis Test methods 13

14 Alternative Period Analysis Cohort 14

15 Alternative Period Analysis Cohort Limitations Tradeoff between data timeliness and quantity Shorter time intervals mean more recent data and less overlap between PSR cohorts, but smaller sample sizes: fewer events and persons at-risk Changes in power could require changes to flagging criteria or produce different flagging probabilities Some failures or deaths could be "lost" during a transition Occurred too long ago to be included in new cohort Too recent to be included in the old 3-year cohort 15

16 Questions?