Are patient-reported outcome measures useful tools to measure and improve the quality of healthcare? Professor John Browne University College Cork

Transcription

1 Are patient-reported outcome measures useful tools to measure and improve the quality of healthcare? Professor John Browne University College Cork

2 What are Patient-Reported Outcome Measures (PROMs)? Instruments (usually questionnaires) measuring health status and related concepts. Completed by patients either at home or in a care setting, usually with pen/paper. Generic (e.g. EQ-5D) or disease-specific (e.g. Oxford Hip Score). Pre-treatment score subtracted from post-treatment score to derive measure of treatment benefit.

3 PROMs at the individual patient level Patient-level feedback of PROMs to clinicians has been hypothesised to improve care by: Highlighting previously unrecognised health problems. Measuring the effectiveness of different treatment plans. Monitoring disease progression. Stimulating better communication Promoting shared decision making. A long history of use in certain fields (e.g. mental health) with mixed results.

4 There has been a surge of interest in the use of PROMs to measure and improve the quality of healthcare We will make payments to hospitals conditional on the quality of care given to patients a range of quality measures [including] PROMs will be used NHS Next Stage Review: High Quality Care for All, pp 41-42

5 Professor Nick Black Patient reported outcome measures could help transform healthcare BMJ 2013

6 Conditions where PROMs are being used/considered in the NHS Elective surgery LTCs Cancer Hip Knee Hernia Veins Breast Upper GI Diabetes COPD Asthma Epilepsy CHF Stroke

7 NHS use of PROMs: organisational level Clinical governance Pay for performance PROMs Patient choice Purchasing decisions

8 Are PROMs useful quality improvement tools?

9 Does providing feedback on patientreported outcomes to healthcare professionals result in better outcomes for patients? A systematic review. Boyce and Browne, Quality of Life Research (2014) 16/17 studies focused on individual level feedback. Only one study found a significant effect in favour of PROMs feedback: rest found no effect. Studies are methodologically weak. Qualitative research is required to provide a deeper understanding of the practical issues surrounding the implementation of PROMs and the methodological issues associated with the effective use of the information.

10 Measure Will the English PROMs programme improve quality of care? mean treatment benefit mean treatment benefit Oxford Hip Score 19.7 (n = 27,782) 19.7 (n = 36,843) Oxford Knee Score 14.6 (n = 30,902) 14.9 (n = 37,260) Aberdeen Varicose Veins 7.9 (n = 18,874) 7.5 (n = 18,530) EQ-5D hip replacement 0.41 (n = 25,213) 0.41 (n = 33,208) EQ-5D knee replacement 0.30 (n = 28,370) 0.30 (n = 34,154) EQ-5D groin hernia 0.08 (n = 18,280) 0.09 (n = 19,957) EQ-5D varicose veins 0.09 (n = 7,320) 0.09 (n = 7,516)

11 Impact on hospital performance of introducing routine patient reported outcome measures in surgery. Varagunam, Hutchings, Neuburger, Black. JHSRP (2013). There was little apparent impact. There was little interprovider variation and it did not change significantly over time. There were slight improvements in outcomes for hip and knee replacement (SES for disease-specific and generic PROMs 0.03) though not for hernia repair and a slight worsening for varicose vein surgery. The extent of variation in performance between providers was unchanged. The proportion of providers deemed to be outliers did not change over time.

12 How is outcome benchmarking hypothesised to improve quality? 1. Hawthorne effect? may focus clinical minds on optimising treatment for a particular patient (Bridgewater 2007) [Hawthorne effect] data patterns prove to be entirely fictional. NBER Working Paper No

13 2. Natural selection? [outcome data] allows patients, referring doctors and purchasers preferentially to select units or surgeons with good results, and... motivate hospitals and surgeons to compete on quality and thereby improve overall outcomes Bridgewater (2007)

14 30-day mortality (%) Some evidence of effectiveness 4 p <0.001: expected mortality rises 3 2 p =0.01: observed mortality falls 1 0 Introduction of public outcome reporting For NHS cardiac surgery Bridgewater 2007

15 But evidence is generally weak Evidence is scant... Rigorous evaluation is lacking. Publicly releasing performance data stimulates quality improvement activity at hospital level. Effect on effectiveness safety, patient-centredness remains uncertain Fung et al, Arch Intern Med (2008).

16 3. Change the causal focus of quality improvement activity? Outcome benchmarking drives up quality throughout the NHS by encouraging a change in culture and behaviour focused on health outcomes not process NHS Outcomes Framework (2014/15)

17 But outcome data is not actionable : only the causes of outcomes can be dealt with. Care structures e.g. tertiary status Societal determinants Patient risk factors e.g. comorbidities Process of care e.g. type of operation Patient experience Treatment morbidity (e.g. complications HRQOL

18 How to move forward? 1. Build a theoretically coherent intervention from the bottom up. 2. Focus on where improvement is possible. 3. Use the right measurement tools.

19 The use of patient reported outcome measures in routine clinical practice: lack of impact or lack of theory? Greenhalgh J et al Soc Sci Med (2005) A theory-driven approach involves combining knowledge of whether and how an intervention works. It is argued that such an approach is currently lacking within the literature evaluating the effectiveness of feeding back HRQoL information to clinicians.

20 The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research. Boyce, Browne, Greenhalgh. BMJ Quality and Safety (2014) Practical problems (technical support, workload) Negative attitudes (suspicion of managerial objectives, concerns about impact on patient-clinician relationship) Methodological concerns (validity, interpretability) Doubts about impact (cannot turn the data into practical solutions).

21 2. Focus on the areas where improvement is possible At least > 10% variation explained at the provider level is required to justify provider comparisons. PROMs and many other outcome measures fail this test. Signal of important between-provider variation is hidden in the noise of patient-level variation. Makes it hard to shift distribution if all the focus is on between-provider comparisons: large, expensive, untimely research needed to detect outliers because of low reliability.

22 Between-provider variation in aesthetic outcomes for women undergoing reconstruction after mastectomy (IQR = 3 for aesthetic scale of Breast-Q) Mean Score Provider volume Overall average Provider rate 95% limits 99.8% limits

23 Provider level variance is low Satisfaction with breast area Emotional well-being Physical well-being Sexual well-being Partition of variance (unadjusted) Trust-level Patient-level ICC 2.9% 2.1% 0.7% 3.3%

24 So what? For the patient, purchaser, or regulator making choices about care, provider-level comparisons are relatively unimportant From the population perspective it is hard to shift the distribution if all the focus is on between-provider comparisons: needle in a haystack.

25 Caveat When total variability is great, even small ICCs can be worthy of intervention (Fung et al 2010, Medical Care) Small ICCs do not necessarily mean there are not clinically important differences (in absolute terms) between providers.

26 Type of operation does matter Breast area appearance* Emotional well being Physical well being Sexual well being Unadj Score Adj. score Unadj Score Adj. score Unadj Score Adj. score Unadj score Adj. score Immediate Implant-only Pedicle with implant Autologous pedicle flap Free flap Delayed Implant-only Pedicle flap with implant Autologous pedicle flap Free flap

27 Summary of results Pedicle and free flap reconstruction surgery has better outcome on the aesthetic, emotional well-being and sexual well-being scales than implant-only reconstruction. Consistent for immediate and delayed reconstructions. Provider-level comparisons are relatively uninformative.

28 Better outcomes Gap to average MID 3. Use the right measurement tools Number of operations

29 What is wrong with Classical Test Theory measures? Measures like the Oxford Hip Score and EQ-5D cannot be used to pinpoint individual patients who have had good or bad outcomes data is only usable at group level. Data is very difficult to interpret as there is no link between patient locations and item locations.

30 Numbers without meaning are dangerous! Clinicians Purchasers Researchers Patients

31 Advantages of Rasch Measures 1. Person abilities and the item difficulties are additive (linear) measures on the same latent variable: the spacing between the measures is meaningful. 2. The hierarchy of person and item abilities, and the use of an explicit mathematical model of item function supports strong validation. 3. We can locate persons and items on the same scale and test how well our measure matches our sample. 4. The expected performance of any person on any item can be inferred: what is the next treatment milestone for this patient?

32 Interpretability of the Breast-Q DR implant only (55) IR implant only (59) DR pedicle/implant (63) DR free flap (64) DR autologous pedicle (65) IR pedicle (both forms 69) IR free flap (74) Equal, feel natural: somewhat satisfied (52) Closely matched: somewhat satisfied (54) Feel to touch: somewhat satisfied (58) Size: very satisfied (59) Feel normal in clothes: very satisfied (60) Shape in a bra: very satisfied (65) Appearance clothed: very satisfied (69) Can wear fitted clothes: very satisfied (70) Breast lined up: very satisfied (73) Comfortable with bra fit: very satisfied (75) *Adjustment for age, deprivation, comorbidity, radiotherapy.

33 Overall message Great news that data collection infrastructure is in place and PROMs are now accepted in principle as having a role to play in investigating variation in outcome. Large routine datasets allows us to perform real-world comparative effectiveness research. BUT PROMs do not seem to meet the normal criteria for quality improvement tools. 1. Provider comparisons have very little value other than outlier detection and this provides expensive and untimely data. 2. Even for the limited role of outlier detection current PROMs are of little value for promoting change.

34 Solutions 1. Theoretical focus of PROMs initiatives in a quality improvement context must be developed many of the current claims made about potential uses are unlikely to be correct. Great care is needed around facilitating patient and purchaser choices based on PROMs data 2. Treat quality of care studies primarily as cohort studies where provider identities are only one of many possible sources of variation to be explored. 3. Use outcome measures that are meaningful to clinicians and patients and can be used at the individual level if you want to investigate quality of care at the individual level.

35 One page summary