THE ATLANTIC STUDY. Novel applications of modern measurement theory, now and in the future

Transcription

1 THE ATLANTIC STUDY Novel applications of modern measurement theory, now and in the future Chris Gibbons Cambridge Centre for Health Services Research & Psychometrics Centre University of Cambridge

2 INTRODUCTION NIHR Research Fellow at University of Cambridge - Centre of Health Services Research - Psychometrics Centre Developed and validated PROMS for mental health, neurology, quality of life and primary care and some computer adaptive tests (CATS) Clinical impact of PROMS use Technological development and application of modern testing theories

3 THIS PRESENTATION Summarise evidence for the clinical use of PROMS Introduce a computer adaptive test (CATs) using items from the WHOQOL-100 quality of life measure Describe the simultaneous calibration of multiple item banks, to account for differential item functioning by country Discuss a methodology for assessing the validity of CATs using userrated predictions Speculate on the future of epsychometrics, including integration with big data, machine learning and inferential psychometrics

4 PROMS EVIDENCE 28 Studies reviewed 50% of studies in mental health Compared PROM use with feedback to no feedback 67% of studies showed impact on processes 47% of studies showed impact on outcomes

5 UPDATED REVIEW PLANNED 2016

6 WHAT SEEMS TO WORK? Specific PROM training Frequent and timely feedback Feedback processes well aligned with practice Information integrated into available systems Information that is linked to specific clinical action. Interventions with a formal, structured, feedback process perform better (Krageloh, 2014).

7 WHAT DO PATIENTS THINK? PROMS for Depression in primary care in the UK Patients favoured the PROMS They saw them as an efficient and structured supplement to medical judgement They saw them as evidence that clinicians were taking their problems seriously through full assessment.

8 IMPROVING MEASUREMENT IN CLINICAL PRACTICE Make measurement more accurate Make measurement more efficient Engage patients and clinicians with instant feedback Align feedback more closely with care

9 COMPUTER ADAPTIVE TESTING Machine learning algorithm which iteratively learns about the test taker and selects the best item from a large bank of items Can make PROMs more efficient, more reliable and better targeted. Mathematically simple, but constrained until recently by computational power, IRT knowledge, software and lack of a suitable platform to bring together necessary elements of a CAT test. Widely used in educational tests (USA, UK and Australia) but less in health.

10 Item anchoring Overview Psychometric calibration of the WHOQOL-100 to create short item banks for four domains of QoL (ATLanTiC study) Differential item functioning Linked item banks for countries (proof of concept) Simulated computer adaptive tests

11 The WHOQOL Multi-dimensional measure of global quality of life Physical, psychological, social and environmental domains Develop using spoke-wheel methodology in fifteen field centers in native languages Cutting edge developmental methods to ensure global relevance, importance and equivalence.

12 Item bank development Rasch analysis of WHOQOL-100 on UK sample (n = 320) Combination of non-parametric and 1-parameter item response theory analyses Four domains fit the Partial Credit Model (p>0.5) 52 items removed Mean 11 items per bank CAT simulation demonstrated good potential

13 Computer adaptive test UK item banks Developed using MySQL, HTML and R (Concerto) Options to chose - IRT Model (GRM/PCM/GPCM/MGRM) - Scoring (ML, BM, EAP, WL) - Stopping rule (reliability, number of items) Gives tailored feedback of scores (graphical/text) Normative scores given on UK sample Feedback on psychometric performance given Live testing confirmed simulations were accurate (n=1020) Took about 120 seconds (30 seconds per domain)

14 Computer adaptive test Adaptiveqol.com

15 ADAPTIVE FEEDBACK Adaptiveqol.com

16 Hypotheses But.. Solution did not work across entire dataset (15 countries) Hypothesis 1 (H1) : Failure to work across cultures caused by expected differences in cultural standards, values and expectations and not semantic inequivalence. If H1 is supported then item anchoring could be used to calibrate linked item banks for each country. (e.g., Tennant, 2004) Hypothesis 2 (H2) : Linked item banks will fit the Rasch model and perform well* under simulated computer adaptive testing. * Defined by greater precision or efficiency than two paper versions.

17 Testing H1 Added three countries (selected randomly using RNG) England (n=320), Zimbabwe (n= 300), India (n=412) and Russia (n=300) Fit the data to the Partial Credit Model Differential item functioning (DIF) analysis (ANOVA and post-hoc Tukey Tests) Items with DIF split by country Country-specific item banks linked by common thresholds (i. e., shared parameters for items and categories without DIF) Analyses conducted in RUMM2030 and R

18 Differential item functioning People from different groups with the same level of the latent trait (quality of life) have a different probability of giving a certain response to a questionnaire item Male Female Male Female

19 How much do you value yourself?

20 How much do you value yourself? An extreme amount Very much A moderate amount A little Not at all LESS Psychological Quality of life MORE

21 How much do you value yourself? An extreme amount Very much A moderate amount A little Not at all LESS Psychological Quality of life MORE

22 How much do you value yourself? An extreme amount Very much A moderate amount A little Not at all LESS Psychological Quality of life MORE

23 How satisfied are you with your financial situation?

24 How satisfied are you with your financial situation? $614 GNI

25 How satisfied are you with your financial situation? $416 GNI $614 GNI GNI Data accessed from on 14/07/2015

26 Item anchoring What do we do when items mean different things across cultures or demographic groups How can we account for this?

27 Item anchoring Example Physical QoL domain Each item individually assessed for DIF using ANOVA of group differences in response to each item = Untested item that fit the Rasch model in the UK sample

28 Item anchoring = DIF Present for item = DIF Absent for item

29 Item anchoring = DIF Present for item = DIF Absent for item

30 Tukey post-hoc tests

31 Item anchoring

32 Item anchoring Repeat this process for all items with DIF by country

33 Item Bank no DIF = DIF Absent for item

34 Country item banks = Item with common metric for all countries = Item with metrics for 2+ countries = Item with single-country metrics

35 Item bank comparison

36 ITEM ANCHORING Disease specific and generic measurement Cross-cultural research Core item sets in clinical trials But.. Not always clear how much of a problem DIF is (is this wasted effort?)

37 PREDICATE STUDY PREDictions In a Computer Adaptive Testing Envrionment Validity of CATs is somewhat unclear Can we use item response theory to predict Quality of Life information (responses to questionnaires)? Based on the ATLanTiC Quality of Life platform. CAT is completed until reliability is.90 Predictions are made based on an IRT model and presented to test-takers Test takers rate the predictions PREDICTMYQOL.com (try it, please )

38 PREDICATE Study Questionnaire Score estimation Scored by a CAT algorithm

39 Score estimation Score estimation

40 PREDICATE Study

41 PREDICATE Study

42 PREDICATE Study

43 PREDICATE Study (For example) We think that you would have said Strongly Disagree To the question: I am satisfied with my access to information User rates this prediction as: Correct Close, but not quite right Wrong

44 PREDICATE Study Study still in progress Interim analysis suggest predictions are good (up to 98% are rated as close, 74% as exactly right ) Could be useful for backfilling questionnaire items when CAT has been employed, to maximise clinical information with minimal input from the patient Validate the model, are predictions as good as we thought? Enjoyable process? Would love to hear what you think!

45 ATLanTiC future work Increased stakeholder involvement in feedback development and qualitative research Addition of patient experience/emotional modules Integration into electronic health records

46 ATLanTiC conclusions The item bank is brief and can be administered in 120 seconds As much as a 78% increase in efficiency compared to paper-based versions Simulations translated well to real world CAT Prediction study supports the validity of the measurement model and backfilling unanswered item bank items Feedback well received and instantaneous Well aligned with clinical practice and useful for research. But, how can we do better? Do we need to ask people questions at all?

47 The PROMS future Measure twice cut once Electronic Health Record integration for comparative research e.g., Health North in the UK What might the integration of Big Data and psychometric look like? Can we do more than simply put questionnaires into the EHR? How may we leverage the opportunities brought forth by the Big Data revolution to improve patients care, outcomes and experience

48 MYPERSONALITY Psychometric tests (BIG-5) hosted on Facebook Developed and managed by psychometrics centre members More than 5,000,000 people have completed the Big-5 questionnaire Large proportion of users shared their Facebook Like information

49 MYPERSONALITY Question: Can you build a predictive model of a psychometric test (BIG-5) using Facebook likes

50 LIKES AND ITEMS Do you like AUT? Yes No

51 MY PERSONALITY

52 MY PERSONALITY

53 DOES IT WORK?

54 APPLYMAGICSAUCE.COM

55 WHY MIGHT THIS APPROACH BE USEFUL FOR HEALTH SCIENCES Data are available on a massive scale (and increasingly so) Learning data cities are being planned May be less biased than questionnaire items (less obvious) Real-time monitoring/assessment/feedback No measurement latency (can be done anywhere)

56 WHAT MIGHT WE USE FOR INFERENTIAL PSYCHOMETRICS? Facebook likes Twitter follows Twitter updates Images (image recognition e.g., Facebook profile photos) Wearables Geo-location data Movement data Phone activity Interactions with friends (proximity of two devices)

57 INTERNET OF THINGS

58 SF-36 & GOOGLE DATA?

59 DEPRESSION SCALE AND GOOGLE DATA? Do you have thoughts of suicide? Have you been experiencing delusions?

60 GOOGLE AND THOMAS INSEL Former head of NIMH in the USA Moved to Google in late Psychometricians very interested in what might happen! 'Technology can have greater impact on mental healthcare than on the care for heart disease, diabetes, cancer or other diseases It could transform this area in the next five years. (Insel, 2015)

61 IS USING DATA LIKE THIS A BIT CREEPY?

62 CLINTOUCH A mobile support technology for people with psychosis Real world screening, medication management and service-level interventions Monitor patients in the community, respond to crises and avoid unnecessary appointments. Patients like it small qualitative sample..

63 PredictiveDataProject.com Assessment 27 Yes/No Questions on aspects of Big Data Measuring attitudes towards Personalisation, Ethics, Wearable Tech, Internet of Things, Finance, Policy-Making, Cloud Storage, etc. 20-item BIG5 Personality test at the end Feedback Participants given real-time feedback on the similarity of their answers to all previous participants

64 Sample (by Oct 15) No. of test sessions in total (Sept) Average age 30 43% Europe 9% Asia 27% North America 15% South America 33,937 19,126 10,411 8,871 Answered At Least One Question All Big Data Questions All Personality Questions

65 Results: Audit of Global Public Opinion Are those with access to your personal data able to accurately predict your future behaviour? (n=19,100) 62% NO 38% YES Should predictive technologies be used to improve the quality of healthcare, for example by helping doctors to recommend personalised nutrition and exercise plans? (n=14,187) 16% NO 84% YES Should predictive technologies be used assess your eligibility for a mortgage? (n=13,725) 62% NO 38% YES Do you think your organisation ought to invest in predictive technologies? (n=2,830 CMOs) 62% NO 73% YES Is it important for you to understand the psychological attributes of your customers? (n=2,489 CMOs) 8% NO 92% YES

66 PATIENTS AND SOCIAL MEDIA 71% of patients were OK with sharing their social media information with their doctor

67 CONCLUSION Modern psychometrics offers many opportunities for improved research and clinical practice Computer adaptive tests offer, prima facie, solutions to better PROM interventions mypersonality showed what will/could be possible for inferential psychometrics The future is exceptionally exciting in this field!