Developing a Valid Measure of Opioid Overdose and Abuse/Addiction with Coded Medical Terminologies: Supervised Training Using Experts

Transcription

1 Developing a Valid Measure of Opioid Overdose and Abuse/Addiction with Coded Medical Terminologies: Supervised Training Using Experts Alexander M. Walker, MD, DrPH See Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet 2001; 358: for an introduction to supervised training. 1

2 Moving between clinical definitions and the data available in data bases is NOT an exercise in mapping codes Defining elements include considerations of Specificity of diagnostic terms Overlap of diagnostic terms Timing Clinical plausibility Differential diagnosis Iterative algorithm development Supervised learning Clinical expertise Database expertise 2

3 What will follow in this talk 1. Supervised training using experts with no charts There is rich information in longitudinal data, far beyond the sensitivity/specificity of single elements, readily apparent to a subject-matter expert 1. Expanding the technique with charts Expert s assessment of the medical record replaces the expert s assessment of the longitudinal data elements as a gold standard (Note that we are moving rapidly toward fully retrievable medical records. The techniques described here still apply.) 3

4 Goal Machine algorithms that perform like the expert when presented with the same data. 4

5 Creating an algorithm that mimics expert assessment in a longitudinal electronic record 1. Start with a sensitive but non-specific rule 2. Draw sample histories from the data base 3. Expert classifies sample histories according to conformity with clinical expectations 4. Expert records the basis for classification 5. Turn the expert-derived observations into classification rules 6. Compare the classification of the newly sampled cases to clinical expectation 7. Resample and classify by algorithm and expert 8. If expert and rule are concordant, STOP Otherwise revise the rule and go to 2 5

6 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Source population A priori screen Candidate cases 6

7 Creating an algorithm that mimics expert assessment in a longitudinal electronic record. Example: Opioids Source population A priori screen Candidate cases All users of ER/LA opioids in research partner s data 7

8 Creating an algorithm that mimics expert assessment in a longitudinal electronic record. Example: Opioids Source population A priori screen Candidate cases Structured codes e.g. 304.x Drug dependence 305.x Nondependent abuse Poisoning by opiates and related narcotics 8

9 Creating an algorithm that mimics expert assessment in a longitudinal electronic record. Example: Opioids Source population A priori screen Candidate cases N = e.g. 1% 9

10 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 10

11 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 11

12 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Source population A priori screen Candidate cases Sample First cycle only Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 12

13 Creating an algorithm that mimics expert assessment in a longitudinal electronic record. Example: Opioids Source population A priori screen Candidate cases Sample Classify by machine rule Structured codes e.g. 304.xx Drug dependence 305.xx Nondependent abuse 965.0x Poisoning by opiates and related narcotics Compare Classify by expert Similar Dissimilar Update machine rule Done 13

14 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 14

15 A profile is a chronologic sequence of all structured elements available in the data, formatted so as to promote easy human reading. Spacing, indents, bold, colors distinguish data types, normal and abnormal values, codes and terms of special interest. 15

16 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample 50 Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 16

17 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample 50 Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 17

18 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 18

19 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 19

20 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 20

21 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 21

22 Creating an algorithm that mimics expert assessment in a longitudinal electronic record Candidate cases Sample Profiles Classify by machine rule Classify by expert Compare Done Similar Dissimilar Update machine rule 22

23 The expert-generated updated rules contain elements derived from the profiles, with addition of timing and Boolean logic Diagnostic codes Drug codes NLP (Natural Language Processing) terms derived from EMR Sequences of codes (repeated visits, escalating doses for drugs) Services (e.g. repeated ED utilization, multiple doctors, overlapping prescriptions) Supplemental codes that would be nonspecific by themselves, but increase specificity in context Disqualifying situations 23

24 Example: An algorithm to classify epilepsy from insurance claims A diagnostic code from insurance claims is accepted (excerpts) if there is multiplicity or if timing of testing and diagnosis follows in a characteristic sequence or if the code comes from a standardized source or if there is support pharmacy data The code appears on two different days The code appeared once, preceded by an epilepsyspecific diagnostic test electroencephalography brain imaging long-term seizure monitoring test to identify brain regions The code appeared as a primary hospital discharge code The code appeared with a minimum of three dispensings of epilepsy-specific medications Kurth T, Lewis BE, Walker AM. Health care resource utilization in patients with active epilepsy. Epilepsia May;51(5):

25 Incorporating chart validation or interview Process 1. Machine algorithm without chart, previously described 2. Expert evaluates a sample of charts or interview results 3. Compare algorithm to results for one patient 4. Update algorithm 5. Continue steps 3-4 through the entire sample 6. Return to first patient 7. Continue steps Stop when the algorithm stabilizes 25

26 Example chart review-based algorithm for colonic ischemia Start with all instances of ICD9-CM code 557, associated with inpatient or outpatient physician service (that a dx motivating a diagnostic test is not sufficient) Exclude Individuals with codes for hemolytic uremic syndrome, idiopathic thrombocytopenia purpura, and aortic aneurysm Abstract charts 57 obtained Sands BE, Duh MS, Cali C, Ajene A, Bohn RL, Miller D, Cole JA, Cook SF, Walker AM. Algorithms to identify colonic ischemia, complications of constipation and irritable bowel syndrome in medical claims data: development and validation. Pharmacoepidemiol Drug Saf. 2006;15(1):

27 Create chart abstraction elements for case definition 1. Symptoms e.g. abdominal pain, altered bowel function, nausea, vomiting, anorexia 2. Physical findings e.g. abdominal distension, tenderness 3. Laboratory leukocytosis, but no evidence of infection 4. Diagnostic studies endoscopic, radiographic, surgical, and/or histopathologic Sands BE, Duh MS, Cali C, Ajene A, Bohn RL, Miller D, Cole JA, Cook SF, Walker AM. Algorithms to identify colonic ischemia, complications of constipation and irritable bowel syndrome in medical claims data: development and validation. Pharmacoepidemiol Drug Saf. 2006;15(1):

28 Cycle through the steps 1. Machine algorithm without chart, previously described 2. Expert evaluates a sample of charts or interview results 3. Compare algorithm to results for one patient 4. Update algorithm 5. Continue steps 3-4 through the entire sample 6. Return to first patient 7. Continue steps Stop when the algorithm stabilizes 28

29 Performance for the sampled cases Sands BE, Duh MS, Cali C, Ajene A, Bohn RL, Miller D, Cole JA, Cook SF, Walker AM. Algorithms to identify colonic ischemia, complications of constipation and irritable bowel syndrome in medical claims data: development and validation. Pharmacoepidemiol Drug Saf. 2006;15(1):

30 Summary Moving between clinical definitions and the data available in data bases is NOT an exercise in mapping codes Defining elements include considerations of Specificity of diagnostic terms Overlap of diagnostic terms Timing Clinical plausibility Differential diagnosis Iterative algorithm development Supervised learning Clinical expertise Database expertise 30

31 Summary Moving between clinical definitions and the data available in data bases is NOT an exercise in mapping codes Defining elements include considerations of Specificity of diagnostic terms Overlap of diagnostic terms Timing Clinical plausibility Differential diagnosis Iterative algorithm development Supervised learning Clinical expertise Database expertise 31

32 Summary Moving between clinical definitions and the data available in data bases is NOT an exercise in mapping codes Defining elements include considerations of Specificity of diagnostic terms Overlap of diagnostic terms Timing Clinical plausibility Differential diagnosis Iterative algorithm development Supervised learning Clinical expertise Database expertise 32

33 Readings and Sources Supervised training Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet 2001; 358: for an introduction to supervised training. An early example of rules involving relative timing Lanza LL, Dreyer NA, Schultz NJ, Walker AM. Use of insurance claims in epidemiologic research: Identification of peptic ulcers, GI bleeding, pancreatitis, hepatitis and renal disease. Pharmacoepidemiol Drug Safety. 1995;4: Creating an algorithm to classify epilepsy from insurance claims Kurth T, Lewis BE, Walker AM. Health care resource utilization in patients with active epilepsy. Epilepsia May;51(5): Incorporating chart abstraction into case definition Sands BE, Duh MS, Cali C, Ajene A, Bohn RL, Miller D, Cole JA, Cook SF, Walker AM. Algorithms to identify colonic ischemia, complications of constipation and irritable bowel syndrome in medical claims data: development and validation. Pharmacoepidemiol Drug Saf. 2006;15(1):

34