Combining structured data with machine learning to improve clinical text de-identification

Transcription

1 Combining structured data with machine learning to improve clinical text de-identification DT Tran Scott Halgrim David Carrell Group Health Research Institute

2 Clinical text contains Personally identifiable information (PII): information that can identify an individual in context. Some PII are protected by the Health Insurance Portability and Accountability Act (HIPAA) HIPAA-protected PII Patient name Medical record number Age Social security number Dates (including birthday) Address, room Url, IP address Others Not HIPAA-protected PII Provider name, initials Organization name 2

3 Steps to de-identify PII in clinical text Step 1: Find PII text spans Step 2a: Replace PII spans Step 2b: Remove PII spans 3

4 Finding PII text spans Measurement Recall Precision Purpose The number of identifiers (PII) detected divided by the total number of PII in the reference standard The number of correctly predicted identifiers divided by the number of predictions 4

5 Hypothesis 1. We can train a high performing machine learning model to find most PII with reasonable precision 2. We can increase recall of sensitive, HIPAA-protected PII with a data matching algorithm 5

6 Experiment description Chart reviewers annotate PII to create our gold standard Train a machine model on corpus A Apply model to corpus B Compare machine model vs. hybrid on corpus C Develop a secondary algorithm Review documents where recall is not 100% (corpus B ) 6

7 Experiment corpus Corpus Corpus A Description Training data to develop a machine learning model 635 Family Practice, 70 Internal Medicine, 131 Oncology, 70 OBGYN notes Corpus B Baseline test data to get documents for Corpus B 129 Family Practice notes Corpus B Corpus C Documents in corpus B where the machine learning model did not have 100% recall. Used to inform the data matching rules Reserved test data to assess whether the hybrid approach can improve performance on unseen documents in the future 7

8 The machine learning tool used: MITRE Identification Scrubber Toolkit 1 MITRE Identification Scrubber Toolkit 1 (MIST) is an open source machine learning toolkit specifically designed to de-id PII in natural text Scalable and robust User-friendly interface Powerful commands Well documented 1. An earlier version of MIST was the highestperforming automated system in the Informatics for Integrating Biology and the Bedside (i2b2) deidentification Challenge 8

9 Machine model tested on corpus B Precision Recall 9

10 Examples of PII* in corpus B SUBJECTIVE: Abcdef G Hill is a 44 year old male here to follow up on diabetes and pain. Medicine regimen: In am Mr. Hill takes glyburide 10 mg In pm Mr. Hill takes glyburide 10mg pain level between 5-7/10 Lives with parents (Will and Jane Miller) Get a tdap ( tetanus shot) call optometry at yo recent Phd engineering grad from NYU is currently in europe *All PII shown and redacted are fictional (name, age, date, etc.) 10

11 Data matching algorithm For each note and patient id pair in the corpus Read in the output from MIST (text, PII offsets) Get encounter and patient information Attempt to match then edit/add PII in the following order (with higher certainty first): reg. ex pattern for , date, zip, phone, medical record id, social security number reg. ex pattern to rule out blood pressure, decimal values, pulse match on patient data match on encounter data match from a list of hospitals for organization name match from a list countries and states for address For each token match from a list of US census names for patient name If token is a noun or proper noun phrase match from a dictionary of providers by last name 11

12 Data matching algorithm summary Discrete EMR data Regular Expression Lookup list Address Age ~ - Date - Doctor name - - Medical record number - Organization name - Patient name - Phone - SSN - 12

13 Data matching programming tools Python 1 Developed in IronPython 2 (optional) pyodbc to connect to a Clarity SQL database 3 Regular expression 4 Natural Language Tool Kit (NLTK) 5 unittest (32 bit on Window 7) Bird, Steven, Edward Loper and Ewan Klein (2009), Natural Language Processing with Python. O Reilly Media Inc

14 Recall increased when using the hybrid approach in corpus B Recall Machine model Hybrid 14

15 Precision decreased when using the hybrid approach in corpus B Precision Machine model Hybrid 15

16 Net performance gained overall and in HIPAAprotected PII when using the hybrid approach in corpus B Net performance Recall change Precision Change Recall change + Precision change 16

17 Challenges Person names matching Patient names, as defined, include any non provider names Providers are not always Group Health providers Incomplete sentences, lack of grammar and formatting means part of speech tagging is not reliable? Many false positives matches like Will, May, Major, etc. Organization name in the gold standard include non medical facilities Age appears in many form, matching on a number is not good Tried regular expression to rule out age Dates are in unpredictable, non-distinct formats Performance measurements did not give credit to partial span match Occasionally the gold standard gets updated Only reviewed 36 documents What is reasonable precision lost? How to measure it? 17

18 Conclusions With an integrated delivery system, we often have metadata about each chart note De-identified clinical text can still contains important clinical data useful for research if we weight the value of high recall over precision loss differently for each PII type Continue effort to increase PII More patient and encounter identifier data variables Clever surrogates to hide residual PII Hiding in plain sight 1 1. Carrell D, Malin B, Aberdeen J, et al. J Am Med Inform Assoc (2012). doi: /amiajnl