DeMISTifying Deidentification of PHI in Free-formatted Text

Transcription

1 DeMISTifying Deidentification of PHI in Free-formatted Text Cathy Petrozzino March 2016 Approved for Public Release; Distribution Unlimited. Case Number The MITRE Corporation. All rights reserved. mitre

2 2 Overview Introduction Tool Rationale MITRE Identification Scrubber Toolkit (MIST) Use Case 1 Deidentification Hiding in Plain Sight Use Case 2 Identification of PHI in Privacy Risk Identification and Management Engine (PRIME)

3 3 MITRE We are a (not-for-profit) public interest company, working with industry and academia to advance and apply science, technology, systems engineering, and strategy, enabling government and the private sector to make better decisions and implement (publicly available) solutions to complex challenges of national and global significance. including in the areas of natural language processing, privacy, and cybersecurity.

4 4 Open Source MITRE Identification Scrubber Toolkit (MIST)

5 5 Use Case #1: Research Doctor s Notes: research involving free-formatted text Treasure trove of information How to disclose free-formatted text to external researchers Take advantage of linguistics experts Mitigate hurtles/risk of sharing PHI How to mine while respecting patient privacy? Solution: De-identify Protected Health Information (PHI) Identifiers

6 6 PHI in Free-Formatted Text: De-Identification Challenge Start with known PHI object, locate PHI elements, and de-identify HISTORY OF PRESENT ILLNESS: The patient is a 77-year-old-woman with long standing hypertension who presented as a walk-in to me at the Sun Hill Medical Center on August 12th. Recently had been started q.o.d. on Clonidine since June 8th to taper off of the drug. Was told to start Zestril 20 mg. q.d. again. Patient sent to Jones Cardiac Unit for direct admission for cardioversion and anticoagulation, with the Cardiologist, Dr. Pearson to follow. Sample PHI Object Doctors notes Discharge summaries Letters

7 7 Today s Products Consumer Off the Shelf (COTS) de-identification tools While some are standalone, many are components of larger, expensive data and network management tools For unstructured data, identification tends to rely on brute force keywords and regular expressions Lists of names Hand-crafted patterns requiring skilled developers $$$$ There is no solution that is 100% full proof including manual de-identification.

8 8 MIST: Training a De-identification System mark PHI automatically in more documents using model train (better) train model model from from initial (more) documents documents hand-correct automatically marked documents Tag-A-Little, Learn-A-Little (TALLAL) Methodology mark PHI by hand in initial documents redact or resynthesize marked documents

9 9 MIST: Training a De-identification System HISTORY OF PRESENT ILLNESS: The patient is a 77-year-old-woman with long standing hypertension who presented as a walk-in to me at the [HOSPITAL Oak Sun Valley Hill Medical AA Health BB CC Center DD] on August [DATE July 9th 12th YY]. Recently had been started q.o.d. on Clonidine since [DATE June May 5th 8th ZZ] to taper off of the drug. Was told to start Zestril 20 mg. q.d. again. The patient was sent to the [HOSPITAL Jones Smith Cardiac EE FF Unit GG] for direct admission for cardioversion and anticoagulation, with the Cardiologist, Dr. [DOCTOR Faulkner Pearson HH] to follow. Step 1 Find the identifiers Step 2 Transform the identifiers

10 10 DeMISTifying Deidentification Study 8 hours of marking (training) data Narrative from patient records 95% accurate as a measure of precision (false positives) and recall (false negatives) (Favorably) comparable to manual reviews The MIST results Top score in the first i2b2 De-identification Challenge Evaluation Used to de-identify medical records by hospitals Has led to numerous collaborations on MITRE projects Rapidly portable adaptable to other domains The MIST TALLAL approach works well for a large corpus Precision/recall tradeoff can be adjusted

11 11 How Good is Manual De-id by Humans? 120 Counts of overlooked PHI # Leaked PHI Instances MIST (Single Model) In 100 Family Practice notes Containing 1,093 PHI instances Reviewed by: Individuals Pairs of reviewers Trios of reviewers 0 Individuals Pairs Trios From: Is the Juice Worth the Squeeze? Costs and Benefits of Multiple Human Annotators for Clinical Text De-identification. D. S. Carrell; D. J. Cronkite; B. A. Malin; J. S. Aberdeen; L. Hirschman, submitted to Meth of Info in Medicine

12 DeMISTifying Deidentification On Steroids Hiding in Plain Sight (HIPS)* Research 12 Resynthesis: HISTORY OF PRESENT ILLNESS: The patient is a 77-year-old-woman with long standing hypertension who presented as a walk-in to me at the Sun Hill Medical Center on August 12th. Recently had been started q.o.d. on Clonidine since June 8th to taper off of the drug. Was told to start Zestril 20 mg. q.d. again. Patient sent to Jones Cardiac Unit for direct admission for cardioversion and anticoagulation, with the Cardiologist, Dr. Pearson to follow. HIPS Principles: *HIPS is a collaborative research effort among GroupHealth, Vanderbilt, and MITRE Hypothesis: With good resynthesis, it can be nearly impossible to detect leaked PHI manually OR using data mining hackers. Initial research results: Good resynthesis reduced the detection of PHI leaks by at worse case an additional 85%. Additional testing with larger sample sizes is necessary to validate.

13 Use Case #2 - Identifying PHI in Free-Formatted Text 13 Dr. Famous and his Laptop (1) - Loss of control over unencrypted laptop which contains that may have protected health information (PHI) back-up is available Need to establish extent of PHI content in for Health Information Portability and Accountability Act (HIPAA) Solution: Intensive manual review ($$$$$$) Research Seedling: Is there a possibility MIST can facilitate PHI discovery in ? 1. Tagging and Modeling: Can MIST successfully tag PHI identifiers in and be trained to model e- mail tagging? 2. If Step 1 is successful, identify prospective next steps for re-purposing MIST as an identification tool for locating PHI in (1) Halamka, Dr. John D., Surviving the Cybersecurity Cold War: A CIO s Practical Guide for Risk Management, slide 1

14 Problem Scope Compromised Assets 14 HIPAA 700,000 covered entity providers under HIPAA(1) Breaches 2012 Ponemon Institute Study(2) on Provider Breaches: 94% of healthcare organizations had at least one breach in the prior 2 years 45% had more than 5 breaches in the prior 2 years 46% of breaches caused by lost or stolen computing device Connectivity 81% of organizations permit employees and medical staff to use their own personal mobile devices to connect to their networks or enterprise systems (such as ) Enterprise data-at-rest encryption solutions offer partial risk mitigation (1) Department of Health and Human Services, 45 CFR Parts 160 and 164, Federal Register, Vol. 78, No. 17, Part II, January 25, 2013 (2) Ponemon Institute Research Report, Third Annual Benchmark Study on Patient Privacy and Data Security, Dec 2012

15 15 Problem Creep Identifying other types of sensitive data of in compromised assets Sensitive personal identifiable information (PII) such as financial information (as per Gramm-Leach-Bliley) PII defined in state consumer protection and/or breach notification laws Proprietary data, Sensitive but Unclassified (SBU), etc. Loss of control over a device with potentially multiple sources of sensitive unstructured data; Risk assessment requires reconstructing the contents of the device Backups Information on application servers , SharePoint, Database, Etc.

16 16 The Challenge of PHI Discovery Protected Health Information is: Created or received by a covered entity AND Identifies an individual (or is identifiable) (i.e., contains PHI identifiers) AND Relates to the individual s past, present, or future physical or mental health, the provisioning of health for the individual, or payment HIPAA de-identification is governed by a safe-harbor standard 17 fields plus a catch-all PHI identifiers In with MIST is different from the narrative, clinical note domain Structure, context mix of text and control characters

17 MIST PHI Identification Results 17 Based on one run Precision: Fraction of identified terms that were relevant = Recall: Fraction of relevant terms that were identified = F-measure: 2 ((precision recall) / (precision + recall)) = Additional modeling Precision: Recall: F-measure: 2 ((precision recall) / (precision + recall)) = (was 0.736) Given (many) constraints, MIST performance was (very) encouraging for processing Opportunity for significant improvement with an enhanced tag set and -aware blocking software Additional hours/dollars needed for product-ready solution

18 18 Privacy Risk Management Automated Tool Automated tools to support data-associated risk management PHI de-identification of doctor s notes Hiding in Plain Sight Resynthesis of believable fake data Tailorable (TALLAL) tool at the ready for assessing leakage of {PHI, PII, proprietary information, sensitive information, } in freeformatted text (e.g., )? How about a tailorable automated tool that supports privacy risk management?

19 What is MITRE s Privacy Risk Identification and Management Engine ((P)RIME*)? 19 RIME is a foundation for organizations to normalize and manage risk Organization defines RIME-hosted instance Web front-end with database backend RIME provides engines for: Risk Managers Risk metrics Dashboards Automated compliance document generation Risk management (raw vs. residual risk) PII Owners (Business Application, Program, System) Dynamic question-naires Cursor sensitive, as-needed help Immediate risk feedback * (P)RIME - Initially designed with Privacy as the risk focus

20 20 Risk Identification and Management Engine User- facing artifacts Questionnaire Templates Risk Areas & Priorities Risk Mitigation Suggestions Document Templates R I M E WEB FRONT-END: ENGINES: DATABASE: Context-dependent questions Cursor-sensitive help Dashboard Risk identification, priority and metrics (Compliance) document generation Dynamic Questionnaire Processing Risk Analysis Dashboard Generation Risk Management Document Creation Organizationally-defined elements Privacy Impact Assessments (PIA s) Privacy Threshold Assessment (PTA s) Data-Rich Completed Questionnaires Access Control Document Templates Query Results Dashboard Contents Key: Defined by the Organization Management-facing artifacts

21 [P]RIME Big Ideas 21 1 Move away from Word, etc., documents as risk tool 3 Push privacy SME-ness to PII owners 2 Separate data gathering from risk analysis and [compliance or other] document generation Eliminate redundancies (and wasted time), reduce inconsistencies 4 Covert, timely awareness by making privacy risk more explicit Empower risk managers with risk metrics and tools Inject discipline and consistency The The MITRE Corporation. All All rights rights reserved. For Internal MITRE Use. mitre mitre 21

22 22 P[RIME] Demo Functionality is real Production Use Ready for transfer to sponsor and to industry For optimal results, MITRE should assist with initial instantiation effort Instantiation Examples Traditional PIA, breach, inventory support Down-select from a complete set of possible risk issues Automated requirements/testing support

23 MIST, HIPS, PRIME, and other Privacy or Cybersecurity Tools 23 For more information, please contact Cathy Petrozzino at