Big Data Analytics for Mitigating Insider Risks in Electronic Medical Records

Transcription

1 Big Data Analytics for Mitigating Insider Risks in Electronic Medical Records Bradley Malin, Ph.D. Associate Prof. & Vice Chair of Biomedical Informatics, School of Medicine Associate Prof. of Computer Science, School of Engineering Vanderbilt University 21/8/2015

2

3 January 1, 2015 Logged over 2,000,000 users interactions Alice s Alice s Electronic Alice s Electronic Medical Electronic Medical Medical Record ecord Record Alice s Electronic Medical Record

4 January 2, 2015 Logged over 2,000,000 users interactions

5 January 3, 2015 Logged over 2,000,000 users interactions

6 Auditing Requirements Federal (US) 1. Access control 2. Track & audit employee accesses 3. Store logs for 6 years 6

7 7

8 How (Not) to Use Access Control Central Norway Health Region enabled break the glass 1/2 of 99,000 patients broke glass 1/2 of 12,000 users broke glass Role Users Break Grass Nurse % Doctor % Health Secretary % Physiotherapist % Psychologist % ~300K events in 1 month (Røstad & Øystein 2007) 8

9 Oct 2007 Palisades Medical Center Dozens of Employees 9

10 July 8, 2011 UCLA HHS Investigation $1 million fine 10

11 The Model is Wrong 11

12 Learning Suspicious EMR Access Behavior (Boxwala et al, JAMIA, 2011) Manually select 505 potential cases / controls based on previous breaches at Partners Healthcare Model Support Vector Machines Logistic Regression LABEL Human experts label cases as + / SELECT New unlabeled events from DB PREDICT Calculate the prediction probabilities using classifier on all events BUILD Build classifier from labeled events 12

13 Learning Suspicious EMR Access Behavior (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department Same street address Same family name Over 200 accesses in a day VIP Patient Same Zip Code Is Provider Care unit visit match

14 Role Refinement Northwestern Memorial Hospital 3 months of access logs to inpatient records 8K users, 16K patients, 1.1M accesses User ID Patient ID User position Date / Time Number of Orders Entered Patient Location in hospital Service patient is on (Zhang, Gunter, Liebovitz, Tian, & Malin AMIA 2011) 14

15 Predictability is Job Dependent MOST PREDICTABLE Rank Most Predictable Accuracy Users 1 (tie) ED Assistant 100% 26 1 (tie) ED Physician CPOE 100% 43 1 (tie) NMH Resident/Fellow ID Clinic-CPOE 100% 10 LEAST PREDICTABLE Rank Least Predictable Accuracy Users 140 Patient Care Staff Nurse 7.6% Rehab Occupational Therapist (OT) 14.3% Patient Care Staff Nurse (Pilot) 22.1%

16 Where are We Going Wrong? Actual Role Predicted Role Probability Rehab Rehab Occupational Therapist Physical Therapist 85.7% Rehab Rehab Physical Therapist Occupational Therapist 60.0% 16

17 Suspicious? 17

18 Suspicious or Anomalous? 18

19 Defining Access Control Detecting Suspicious Behavior 19

20 January 1 EMR users linked if they accessed 1 patient in common (Malin, Nyemba, Paulett 2011) 20

21 Mining to Model the System (Malin, Nyemba, Paulett 2011) 2 nd Principal Component University Hospital Children s Hospital 1 st Principal Component 21

22 Hypothesis! Collaborative systems are about social phenomena People should form communities We should be able to measure deviation from community structure Note: other social phenomena could be studied (temporal workflow*, function invoked* if any, etc.) (*Chen et al, IEEE TDSC 2012; Zhang et al. ACM SACMAT 2013; Zhang et al. ACM TMIS 2013) 22

23 Community Based Anomaly Detection (CADS) Pattern Extraction Anomaly Detection Access Logs Social Relation Construction User Communities Distance Measurement User Specific Deviation Scores Community Deviation Deviation Measurement (Chen & Malin ACM CODASPY 2011) 23

24 Example 6 Nearest Neighbor Network (1 day of accesses) The average cluster coefficient for this network is 0.48, which is significantly larger than for random networks Users exhibit collaborative behavior in the health information system 24

25 Auditing Strategies of the Past Principle Components Analysis (PCA) Graph based anomaly detection (Shyu et al 2003) (How similar am I to spectral clusters of users?) K Nearest Neighbor (KNN) Nearest neighbor based anomaly detection (Liao et al 2002) (How similar am I to my friends?) High Volume Model (Gallagher et al 1998) (Do I access way more people than my relations?) 25

26 Social Structure Wins the Day! True Positive Rate False Positive Rate 26

27 Gripes & Future Musings Different providers within the same ward have different behavior! Different wards within the same healthcare institution have different behavior! Different healthcare organizations use different languages! Logic (i.e., access control) and AI (i.e., data mining) need to play nicely together 27

28 Questions? Health Information Privacy Laboratory 28

29 29

30 High Confidence Rules Rule Support Confidence Weeks Center for Patient & Professional Advocacy Hearing & Speech Practice City A Clinic City A Infectious Disease Clinic Infectious Disease NICU Neonatology VMG Family Practice Clinic City A Vanderbilt Hearing School Hearing & Speech

31 Low Confidence Rules (but occur in at least 3 weeks) Rule Support Confidence Weeks Anesthesiology Vanderbilt Hearing School Anesthesiology 4N Labor & Delivery Anesthesiology Physician Liaison Program Emergency Medicine Nutrition Clinic Anesthesiology Cardiac Cath Lab Emergency Medicine Diabetes Ctr Anesthesiology Center for Clinical/Research Ethics Anesthesiology Infectious Disease Clinic Anesthesiology Pediatric Immunology Anesthesiology Mental Health Center

32 Big Data Audits Must Be Understandable to be Actionable 32

33 What Makes Sense? Dr. Smith s access of Peggy Johnson s medical record was strange Dr. Smith s access was 10 standard deviations away from normal behavior in his hospital Dr. Smith s access was strange because he is a neonatologist and he accessed the record of a 100 year old woman who, for the past year, has only been treated by gerontologists 33

34 So Do You Believe Inferred Patterns? 34

35 Hypothesis: Locally Knowledgeable of Class Anethesiologists Psychiatrists Coding & Charge Entry Medical Information Services Ane. Rules Psych. Rules Code Rules MIS Rules High (10) High (10) High (10) High (10) Medium (10) Medium (10) Medium (10) Medium (10) Low (10) Low (10) Low (10) Low (10) 35

36 Survey Employees presented with questions asked to report likelihood of rules on a 5 point Likert scale All employees asked the same set of 120 questions (four sets of 30) Someone from Anesthesiology accessed the record of patient John Doe. How likely is it that someone from the following organizational area accessed the same patient's record? Anesthesiology Psychiatry Not at all Not at all Slightly Moderately Very Completely Slightly Moderately Very Completely 36

37 Hypothesis: Locally Knowledgeable of Class Anethesiologists Ane. Rules High Medium Low Employees can distinguish between high, med, and low for their own rules Anesthesiologists evaluated with anesthesiology rules Tested hypothesis with linear mixed effects (LME) model 37

38 Hypothesis: Locally Knowledgeable of Class Anethesiologists Confirmed for every organizational area at 95% confidence level! Ane. Rules High Medium Low Area Strength p value ANE CODE MIS PSY

39 Learning Suspicious EMR Access Behavior (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio 39

40 Learning Rules for Suspicious Access Detection (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department

41 Learning Rules for Suspicious Access Detection (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department Same street address

42 Learning Rules for Suspicious Access Detection (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department Same street address Same family name

43 Learning Rules for Suspicious Access Detection (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department Same street address Same family name Over 200 accesses in a day

44 Learning Rules for Suspicious Access Detection (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department Same street address Same family name Over 200 accesses in a day VIP Patient

45 Learning Rules for Suspicious Access Detection (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department Same street address Same family name Over 200 accesses in a day VIP Patient Same Zip Code

46 Learning Rules for Suspicious Access Detection (Boxwala et al, JAMIA, 2011) Feature Coefficient Odds Ratio Works in the same department Same street address Same family name Over 200 accesses in a day VIP Patient Same Zip Code Is Provider

47 Predictability is Job Dependent Prediction Accuracy 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Med Student CPOE NMH Resident / Fellow CPOE Patient Care Staff Nurse Rehab OT Number of Users in Role 47

48 Another Healthcare Environment Vanderbilt EMR Logs 6 months Arbitrary Week 2,500 users 35,000 patients 66,000 <user, patient> distinct accesses 48