Copyright 2014 Splunk Inc. Using Splunk to Protect Pa=ent Privacy and Achieve Meaningful Use Ant Lefebvre ant@midhosp.org Senior Systems Engineer Middlesex Hospital
About Middlesex Hospital Complete range of medical services Some of Connec?cut s highest quality and pa?ent sa?sfac?on ra?ngs 30 Networked Offsite Loca?ons 10 Primary Care Offices 3 Emergency Departments Recipient of the CIO 100 award for our use of Splunk sonware 100 Top Hospitals list for two years running HealthCare s Most Wired List 2012-2014
whoami? Systems Engineer Network Engineer Security / Compliance Wireless/Wired IT Director IT Consultant Splunk.conf 2013 Revolu?on Award Winner!
Hospital Network Opera=ons
Challenges in Healthcare Virtualiza?on Management Applica?on Performance Event Log Correla?on Global View of Environment
Hospital s Visibility Gap Not easy to navigate Windows event viewer Log by log review for troubleshoo?ng Manual event correla?on spanning mul?ple systems No log access when host down or off network Hours/days to find root cause(s) for end user device issues Wasted (me and effort to track down issues
Steps to success: Splunk Solves Visibility Gap 1. Downloaded free demo 2. Globally installed Splunk Universal Forwarders on Windows server and client opera?ng systems 3. Indexed Windows event log data 4. Instantly gained visibility into Windows environment like never before Troubleshoo(ng (me now a frac(on of what it used to be
Splunk in Produc=on Finding new use cases EVERY DAY!! Audit consolida?on One tool to monitor all systems Event correla?on Is the issue happening everywhere? When? Recognize anomalous ac?vi?es Something strange going on? Add new log sources See what shakes out No need to purchase addi(onal products. Index the data in Splunk.
Success Stories Mystery wireless disconnects persisted for years. Using Splunk searched on User ID / tablet name at drop?mes. Discovered crashing process on Citrix server at dropping event?me! Wireless disconnects reported HERE Root cause was back end service crashing in datacenter
What computer am I connected to? Mystery name resolu?on issues. Connec?ng to wrong worksta?ons when using hostname. *error* search found DNS record scavenging was accidentally off aner AD/DNS server migra?on. Want to connect to PC A Helpdesk But get connected to PC B
Finding a Botnet Index firewall traffic logs using Splunk and Google Maps Discovered a health library machine connected to an interna?onal botnet No business need to communicate with Peru
Boot Times Table
Found File Dele=on Incident User files vanish with no insight from file audit tool Search for user id AND delete finds over 300 events in an hour over the weekend User accidentally deleted one too many folders
Blocking streaming HDTV through Firewall
Program Intelligence into Apps/Dashboards Created useful dashboards for opera?ons/helpdesk team No need to know Splunk search commands to use Help less knowledgeable staff troubleshoot environment issues Each new dashboard is created in- house. No need for addi?onal purchase. No need to ask for product enhancement or feature from vendors. Single point of reference for mul?ple uses The Splunk Admin can create point and click knowledge
Citrix User Login Finder
Find Server Behind Load Balancer
Where has this user logged in?
Most Numerous Cisco Syslog Messages
Web Traffic!
Power Dashboard
Windows NPS RADIUS Dashboard
Print Server Log Dashboard
Print User to IP Correla=on Print logs do not contain where user prints from. Windows Event logs show where user last logged in.
Viral Spread of Splunk Word of Splunk s capability to audit systems and solve mysteries trickled through other IT staffers. Addi?onal systems I didn t even know we had were added to Splunk.
IT Director s Challenge A system to audit our Electronic Health Record access. A single solu?on to audit mul?ple systems. Easy to manage. Cost is always a factor. We have two op?ons. Which one is beher? The answer: Op?on 3 Splunk!
Pa=ent Privacy & Meaningful Use
EMR/EHR - Electronic Pa?ent Records Healthcare Jargon PHI/ePHI (electronic) Protected Health Informa?on HIPAA - The Health Insurance Portability and Accountability Act of 1996 HITECH Act - Health Informa?on Technology for Economic and Clinical Health Act Meaningful Use Goal is to not just adopt an EHR, but to leverage it to achieve significant improvements in care Cerner - Middlesex Hospital s Primary EHR Results Middlesex Hospital s home grown EHR lookup applica?on eclinicalworks Middlesex Hospital s Primary Care / Family Prac?ce / Mul?specialty EHR McKesson Homecare Middlesex Hospital Home care EHR GE Flowcast Pa?ent registra?on/demographics Lawson Employee Database
Electronic Health Record Audi=ng Federal reimbursement for having cer?fied technologies to audit Electronic Health Record (EHR) access, Meaningful Use Requirement Splunk v6.0 is currently v1.0 Cer?fied (for both Ambulatory and Inpa?ent) 170.314(d)(3) - Audit report(s) EHR provider offers specialized (and expensive) point solu?on Other EHR vendors couldn t correlate between systems/databases Other vendor solu?ons specific to their product. Can t build intelligence.
Splunk for MU2 EHR Module, 2014 Edi=on means EHR technology that is cer?fied to at least one of the 2014 Edi?on EHR cer?fica?on criteria for either the ambulatory or inpa?ent prac?ce sepng. An EHR Module could provide a single capability required by one cer?fica?on criterion or it could provide all capabili?es but one, required by the cer?fica?on criteria for a Complete EHR. Splunk is 1 of 20 modules required to meet Base EHR defini?on for 2014 Edi?on EHR cer?fica?on. 170.314(d)(3) Audit reports Required 170.314(g)(4) Quality Management System Needed for all modules Splunk will not fulfill your EHR product cer?fica?on alone, but will check the (d)(3) Audit Report(s) box on the cer?fied health IT product list: hhp://oncchpl.force.com/ehrcert/ehrproductsearch or hhp://goo.gl/5pshd
Primary vendor solu=on EHR Vendor Audit Repository Data Inputs Similar in ability to Splunk Much more expensive to implement Very lihle if any community support New inputs require vendor services to implement Data elements have to be pre- programed into repository
Other vendor solu=ons Each system has its own audi?ng capabili?es (maybe) No way to centrally look into all system access. Log into each app to run access reports Advance inves?ga?ve dashboards unavailable, limited, or costly to implement
Taking a stab at an EHR audit App Newbie Splunk user s first App Cerner audit data only PoC rolled into preliminary App Much development needed Worked well enough to sa?sfy audi?ng requirements
Challenges in building the App First of it s kind in Splunk I am not a compliance officer I am not a developer Limits on my?me Only IT staffer with end game in focus
Raw EHR formats? XML with checksum to prevent tampering SQL Human Readable Columns Key Value Pairs Splunk Comma Separated Value Splunk to indexes ALL! mysql
Under the Hood Inges?ng Cerner EHR (XML format) audit data into Splunk By far the most comprehensive audi?ng Cerner Audit Outbound Server Cerner Listener / Splunk Universal Forwarder Splunk Indexer Real- (me Audit Events
Under the Hood Part 2 Inges?ng CSV exports into Splunk Results Flowcast Lawson FTP server / Splunk Universal Forwarder Yesterday s Audit Events Splunk Indexer
Under the Hood Part 3 Inges?ng database EHR audit data into Splunk ECW mysql McKesson Homecare SQL DB Connect/ Splunk Heavy Forwarder Engage your EHR vendor EARLY! Near Real- (me Audit Events Splunk Indexer
Healthcare App fields? EHR A: 35 fields Employee Database EHR B: 15 fields Homegrown EHR Splunk EHR C: 5 fields Pa?ent Registra?on App Healthcare common informa(on model?
HIPAA Privacy and Security Scout Healthcare Compliance Splunk App HIPAA Privacy and Security Scout and HIPAA Scout are protected by U.S. and interna?onal copyright and intellectual property laws. Middlesex is able to ensure that staff is compliant with State and Federal privacy regula?ons. The hospital has the ability to monitor user level access to several EHR systems from single interface using Splunk Healthcare CIM. App is available from Splunk Partner Conducive Consul?ng - hhp://www.conducivesi.com
What HIPAA Scout Provides Get right to the facts Compliance isn t prehy Auditors are going to love it! Meaningful Use of EHR logs HIPAA viola?on inves?ga?on made easy Common Informa?on Model Universal EHR Audi?ng App
HIPAA Privacy and Security Scout Auditor Home Page Quick links to most used reports Applica=on Report Categories Ac?vity Audit Admin Audit Disclosure Report Employee Info Login Report Inves?ga?ons Suspicious Ac?vity User Account Sharing VIP Pa?ent Access New reports are only limited by the logs and the imagina=on Every hospital is different. Requirements and problems vary.
HIPAA Privacy and Security Scout Most Useful Dashboards Record Access Inves?ga?on Coworker Record Access Same Last Name Wrong Unit Employee Admission Report Same Street Example Fields Available for Inves=ga=ons User Name User ID Pa?ent Name Medical Record Number Account Number Hospital Unit Number
Example Dash: Same Last Name 44
Example Dash: Wrong Unit 45
Example Dash: Record Access Inves=ga=on 46
Example Dash: Coworker Record Access 47
Example Dash: Record Print by Pa=ent 48
Splunk & Compliance " Re- dran our policies on regarding what a HIPAA viola?on actually is. " Create policies regarding how we will move forward with Splunk & HIPAA Privacy and Security Scout app. " Will we survive an audit? We have the power. Use it! " Educate the masses. Goal is for Splunk to find nothing. 49
Barriers to Progress " Beher at finding poten?al viola?ons. Takes more?me to inves?gate. Splunk is too good! " EHR vendors don t supply enough audit info to automate more. " Finding the informa?on with DB Connect takes lots of?me. Hope the schema doesn t change! " Vendors unable/unwilling to co- operate. 50
Vision into Our Future NOW Compliance Officers, Auditors, Applica?on Staff, Opera?ons Team, Infrastructure Team Splunk search heads with TAs (Technology Add- ons) and a Common Healthcare App Splunk indexing mul?ple diverse, but related systems EHR, Finance, Infrastructure, Clients, Servers, Systems, the list goes on.
Lessons Learned Budget for servers/storage. Don t roll PoC into produc?on system. Start fresh. Sync?mes before indexing (where is that s?nking real?me data?). Expect to frequent answers.splunk.com if you want to be successful. When inges?ng data, it helps to have friends on the inside. If I had known then what I know now
THANK YOU! Ant Lefebvre ant@midhosp.org Senior Systems Engineer Middlesex Hospital