Copyright 2013 Splunk Inc. HIPAA and Meaningful User Audit Reports Using Splunk Ant Lefebvre Senior Systems Engineer, Middlesex Hospital #splunkconf
About Middlesex Hospital!! We offer a complete range of medical services Some of ConnecMcut s highest quality and pament samsfacmon ramngs 25 Networked Offsite LocaMons 9 Primary Care Offices 3 Emergency Departments! Named to 100 Top Hospitals list two years running! Named to HealthCare s Most Wired List 2012 & 2013 2
Who Am I?! Systems Engineer! Network Engineer! Security / Compliance! Wireless! IT Director! IT Consultant! Splunker 3
Splunk for Hospital Network OperaMons
Challenges in Healthcare VirtualizaMon Management ApplicaMon Performance Event Log CorrelaMon Global View of Environment 5
Hospital s Visibility Gap! Windows event viewer is not easy to navigate! TroubleshooMng mulmple hosts means opening each log individually! CorrelaMng event Mmes in mulmple systems a manual process! Host down or off network made it impossible to access logs! Took hours or days to find root cause(s) for end user device issues Wasted (me and effort to track down issues 6
Splunk Solves Visibility Gap Steps to success: 1. Downloaded free demo 2. Globally installed Splunk Universal Forwarders on Windows server and client operamng systems 3. Indexed Windows event log data 4. Instantly gained visibility into windows environment like never before Troubleshoo(ng (me now a frac(on of what it used to be
Splunk Enterprise in ProducMon! Finding new use cases every day! Audit consolidamon One tool to monitor all systems! Event correlamon Is the issue happening everywhere? When?! Recognize anomalous acmvimes Something strange going on?! Add new log sources See what shakes out No need to purchase addi(onal products. Index the data in Splunk Enterprise! 8
Middlesex Splunk Enterprise Success Stories Mystery wireless disconnects persisted for years; using Splunk Enterprise searched on User ID / tablet name at drop Mmes; discovered crashing process on Citrix server at dropping event Mme! Mystery name resolumon issues; connecmng to wrong workstamons when using hostname; *error* search found DNS record scavenging was accidentally off amer AD/DNS server migramon Started to index firewall traffic logs; using Splunk Enterprise and Google Maps app discovered a Health library machine connected to an internamonal bot net; no business need to communicate with Peru Used Splunk Enterprise to discover slowest boomng computers to priorimze new PC rollout; transacmons from first boot service start to last boot service to start User files vanish. File audit tool gave no insight; Splunk search for user id AND delete finds over 300 events in an hour over the weekend; user accidentally deleted one too many folders The list goes on and on 9
Botnet Computer 10
Blocking Streaming HDTV Through Firewall 11
Boot Times Table 12
Found File DeleMon Incident 13
Table of Files Deleted Report 14
Program Intelligence into Apps/Dashboards! Created useful dashboards for operamons/helpdesk team! Don t need to know Splunk search commands to use! Help less knowledgeable staff troubleshoot environment issues! Each new dashboard is created in- house; no need for addimonal purchase; no need to ask for product enhancement or feature from vendors! Single point of reference for mulmple uses The Splunk Admin can create point and click knowledge 15
Citrix Disconnect Dashboard 16
Power Dashboard 17
Windows NPS RADIUS Dashboard 18
Print Server Log Dashboard 19
Print User to IP CorrelaMon Print logs do not contain where user prints from Windows Event logs show where user last logged in 20
Viral Spread of Splunk Enterprise Word of Splunk Enterprise capability to audit systems and solve mysteries tricked through other IT staffers AddiMonal systems I didn t even know we had were added to Splunk Enterprise 21
IT Director s Challenge! A system to audit our Electronic Health Record access! A single solumon to audit mulmple systems! Easy to manage! Cost is always a factor! We have two opmons. Which one is beqer?! The answer: OpMon 3 Splunk! 22
HIPAA and Meaningful Use
Healthcare Jargon! EMR/EHR Electronic PaMent Records! HIPAA The Health Insurance Portability and Accountability Act of 1996! HITECH Act Health InformaMon Technology for Economic and Clinical Health Act! Meaningful Use Goal is to not just adopt an EHR, but to leverage it to achieve significant improvements in care! Cerner Middlesex Hospital s Primary EHR! Results Middlesex Hospital s home grown EHR lookup applicamon! eclinicalworks Middlesex Hospital s Primary Care / Family PracMce / MulMspecialty EHR 24
Sweetening the Deal Managing EHRs! Federal reimbursement for having cermfied technologies to audit Electronic Health Record (EHR) access, enforce Meaningful Use! EHR provider offers specialized (and expensive) point solumon Only shows who s logged in to app! Experiment: EHRs into Splunk no problem Splunk provides audi(ng capabili(es & delivers opera(onal intelligence. 25
Raw EHR Audit Data <audit_list><audit_version>1</audit_version><event_dt_tm>2013-08-23 08:30:06.00</event_dt_tm><outcome_ind>0</ outcome_ind><user_name>system</user_name><prsnl_id> 1.000000</prsnl_id><prsnl_name>SYSTEM</prsnl_name><role>DBA</ role><role_cd>24209801.000000</role_cd><enterprise_site>hnam</enterprise_site><audit_source>test/default Logical Domain</ audit_source><audit_source_type>274986</audit_source_type><network_acc_type>1</network_acc_type><network_acc_id>mhscnpap</ network_acc_id><context><![cdata[mzq2nzgyotc3fdi3ndk4nnwynzmwmjj8mjy1mte4fdi0fa==]]></context><application>scs Netting Server</ application><task>update SCS Netting Task</task><request>scs_get_proc_server_netting</request><appl_ctx>346782977</ appl_ctx><perform_cnt>24</perform_cnt><event_list><event_name>maintain Order</event_name><event_type>Tasks</ event_type><participants><participant_type>system Object</participant_type><participant_role_cd>Order</ participant_role_cd><participant_id_type>order</participant_id_type><participant_id>419526210.000000</ participant_id><participant_name>blood, Timed Study collect, 08/23/13 5:00:00, Lab Collect</participant_name><data_life_cycle>Origination/ Amendment</data_life_cycle><person_id>4480371.000000</person_id><person_name>BCMA, Dana</person_name><vip_display></ vip_display><encounter_id>15571493.000000</encounter_id><encounter_org>middlesex HOSPITAL</encounter_org><medical_service>Medical Services</medical_service><location>CCU</location><encounter_confid_level></encounter_confid_level><admit_dt_tm>2012-11-21 13:53:23.00</ admit_dt_tm><discharge_dt_tm>0000-00-00 00:00:00.00</discharge_dt_tm><encounter_type>Inpatient</encounter_type><encounter_status>Active</ encounter_status><encounter_mrn>9913</encounter_mrn><encounter_fin>11452</encounter_fin><relationship_creation_reason></ relationship_creation_reason><relationship_creation_dt_tm>0000-00-00 00:00:00.00</relationship_creation_dt_tm><relationship_created_by></ relationship_created_by><relationship_creation_type></relationship_creation_type><relationship_type></ relationship_type><participant_query></participant_query><facility>middlesex HOSPI</facility><building>Middlesex Bld</ building><nurse_unit>ccu</nurse_unit><room></room><bed></bed><external_source></external_source><person_alias></ person_alias><sensitivity_codes></sensitivity_codes></participants></event_list><alt_user_name></alt_user_name><user_organization_name></ user_organization_name><user_organization_cd> 0.000000</user_organization_cd><personnel_role></ personnel_role><application_number>274986</application_number><task_number>273022</task_number><request_number>265118</ request_number><prsnl_alias></prsnl_alias><user_organization_alias></user_organization_alias></audit_list>! Splunk to the rescue 26
Under the Hood IngesMng Cerner EHR audit data into Splunk Cerner Audit Outbound Server Cerner Listener / Splunk Universal Forwarder Splunk Indexer 27
Under the Hood Part 2 IngesMng Results EHR audit data into Splunk Results Backend Server FTP server / Splunk Universal Forwarder Splunk Indexer 28
Not Sure What Hood to Look Under IngesMng eclinicalworks EHR audit data into Splunk??? / Splunk Universal Forwarder Splunk Indexer Engage your EHR vendor EARLY! 29
Vision Into Our Future Compliance Officers, Auditors, ApplicaMon Staff, OperaMons Team, Infrastructure Team Splunk search heads with TAs (Technology Add - ons) and a Common Healthcare App Splunk indexing mulmple diverse, but related systems EHR, Finance, Infrastructure, Clients, Servers, the list goes on. 30
Middlesex Hospital s Cerner EHR App! ApplicaMon Report Categories AcMvity Audit Admin Audit Disclosure Report Login Report PaMent Record Access Suspicious AcMvity User Account Sharing VIP PaMent Access! New reports are only limited by the logs and the imaginamon 31
Cerner EHR App Demo Before we begin Application demo with test environment data. Application written specifically for Cerner EHR for MU2 Common Information Model in development! Get right to the facts! Compliance isn t preqy! Auditors are going to love it!! Meaningful Use of EHR logs! HIPPA violamon invesmgamon made easy Universal Healthcare App in development 32
What s Next?! Common InformaMon Model for Healthcare! Universal Meaningful Use and HIPAA App across mulmple systems! Onboard more systems. Greater visibility!! VMware and Citrix Apps on hqp://apps.splunk.com/ When we need to know what happened in and on our systems, We turn to splunk> 33
THANK YOU