Copyright 2014 Splunk Inc. Security OperaCons with Splunk App for Enterprise Security David Casey, Vice President, IT Security OperaCons Manager Flagstar Bank
Disclaimer During the course of this presentacon, we may make forward- looking statements regarding future events or the expected performance of the company. We caucon you that such statements reflect our current expectacons and escmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentacon are being made as of the Cme and date of its live presentacon. If reviewed aser its live presentacon, this presentacon may not contain current or accurate informacon. We do not assume any obligacon to update any forward- looking statements we may make. In addicon, any informacon about our roadmap outlines our general product direccon and is subject to change at any Cme without nocce. It is for informaconal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligacon either to develop the features or funcconality described or to include any such feature or funcconality in a future release. 2
Personal Background! CISSP/CISM/SnortCP/Dr.Evil! Joined Flagstar Bank in early 2013! 15+ Yrs IT Security! 18 Yrs U.S. Army Military Intelligence Experience in the following sectors:! DoD (Lockheed MarCn, NCI, SAIC)! EducaCon! Energy! Finance Specialize in building Security OperaCons programs from the ground up and major security ops overhauls due to compliance failures 3
Company Background Flagstar Bank! Full- service bank (Troy, Michigan)! $9.4 billion in total assets! 100 + Branches in Michigan! 39 Home loan centers in 19 states! NaConwide mortgage lender! One of the nacon's top 10 largest savings banks 4
In the Beginning There was Darkness! 2009-2012 Flagstar expanded business operacons very fast! Infrastructure changes, mind sets, technology could not keep pace! As Flagstar bank grew federal oversight shised! New auditors were assessing security in the same manner as the Chase s and Bank of America s! Flagstar had many, many audit findings! People, processes, and technology had to change! IT Security OperaCons grew significantly in order to meet compliance requirements effeccvely! A SIEM was a criccal component! The One Ring to rule them all! 5
SIEM Technology Decision When looking for a SIEM solu7on for Flagstar leveraged 12+ years of SIEM deployment experience as its guide. Lessons learned: Difficult geong data in (ingescng data) Hard to get clear results from ad- hoc queries Limited plaporm opcons Costly to operate/maintain Inflexible SIEM sales hype. Product vendors only want to sell you their product. No interest in truly helping you protect your organizacon. Splunk Experience ü Easy to get all machine data into the system ü Simple plain language search ü Uses commodity hardware ü IntuiCve, easy to use ü Flexible and easy to customize ü They actually want you to be successful and take great strides to make it so! 6
Splunk Deployment Current Design:! 2 Search heads, 3 indexers, 300+ GB/day! Data sources (current) All servers via forwarders Windows, 4 flavors of UNIX All networking devices (switch, router, wireless, VPN, etc.) Syslog systems Firewall, IPS, DLP, AnC- Virus Web proxy logs DNS, DHCP email Applica7ons Splunk for Windows apps (3) Splunk for UNIX app Various vendor security apps (<10) DBConnect More 7
Splunk Deployment Disaster Recovery (DR) Design:! Overall Splunk ecosystem managed from HQ site! 2 Search heads, 2 indexers! DR site forwards all logs to HQ site! HQ replicates last 72 hours of logs to DR Future State:! All data 100% replicated! Heavy Forwarders Deployed to both HQ and DR sites! SAN improvements >1000 IOPS sustained 8
Security OperaCons Monitoring Challenges Some7mes security technology is simply not enough it takes a human to help it all make sense! The cyber security threat landscape is constantly morphing, ever changing, with threat actors intent on by- passing common security controls that rely on known payerns and deteccon techniques! Humans are primarily a visual- based species! Splunk can provide a visual that speaks a thousand words by taking the complex and making it simple to understand! Take for instance the following case studies 9
Case Study #1 Are We Being Targeted?! Flagstar s IT Security OperaCons Team uses the Splunk Enterprise Security (ES) App to monitor for advance threats, including exploits, malware infeccons, monitoring blacklists, and responding to spikes in threat trends! One common threat gathering technique is finger princng/mapping out a targets public facing systems, its ports and services! Being scanned is very common and generally considered background noise just a part of doing business on the internet! But when the scan is coming from a country that is frequently a hoscle cyber threat, and the scan is performed slowly, non- aggressively, it can osen by- pass security controls that are designed to block more aggressive scans 10
Case Study #1 Are We Being Targeted?! Sample Splunk search:sourcetype="[hidden]" earliest=- 1m inbound geoip src search src_country_code!=us stats count AS count by src_country_name sort - count top limit=5 11
Case Study #1 Are We Being Targeted?! Upon closer inspeccon we were able to isolate the scans as originacng from the City of Nanning, China! We have no legicmate customers in China! Answer? Block the network range 12
Case Study #1 Con$nued There are many hoscle actors all over the world. Some of the top actors are Russia, Ukraine, and China. Take Russia for example. Sure seems like there are a lot of outgoing conneccons to a Russian IP address. Could this be a compromised host? Using Splunk we can watch closely outbound descnacons, by IP locacon, and respond more quickly when we see an increase in potencally risky IP traffic to known hoscle actor countries. 13
Case Study #1 Con$nued! Sample Splunk search (Russia Inbound): sourcetype= [hidden]" src_ip!= [internal networks excluded]" iplocacon src_ip search Country="Russia" where Country="Russia" chart count by src_ip sort - count top limit=5! Sample Splunk search (Russia Outbound): sourcetype= [hidden]" src_ip!=[exclude DNS server IP, web proxies, etc.] " iplocacon dest_ip search Country="Russia" where Country="Russia" stats count by src_ip,dest_ip rename src_ip AS "Client" dest_ip AS "Russia IP Address" count AS "Count" table Client,"Russia IP Address",Count sort - count by Count top limit=5 14
Case Study #2 Firewall Control AyestaCon! Flagstar s IT Security OperaCons Team uses the Splunk Enterprise Security (ES) app to help meet regulatory requirements (for IT Security Dept. only at this Cme)! One example was where federal auditors wanted to see where changes to the perimeter firewall were being monitored against approved firewall changes! If a change occurred outside of the change control process it should be noted and invescgated! Splunk was used to help idencfy all write and execute commands issued on the perimeter firewalls and graphically displayed for easy idencficacon! This solucon was accepted by the federal auditors! YMMV 15
Case Study #2 Firewall Control AyestaCon! Sample Splunk search: evenyype= [hidden]_privileged_accvity" "write" OR "111010" OR "101008" NOT ("Teardown" OR "conneccon" OR "exit" OR "ping" OR [hidden]") Cmechart span=15m count(host) by user sort _Cme 16
Case Study #3 Metrics Across Security Technologies! Flagstar s IT Security OperaCons Team uses the Splunk Enterprise Security (ES) App to track security metrics! Security metrics are commonly requested as *proof* that the $$$ invested in security technology is actually producing results! Rather then running separate reports from each security technology to determine the metrics, using Splunk simplified the process greatly! Remember that a picture tells a thousand words 17
Case Study #3 Metrics Across Security Technologies! Sample Splunk search (IPS): index=[hidden] sourcetype=[hidden] rec_type_simple="ips EVENT" Cmechart span=1mon count 18
Case Study #4 24 x 7 Monitoring! Flagstar s IT Security OperaCons Team uses the Splunk Enterprise Security (ES) app to provide 24x7 monitoring! Instead of spending $$$ on an external Managed Security Services provider that provides aser hours support, Splunk can be used to develop acconable dashboards monitored by the internal Network OperaCons Support Team (which works 24x7)! PotenCal savings can go towards other criccal security budget items NOTE: This case study is currently being developed and tested within Flagstar. It has not yet reached a point where it is ready to replace an external MSS provider 19
Case Study #4 24x7 Monitoring! Sample Splunk search: Available upon request 20
Case Study #5! Flagstar s IT Security OperaCons Team uses the Splunk Enterprise Security (ES) app to detect `Brute Force Login Ayempts and send automated alerts in real- Cme when detected 21
Case Study #6! Flagstar s IT Security OperaCons Team uses the Splunk Enterprise Security (ES) app to detect `Malware InfecCons and send automated alerts in real- Cme when detected 22
Case Study #5 & 6! Sample Splunk search (Brute Force Ayempt Email Alert): EventCode=4625 sourcetype="wineventlog:security" earliest=- 6m latest=now bucket _Cme span=5m stats count by _Cme, Account_Name, src_ip,dest where count > 500! Sample Splunk search (Malware Email Alert): index=[hidden] sourcetype=[hidden] NOT ("Actual accon: Cleaned*" OR "Actual accon: QuaranCned" OR "Actual accon: Deleted") rename "event_cme" as "Detected" actual_accon as "AcCon" dest_nt_host as "Host" dest_ip as "Host IP" user as "User" risk_type as "DetecCon Type" signature as "Malware Name" table "Detected" "Host" "Host IP" "User" "DetecCon Type" "Malware Name" "AcCon" sort by "Detected" 23
The Future of Splunk @ Flagstar! We re planning to bring addiconal data into Splunk over the next 12 months Database logs & custom applicacon server logs Wide range of banking applicacons and regulatory data Endpoint (client) systems Third party hosted logs (various)! Explore the value of the prediccve analysis capability! Bring in Splunk Pro Services periodically to assist in maximizing Splunk's investment and to perform Splunk health checks 24
Ques7ons? Flagstar IT Security Opera7ons (SecOps) Team 25
Learn, share and hack Security office hours: 11:00 AM 2:00 PM @Room 103 Everyday Geek out, share ideas with Enterprise Security developers Red Team / Blue Team - Challenge your skills and learn new tricks Mon- Wed: 3:00 PM 6:00 PM @Splunk Community Lounge Thurs: 11:00 AM 2:00 PM Birds of a feather- Collaborate and brainstorm with security ninjas Thurs: 12:00 PM 1:00 PM @Meal Room 26
THANK YOU