From the Datacenter to the Dean s office

Transcription

1 Copyright 2013 Splunk Inc. From the Datacenter to the Dean s office Mark Runals Sr Security Engineer, The Ohio State University #splunkconf

2 About Me! Started at OSU July 12 = 14 months using Splunk! Splunk training =.conf advanced admin class! ArcSight admin for 3 years medium sized deployment! MoRo - Solve for 80% and move on 2

3 OSU Splunk Environment June 12! 1 beefy server! 350GB license! ~100GB per day Aug 13! Enterprise level centralized log management service! 1,000 Splunk agents deployed, 300+ sourcetypes! 20 indexers! 16 core, 24GB RAM, 10k SAS drives RAID 10! 1TB per day of 2TB license! ~100 people through home- grown training! Mostly covered Office of the CIO (OCIO) Security Logs! Security logs from ~45 colleges 3

4 Agenda! OSU Environment! Program Drivers! Splunk Admin Nerdy Stuff! FTE Requirements! Geeng Funding! Cost Share and Centralizafon! ROI 4

5 OSU Environment! Large: 63,000 students, 32,000 FTE, 14 colleges, 174 undergraduate majors, 12,000 courses.! Highly distributed: 100 IT groups, 30 CIOs, 7+ campuses, 891 buildings.! Complicated: teaching, research, business affiliates, teaching hospital... subject to HIPAA, FERPA, PCI, FISMA, GLB, etc.! Diverse: you name the technology, we probably have it. Many OS plalorms, somware packages, versions, network gear, security gear, etc. 4,000+ web servers, systems. Mulfple acfve directory domains and other authenfcafon sources. Desire2Learn, PeopleSom, and lots of home- grown applicafons. 5

6 Project Scope Change 2 Months into Job Maturafon Complex event correlafon Original plan (with SIEM) More mature with fewer clients (lots of work) Basic monitoring Log review & triage Implemented plan Significantly more clients, quicker, to address audit concerns Log collecfon Rollout & Adopfon 6

7 Program Drivers! Internal audit Mulfple colleges not managing and/or monitoring logs Huge driver for tool selecfon, deployment, adopfon! RFP key strategy decision point Start with log management or go right to advanced correlafon (SIEM)! Powell s Axiom: There are 3 desired business outcomes Minimize risk Reduce cost Generate revenue Goal Achieved (in part) via offering this 7

8 General User Perspecfve No write access in Search app Not able to schedule searches Not able to run real fme searches Each group has individual app InfoSec Web Portal 2FA SSO x 250! Based on locally created template! Contains standard audit content! Place for them to share among team Separate SH for! Security group 1 Search head Addifonal one on deck! Deployment server! Job server! Operafonal dashboards (future) 8

9 Distributed vs. Centralized! Opfon 1 Indexer/search head at each college or department Concerns over lack of control Higher overall admin TCO Distributed hardware funding! Op#on 2 Centralized hardware & provide access to web console Reduced barrier to entry/onboarding Central management of system More secure Setup beefy syslog server to receive syslog data 9

10 Index Creafon Strategy Colleges! 1 5 admins for enfre technology stack! Primary focus audit compliance! Large variety of log sources! Easy RBAC! Office of the CIO! Service organizafon! Dedicated teams at various fers! RBAC about to become a PITA Syslog IIS Firewall x Servers Apache IDS Firewall y Servers Middleware DC Firewalls Server Management Basketweaving Psychobotany Xenopsychology 10

11 Updafng Configs! Do you have anything in- house? Chef, Puppet, Other?! Our challenges Each college IT shop is autonomous Nothing is standard No centralized asset management Most IT shops have < 20 servers Splunk deployment server 11

12 Deployment Server! Used deployment server to manage inputs Use the clientname feature of deploymentclient.conf ê Standardize naming convenfon for forwarder type, group, & OS from start!! ê No CLI support for feature (yet) ê We didn t provide a scripted install solufon to the colleges! Other fps One DS can manage ~2k check- ins a minute Change default phonehome interval via deployment package ê great for troubleshoofng Use DS to manage index.conf files on idx/sh Put tech X props/transforms in same package; deploy to both idx/sh 12

13 FTE Requirements Centrally hosted service work items! New client interacfon! Onboard new data! Data management! Knowledge management! Deploying apps! Training! Content creafon! Tesfng! Tuning Splunk! Customer interacfon! Deployment management! Polifcs! Data requests! General program management issues! Planning! Services support! Fixing stuff! General BS 3 FTE 2 FTE Care and feeding 1 FTE Program & service management Content creafon OSU current = ~1 person working 1.5 FTE+ OSU opfmal = 3 FTE 13

14 General Program Funding! CISO pushed back on accepfng risk Understand the appefte for security/response > ask for appropriate resources Our metrics are probably the same as yours > tell stories on what is happening If you are given money 1) spend it 2) on the things you requested funding for! Our focus was on informafon security In hindsight bringing more folks to the table could have led to more money ê Who doesn t need analyfc capabilifes? ê How complex do you want your RFP to be? 14

15 What About Cost Share?! Ini#al cost share solu#on to colleges Program side salaries funded centrally as our porfon Esfmate use and buy Splunk license capacity upfront Buy hardware to support license capacity eg 1 idx per 100GB Take average/peak use per month for first 3 months ê metrics.log query to determine usage (next slide) Cost per GB x Usage! Pay to play lessons learned Typical group has NO idea volume of log producfon Groups appreciated being able to kick the fres no charge for first 3 months Lower adopfon rates (use not mandated) Metrics log direcfonally correct; not exactly accurate 15

16 Usage Query! Created in first week of using Splunk and haven t revisited it /shrug index=_internal source=*metrics.log group=per_index_thruput chart sum(ev) AS Total_Logs eval(sum(ev)/86400) AS Avg_EPS eval(sum(kb) / ) as GB by series eval Total_Logs=tostring(Total_Logs,"commas") eval Avg_EPS=round(Avg_EPS) eval GB=round(GB,1)!! If you come up with a berer query let me know! 16

17 Cost Share Resolved(ish)! University provided central funding for a number of security efforts Not charging colleges to onboard security logs ê Loose definifon - logs used in typical invesfgafon ê Firewall, OS, AV, web, authenfcafon, etc. (focus on IP, MAC, username)! Outcome Major upfck in adopfon (surprised?) use sfll not mandated Kicked the can what about when the colleges realize Splunk s potenfal and want to send non security logs? 17

18 ROI! Providing university free solufon to meet audit requirements! Colleges don t have to resource in- house solufon! All teams centralized log collecfon; powerful search capability! Database group user account audifng on 140 db! Networking group capacity planning for wireless environment! Security group broader view; richer invesfgafons; quicker resolufon; powerful alert engine! Using Exchange, AD, and VMware Splunk apps 18

19 Takeaways! Don t undersize your server hardware especially inifal install! As you are able / have resources try to think how decisions today will provide or restrict opfons 6 months from now! If you have limited Splunk admin personnel you need engineers over analysts! Limited FTE + data diversity + aggressive rollout = your Splunk admin(s) will NOT be experts on finding/understanding/using the data in Splunk 19

20 Quesfons? 20

21 Contact Info runals.blogspot.com 21

22 Next Steps Download the.conf2013 Mobile App If not iphone, ipad or Android, use the Web App Take the survey & WIN A PASS FOR.CONF2014 Or one of these bags! Go to Planning and Execu#on for Successful Deployments Room: Brera 2&3, Level 3 Today, 10:15-11:15am 22

23 THANK YOU