Splunk Search Pro Tips

Transcription

1 Splunk Search Pro Tips Dan Aiello, Principal Cyber Security Engineer Splunk.conf2015 Dan Aiello Principal Cyber Security Engineer, MITRE Approved for Public Release; Distribution Unlimited

2 Agenda 2 My background Comments Search by index In the year 2000 Red Card Watch Lists Search Job Inspector More fun with subsearches Carrier signal Imaginary events Summary 2

3 My Splunk background 3 4 years Splunk experience SOC is primary user base 6 indexers 350 GB data/day 90 indexes 170 sourcetypes

4 4 comments // No comment

5 5 Comment your Splunk search sourcetype=access_combined_wcookie eval COMMENT="This is my comment" or sourcetype=access_combined_wcookie rename COMMENT - > "This is my comment" * There s nothing special about the word COMMENT, use whatever you like

6 rename vs. eval for comments? 6 In practice, it does not seem to matter In a few odd circumstances, I have seen rename be faster than eval

7 Why would you need comments? 7

8 8 Search by index

9 Searching by index and sourcetype 9 Specifying an index in your search speeds it up This difference is less pronounced in Fast Mode

10 10 In the year 2000 In the year 3000

11 Get with the times 11 Timestamps are extremely important for Splunk data Detected at index time, set forever Cannot be fixed if they re wrong Common errors: Incorrect time zone interpretation Host clock incorrect

12 Past, present, future 12 If this search ever returns events, you have timestamp problems 1 : index=* earliest=+30m latest=+9y This requires some tweaking, depending on your expected delay: index=* eval delta=_indextime- _time where delta>300 1 Or a flux capacitor 2 2 Or a TARDIS

13 13 red card your approximate wait time is

14 Calculate average delay proxy logs 14 index=main eval delta = _indextime - _time timechart span=1h avg(delta) Problem: that s a lot of events

15 15

16 16 Calculate average delay proxy logs Solution: */5 * * * * wget index=main testdomain.zzz eval delta = _indextime - _time timechart span=1h avg(delta) Search terms index=main index=main testdomain.zzz Duration 453 s 6 s

17 17 watchlists better than grep -f

18 Watchlist examples 18 Known evil IP addresses Known evil domain names List of your DMZ web servers Known allowed IP/port combinations in your DMZ

19 Example IP watchlist 19

20 20 Conventional way to use watchlists This is essentially grep -F

21 21 Let s try a subsearch

22 22 What s the difference? Both return the same events

23 23 What s the difference? 71% less time

24 Saving time on a search can be important for large or frequent searches 24 Small watchlists and large datasets make this difference greater

25 How does it work? 25 Just the subsearch

26 26 Subsearches implicitly end with format Add it explicitly to see what s happening

27 sourcetype=access_combined_wcookie [ inputlookup ip_watchlist.csv search type=malicious fields clientip ] 27 after the subsearch is evaluated becomes this: sourcetype=access_combined_wcookie ( ( clientip=" " ) OR ( clientip=" " ) OR ) i.e., the results of the subsearch are appended

28 28 The Search Job Inspector shows us this.

29 29 Why is sourcetype=access_combined_wcookie ( ( clientip=" " ) OR ( clientip=" " ) OR ) Better than sourcetype=access_combined_wcookie lookup ip_watchlist.csv clientip search type=malicious

30 30 Search Job Inspector explains it all

31 This icon means there s some debugging message you should examine 31 Inspect Job is always here

32 32

33 33 Profiling information Debugging message

34 34 The slowest parts of a Splunk search are usually field extraction and reading events from disk.

35 Approximate order of operations for searches Search index for keywords 2. Read matching events from disk 3. Extract fields (as necessary) 4. Match keywords to fields (as necessary) 5. Filter (e.g. additional where or search pipes) 6. Send data to search head

36 What are keywords? 36

37 37 A stitch in time saves nine lookup subsearch 2 21 Check index for keywords 39,000 2,700 Read matching events from disk 39,000 2,700 Extract fields (i.e. regex) 39,000 2,700 Match keywords to fields 39,000 2,700 Filter * This is illustrative and approximate, not precise Pare your data early to save time late

38 lookup method reads and regexes all this data 38 sourcetype=access_ combined_wcookie ( ( clientip=" " ) OR ( clientip=" " ) OR ) subsearch method reads only this data

39 Compare lookup and subsearch methods 39

40 40 More fun with subsearches

41 Field name mismatch with subsearch 41 For lookup and subsearch, sometimes fields need to be renamed lookup method lookup watchlist.csv foo AS bar subsearch method [ inputlookup watchlist.csv rename foo AS bar ]

42 42 If you rename a field to query, you can search anywhere in the event rather than a single field

43 Large subsearches 43 If your watchlist is >10000 lines, the subsearch method chokes 43

44 Large subsearches 44 Add format explicitly to fix it! Warning is gone We have events 44

45 45 carrier signal

46 What s the problem with watchlists? 46 When they don t alert, is it because: the watchlist is broken? there s nothing to alert on?

47 Add test cases to your watchlist 47 Label your test domain as type=test in the watchlist And adjust your subsearch accordingly

48 Add test cases to your watchlist 48 Label your test domain as type=test in the watchlist What s so great about that? We could have just used google.com for that test And adjust your subsearch accordingly

49 49 With your wget, you know precisely how many to expect and you can alert only when it s erroneous

50 50 imaginary events we landed on the moon

51 51 Creating data on the fly

52 How is this helpful? 52 You can add to a watchlist inline

53 Creating data on the fly with timestamps 53 This will also provide a current timestamp 53

54 54 Are we there yet?

55 Overall lessons 55 Read Splunk Search Manual Try multiple methods Use the Job Inspector Understand what Splunk is doing under the hood

56 56 Thank you!