Data Analysis & Visualization for Security Professionals

Transcription

1 Data Analysis & Visualization for Security Professionals Jay Jacobs Verizon Bob Rudis Liberty Mutual Insurance Session ID: GRC- T18 Session Classification: Intermediate

2 Key Learning Points

3 Key Learning Points data helps our understanding of our environment

4 Key Learning Points data helps our understanding of our environment solutions are more from thinking than buying

5 Key Learning Points data helps our understanding of our environment solutions are more from thinking than buying visualizations help communicate complexity quickly

6 Key Learning Points data helps our understanding of our environment solutions are more from thinking than buying visualizations help communicate complexity quickly data visualization is not a natural skill, it must be learned

7 Key Learning Points data helps our understanding of our environment solutions are more from thinking than buying visualizations help communicate complexity quickly data visualization is not a natural skill, it must be learned be truthful: message should match the data

8 Key Learning Points data helps our understanding of our environment solutions are more from thinking than buying visualizations help communicate complexity quickly data visualization is not a natural skill, it must be learned be truthful: message should match the data simple tools can be, data scientist you need not be

9 Make decisions Visual representation of data Visualizing for analysis Helps Understanding Thinking vs buying Ocular biology Amplify cognition Gestalt Quick comprehension Visualize *to* communicate Communicates Complexity Not a natural skill Visual encoding Medium Labeling 3D EVIL Pies R Speaks Truth Uses simple tools Simple yet powerful Python Gephi Tableau Command line prototypes MongoDB Lots of data

11 use information to better understand our world and make more informed decisions Stephen Few

12 use information to better understand our world and make more informed decisions Data helps our understanding of our environment Stephen Few

13 Our Goal: To amplify cognition of data through visual representation and presentation.

14 Our Goal:

15 Our Goal:

16 Visualizing for Analysis

17 Visualizing for Analysis All four data sets: Mean of x: 9.0 Variance of x: 11.0 Mean of y: 7.5 Variance of y: 4.1 Correlation x,y: Linear Regression: y = 3 + 5x

18 Visualized...

19

20

21 Solutions are more from thinking than buying

22 Visualizing for Analysis: Pairs of Threat Actions

23

24 Visualizing to Communicate: The night before Hurricane Sandy... Date: OCT 2012 Hurricane-2 SANDY ADV LAT LON TIME WIND PR STAT /22/15Z TROPICAL DEPRESSION 1A /22/18Z TROPICAL DEPRESSION /22/21Z TROPICAL STORM 2A /23/00Z TROPICAL STORM /23/03Z TROPICAL STORM 3A /23/06Z TROPICAL STORM /23/09Z TROPICAL STORM 4A /23/12Z TROPICAL STORM /23/15Z TROPICAL STORM 5A /23/18Z TROPICAL STORM /23/21Z TROPICAL STORM 6A /24/00Z TROPICAL STORM /24/03Z TROPICAL STORM 7A /24/06Z TROPICAL STORM /24/09Z TROPICAL STORM 8A /24/12Z TROPICAL STORM /24/15Z HURRICANE-1 9A /24/18Z HURRICANE /24/21Z HURRICANE-1 10A /25/00Z HURRICANE /25/03Z HURRICANE-1 11A /25/06Z HURRICANE

25 Visualizing to Communicate: The night before Hurricane Sandy...

26 ...and three months later

27

28 Visualizations help communicate complexity quickly

29 [Tables and graphs] are so common many of us assume that knowledge of their effective use is common as well. I assure you, it is not. Stephen Few Show Me the Numbers: Designing Tables and Graphs to Enlighten

30 count California Attorney GeneralDatabreaches.net Dataloss DB HHS via Databreaches.net HHS via PHIPrivacy.net Media NAID PHIPrivacy.net Security Breach Letter Type CARD DISC HACK INSD PHYS PORT STAT UNKN Month

32 Visualizing Encoding with shape, size, color and position using categorical or quantitative variables possibly over space or time

33 Accuracy of Decoding More Position along a common scale Position on identical but nonaligned scales Length Angle / Slope Area Less Volume / Density / Saturation Hue Graphical perceptions and Graphical Methods for Analyzing Scientific Data, Cleveland and McGill, Science, Photo by Kevin Riggins,

34 Communicate Quantity with Saturation?

35 Quantity Position Length Angle Slope Area Volume Density Saturation Hue Category Position Hue Density Saturation Shape Length Angle Slope Area Volume From: Photo by Kevin Riggins,

36

37 Colorblind Safe Print friendly Photocopy-able

38 Color blindness is common

39 Print Friendly...?

40 Pop Quiz: What do you see?

41

42 Quantity (% of breaches) Category (year) Category (actor)

43 Quantity (% of breaches) Category (position) Category (color)

44 x 74 6x 94 20x length

45 x 74 6x 94 20x Position on common scale

46 x 74 6x 94 20x patterns!

47 Pop Quiz #2: Which is larger?

48 F A E B D C

49 Which is Larger? B A E F C D 0% 3% 5% 8% 10% 13% 15% 18% 20% Position and Length makes the same values easier to compare

50

51 Caution! Adding a third dimension on twodimensional medium creates perspective...

52

53 Pie Charts

54 How are we as an Industry? It seems y all need to go on a diet (too much pie) over 20 industry reports pulled -

55 How are we as an Industry? Data visualization is not a natural skill; It must be learned It seems y all need to go on a diet (too much pie) over 20 industry reports pulled -

56 Avoid them, people don t decode well Use them, people learn how to decode If you must use Pie Charts... Never in 3D Limit categories, 3 to 6 Start at 12, clockwise decreasing in quantity Avoid if angles are small or values are close

57 Slide Workload Distribution Jay Bob

58 Tufte Takeaways Chart Junk: the stuff that doesn t change when the data changes Data Ink Ratio: what percentage of your ink shows data Smallest Effective Difference: the least you can do to highlight

60

61 35% 39.6%

62

63 Selection Bias? [1st, 10th, 16th, and 31st month]

64 Selection Bias? Be truthful! The message should match the data. [1st, 10th, 16th, and 31st month]

66 :52:52 Local4.Info :Apr 13 08:52:52 PDT: %ASA- session : Built inbound TCP connection for W Workstations: / :52:52 Local4.Info ( /4873) to Servers: /135 :Apr 13 08:52:52 PDT: %ASA- session : ( /135) Built inbound TCP connection for W :52:52 Local4.Info :Apr 13 08:52:52 PDT: %ASA- session : %ASA- session : Built Teardown inbound TCP connection TCP connection for Workst for Workstations: / :52:52 Local4.Info ( /4874) to Servers: /43025 :Apr 13 08:52:52 PDT: %ASA- session : ( /43025) Built inbound TCP connection for W :52:52 Local4.Info :Apr 13 08:52:52 PDT: %ASA- session : Built inbound TCP connection for W Workstations: / :52:53 Local4.Info ( /4875) to Servers: /43032 :Apr 13 08:52:53 PDT: %ASA- session : ( /43032) Teardown TCP connection for Workst :52:52 08:52:53 Local4.Info :Apr 13 08:52:52 08:52:53 PDT: %ASA- session : Teardown TCP connection for Workst / :52:53 to Local4.Info Servers: / duration 0:00:00 :Apr 13 bytes 08:52: PDT: TCP %ASA- session : FINs Teardown TCP connection for Workst :52:52 08:52:53 Local4.Info :Apr 13 08:52:52 08:52:53 PDT: %ASA- session : %ASA- session : Built Teardown inbound TCP connection TCP connection for Workst for Workstations: / :52:53 Local4.Info ( /4876) to Servers: /135 :Apr 13 08:52:53 PDT: %ASA- session : ( /135) Teardown TCP connection for Workst :52:52 08:52:53 Local4.Info :Apr 13 08:52:52 08:52:53 PDT: %ASA- session : %ASA- session : Built Teardown inbound TCP connection TCP connection for Workst for Workstations: / :52:55 Local4.Info ( /4877) to Servers: /43025 :Apr 13 08:52:55 PDT: %ASA- session : ( /43025) Teardown TCP connection for Workst :52:53 08:52:55 Local4.Info :Apr 13 08:52:53 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:55 to Local4.Info Servers: /49155 duration :Apr 1:00: :52:55 bytes 1968 PDT: Connection %ASA- session : timeout Teardown TCP connection for Workst :52:53 08:52:55 Local4.Info :Apr 13 08:52:53 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:55 to Local4.Info Servers: / duration 1:00:01 :Apr 13 bytes 08:52: PDT: Connection %ASA- session : timeout Teardown TCP connection for Workst :52:53 08:52:55 Local4.Info :Apr 13 08:52:53 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:55 to Local4.Info Servers: / Firewall duration 0:00:58 :Apr 13 bytes 08:52: PDT: TCP %ASA- session : FINs Logs Built inbound TCP connection for W :52:53 08:52:55 Local4.Info :Apr 13 08:52:53 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:55 to Local4.Info Servers: / duration 0:00:14 :Apr bytes 13 08:52: TCP PDT: FINs %ASA- session : Teardown TCP connection for Workst :52:53 08:52:55 Local4.Info :Apr 13 08:52:53 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:56 to Local4.Info Servers: / duration 0:00:14 :Apr bytes 13 08:52: TCP PDT: FINs %ASA- session : Teardown TCP connection for Workst :52:53 08:52:56 Local4.Info :Apr 13 08:52:53 08:52:56 PDT: %ASA- session : Teardown TCP connection for Workst / :52:56 to Local4.Info Are Servers: / A duration Good 0:00:14 :Apr 13 bytes 08:52: PDT: TCP FINs %ASA- session : Example Teardown TCP connection for Workst :52:55 08:52:52 Local4.Info :Apr 13 08:52:55 08:52:52 PDT: %ASA- session : %ASA- session : Teardown Built inbound TCP connection TCP connection for Workst for W / :52:52 to Local4.Info Servers: / duration 0:00:28 :Apr bytes 13 08:52: TCP PDT: FINs %ASA- session : Built inbound TCP connection for W :52:55 08:52:52 Local4.Info :Apr 13 08:52:55 08:52:52 PDT: %ASA- session : %ASA- session : Teardown Built inbound TCP connection TCP connection for Workst for W / :52:52 to Local4.Info Servers: / duration 0:00:28 :Apr 13 bytes 08:52: PDT: TCP FINs %ASA- session : Teardown TCP connection for Workst :52:55 08:52:52 Local4.Info :Apr 13 08:52:55 08:52:52 PDT: %ASA- session : %ASA- session : Teardown Built inbound TCP connection TCP connection for Workst for W / :52:52 to Local4.Info Servers: / (Use duration 0:00:28 :Apr bytes 13 case 08:52: TCP PDT: FINs %ASA- session : #1) Built inbound TCP connection for W :52:55 08:52:53 Local4.Info :Apr 13 08:52:55 08:52:53 PDT: %ASA- session : Teardown TCP connection for Workst / :52:53 to Local4.Info Servers: / duration 0:00:28 :Apr 13 bytes 08:52: PDT: TCP FINs %ASA- session : Teardown TCP connection for Workst :52:55 08:52:53 Local4.Info :Apr 13 08:52:55 08:52:53 PDT: %ASA- session : Teardown TCP connection for Workst / :52:53 to Local4.Info Servers: / duration 0:00:11 :Apr bytes 13 08:52: TCP PDT: FINs %ASA- session : Teardown TCP connection for Workst :52:55 08:52:53 Local4.Info :Apr 13 08:52:55 08:52:53 PDT: %ASA- session : Teardown TCP connection for Workst / :52:53 to Local4.Info Servers: / duration 0:00:10 :Apr 13 bytes 08:52: PDT: TCP FINs %ASA- session : Teardown TCP connection for Workst :52:55 Local4.Info :Apr 13 08:52:55 PDT: %ASA- session : %ASA- session : Built Teardown inbound TCP connection TCP connection for Workst for Workstations: / :52:55 Local4.Info ( /1440) to Servers: /43032 :Apr 13 08:52:55 PDT: %ASA- session : ( /43032) Teardown TCP connection for Workst :52:55 Local4.Info :Apr 13 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:55 to Local4.Info Servers: / duration 0:00:00 :Apr 13 bytes 08:52: PDT: TCP FINs %ASA- session : Teardown TCP connection for Workst :52:55 Local4.Info :Apr 13 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:55 to Local4.Info Servers: / duration 1:00:01 :Apr 13 bytes 08:52: PDT: Connection %ASA- session : timeout Teardown TCP connection for Workst :52:55 Local4.Info :Apr 13 08:52:55 PDT: %ASA- session : %ASA- session : Teardown Built inbound TCP connection TCP connection for Workst for W / :52:55 to Local4.Info Servers: /49155 duration :Apr 1:00: :52:55 bytes 1941 PDT: Connection %ASA- session : timeout Teardown TCP connection for Workst :52:56 08:52:55 Local4.Info :Apr 13 08:52:56 08:52:55 PDT: %ASA- session : Teardown TCP connection for Workst / :52:55 to Local4.Info Servers: / duration 0:00:28 :Apr bytes 13 08:52: TCP PDT: FINs %ASA- session : Teardown TCP connection for Workst :52:56 Local4.Info :Apr 13 08:52:56 PDT: %ASA- session : Teardown TCP connection for Workst / :52:56 to Local4.Info Servers: / duration 0:00:28 :Apr 13 bytes 08:52: PDT: TCP FINs %ASA- session : Teardown TCP connection for Workst :52:56 Local4.Info :Apr 13 08:52:56 PDT: %ASA- session : Teardown TCP connection for Workst

67 Source: :52:52 Local4.Info :Apr 13 08:52:52 PDT: %ASA-session : Built inbound TCP connection for Workstations: /4873 ( /4873) to Servers: /135 ( /135) Normalized: Date/time,Syslog priority,operation,message code,protocol,source IP,Destination IP,Source hostname,destination hostname,source port,destination port,destination service,direction,connections built,connections torn down 13/Apr/ :52:52,Info,Built,ASA-session ,TCP, , ,(empty),(empty), 4873,135,epmap,inbound,1,0

68 $ grepfield - p Built _ _fw_log* aggregate - p - k 6 - c 6 - d \, sort - n - t, - k2 tail , , , ,10753 (empty), , , , , ,

69 10,452,115 events 1.3GB of data 4.5 hours (not even one day) 1 firewall

70 Command-line tools aren t enough

71

72 Simple, tools can be; Data scientist, you need not be.

73 [ { "Syslog priority": "Info", "Protocol": "TCP", "Destination IP": " ", "Destination port": "135", "Source IP": " ", "Connections torn down": "0", "Direction": "inbound", "Connections built": "1", "Message code": "ASA-session ", "Date/time": "13/Apr/ :52:52", "Destination service": "epmap", "Source port": "4873", "Destination hostname": "(empty)", "Source hostname": "(empty)", "Operation": "Built" }, { "Syslog priority": "Info", "Protocol": "TCP", "Destination IP": " ", "Destination port": "43025", "Source IP": " ", "Connections torn down": "0", "Direction": "inbound", "Connections built": "1", "Message code": "ASA-session ", "Date/time": "13/Apr/ :52:52", "Destination service": "43025_tcp", "Source port": "4874", "Destination hostname": "(empty)", "Source hostname": "(empty)", "Operation": "Built" } ]

74 [ { "Syslog priority": "Info", "Protocol": "TCP", "Destination IP": " ", "Destination port": "135", "Source IP": " ", "Connections torn down": "0", "Direction": "inbound", "Connections built": "1", "Message code": "ASA-session ", "Date/time": "13/Apr/ :52:52", "Destination service": "epmap", "Source port": "4873", "Destination hostname": "(empty)", "Source hostname": "(empty)", "Operation": "Built" }, { "Syslog priority": "Info", "Protocol": "TCP", "Destination IP": " ", "Destination port": "43025", "Source IP": " ", "Connections torn down": "0", "Direction": "inbound", "Connections built": "1", "Message code": "ASA-session ", "Date/time": "13/Apr/ :52:52", "Destination service": "43025_tcp", "Source port": "4874", "Destination hostname": "(empty)", "Source hostname": "(empty)", "Operation": "Built" } ] #!/usr/bin/python import csv import json import sys csv_file = open(sys.argv[1],"r") reader = csv.reader(csv_file) header = reader.next() for row in reader: jsondict = {} for i in range(len(header)): jsondict[header[i]] = row[i] print json.dumps(jsondict)

75 mongoimport db.fw.aggregate( [ { $match : { day : "13" } }, # match the first day { $group : { _id : "$src", count : { $sum : 1 } } }, # group and count source { $project : { _id : 0, src : "$_id", count: "$count" } }, # project into structure { $sort : { count : -1, _id : -1 } }, # sort by counts { $limit : 10 } ] ) # show just top 10 "ok" : 1 { "count" : , "src" : " " }, { "count" : , "src" : " " }, { "count" : , "src" : " " }, { "count" : , "src" : " " }, { "count" : , "src" : " " }, { "count" : 39931, "src" : "(empty)" }, { "count" : 10753, "src" : " " }, { "count" : 3457, "src" : " " }, { "count" : 2752, "src" : " " }, { "count" : 1558, "src" : " " }

76

77

78

79

80 Source Count , , , , , ,518 Analysis Action: Investigate 174/175

81 Packets over Time...

82 Packets over Time...

83 Use Case #2: Geo-location of IP addresses

84 Some botnets are so big you can see them from space (or at least, Google Earth).

85 F-Secure releases 140,000 ZeroAccess geolocations IN,"18.975, " TR," , " US," , " TR," , " VE,"10.5, " US," , " RO," ,24.35" RO," ,24.35" RO," ,26.9" JP,"35.685, " BR," , " IN,"15.15, " CA," , " IT,"44.8, " US," , " CO,"4.6492, " RO,"46.35,25.8" US," , " PL," , " CA," , " SE," ,17.25" US," , " US," , " CA," ,-73.4"

86 Careful How Data is Parsed

87 Google Maps

88 Not Google Maps alpha = 1/33

89 Population The Story Our Data

90 and to wrap things up...

91

92 Key Learning Points data helps our understanding of our environment solutions are more from thinking than buying visualizations help communicate complexity quickly data visualization is not a natural skill, it must be learned be truthful: message should match the data simple tools can be, data scientist you need not be

93 Bob Jay