The Bro Network Security Monitor

Transcription

1 Robin Sommer International Computer Science Institute, & Lawrence Berkeley National Laboratory

2 What is Bro? 2

3 What is Bro? Packet Capture 2

4 What is Bro? Packet Capture Traffic Inspection 2

5 What is Bro? Packet Capture Traffic Inspection Attack Detection 2

6 What is Bro? Packet Capture Traffic Inspection Attack Detection NetFlow syslog Log Recording 2

7 What is Bro? Packet Capture Traffic Inspection Attack Detection NetFlow syslog Log Recording Flexibility Abstraction Data Structures 2

8 What is Bro? Packet Capture Traffic Inspection Attack Detection NetFlow syslog Log Recording Flexibility Abstraction Data Structures 2

9 What is Bro? Packet Capture Traffic Inspection Attack Detection NetFlow syslog Log Recording Flexibility Abstraction Data Structures 2

10 What is Bro? Packet Capture Traffic Inspection Attack Detection Domain-specific Python NetFlow syslog Log Recording Flexibility Abstraction Data Structures 2

11 Philosophy Fundamentally different from other IDS. Need to reset your idea of an IDS before starting to use Bro. Real-time network analysis framework. Primarily an IDS, but many use it for general traffic analysis. Can accommodate a range of detection approaches. Policy-neutral at the core. Highly stateful. Tracks extensive application-layer network state. Supports forensics. Extensively logs what it sees. 3

12 Bro History Vern writes 1st line of code 4

13 Bro History Vern writes 1st line of code LBNL starts using Bro operationally v0.2 1st CHANGES entry v0.4 HTTP analysis Scan detector IP fragments Linux support v0.6 RegExps Login analysis v0.7a90 Profiling State Mgmt v0.7a175/0.8ax Signatures SMTP IPv6 support User manual v0.8ax/0.9ax SSL/SMB STABLE releases BroLite v1.1/v1.2 when Stmt Resource tuning Broccoli DPD v1.0 BinPAC IRC/RPC analyzers 64-bit support Sane version numbers v1.5 BroControl v1.4 DHCP/BitTorrent HTTP entities NetFlow Bro Lite Deprecated Bro SDCI Bro 2.0 New Scripts Bro 2.1 IPv6 Input Framework v0.7a48 Consistent CHANGES 0.8a37 Communication Persistence Namespaces Log Rotation v1.3 Ctor expressions GeoIP Conn Compressor 4

14 Bro History Host Context Time Machine Enterprise Traffic TRW State Mgmt. Independ. State Bro Cluster Shunt Academic Publications USENIX Paper Stepping Stone Detector Anonymizer Active Mapping Context Signat. BinPAC DPD 2nd Path Parallel Prototype Autotuning Input Framework Vern writes 1st line of code LBNL starts using Bro operationally v0.2 1st CHANGES entry v0.4 HTTP analysis Scan detector IP fragments Linux support v0.6 RegExps Login analysis v0.7a90 Profiling State Mgmt v0.7a175/0.8ax Signatures SMTP IPv6 support User manual v0.8ax/0.9ax SSL/SMB STABLE releases BroLite v1.1/v1.2 when Stmt Resource tuning Broccoli DPD v1.0 BinPAC IRC/RPC analyzers 64-bit support Sane version numbers v1.5 BroControl v1.4 DHCP/BitTorrent HTTP entities NetFlow Bro Lite Deprecated Bro SDCI Bro 2.0 New Scripts Bro 2.1 IPv6 Input Framework v0.7a48 Consistent CHANGES 0.8a37 Communication Persistence Namespaces Log Rotation v1.3 Ctor expressions GeoIP Conn Compressor 4

15 Who s Using It? Installations across the US Universities Research Labs Supercomputer Centers Industry Examples Lawrence Berkeley National Lab Indiana University National Center for Supercomputing Applications National Center for Atmospheric Research... and many more sites Fully integrated into Security Onion Popular security-oriented Linux distribution Recent User Meetings Bro Workshop 2011 at NCSA Bro Exchange 2012 at NCAR Each attended by about 50 operators from from organizations 5

16 Deployment Internet Internal Network 6

17 Deployment Internet Tap Internal Network Bro 6

18 Deployment Internet Tap Internal Network Bro Runs on commodity platforms.! Standard PCs & NICs. Supports FreeBSD/Linux/OS X. 6

19 Example Logs 7

20 Example Logs > bro -i en0 [... wait...] > cat conn.log 7

21 Example Logs > bro -i en0 [... wait...] > cat conn.log #fields ts id.orig_h id.orig_p id.resp_h id.resp_p proto service duration tcp http tcp http tcp http tcp http tcp http tcp http tcp http

22 Example Logs > bro -i en0 [... wait...] > cat conn.log #fields ts id.orig_h id.orig_p id.resp_h id.resp_p proto service duration tcp http tcp http tcp http tcp http tcp http tcp http tcp http > cat http.log 7

23 Example Logs > bro -i en0 [... wait...] > cat conn.log #fields ts id.orig_h id.orig_p id.resp_h id.resp_p proto service duration tcp http tcp http tcp http tcp http tcp http tcp http tcp http > cat http.log #fields ts id.orig_h id.orig_p [...] host uri status_code user_agent [...] docs.python.org /lib/lib.css 200 Mozilla/ docs.python.org /icons/previous.png 304 Mozilla/ docs.python.org /lib/lib.html 200 Mozilla/ docs.python.org /icons/up.png 304 Mozilla/ docs.python.org /icons/next.png 304 Mozilla/ docs.python.org /icons/contents.png 304 Mozilla/ docs.python.org /icons/modules.png 304 Mozilla/ docs.python.org /icons/index.png 304 Mozilla/ / 200 Mozilla/5.0 7

24 Example Logs > bro -i en0 [... wait...] > cat conn.log #fields ts id.orig_h id.orig_p id.resp_h id.resp_p proto service duration tcp http tcp http tcp http [...] host uri status_code 80 tcp user_agent http [...] docs.python.org /lib/lib.css tcp Mozilla/5.0 http tcp http docs.python.org /icons/previous.png tcp Mozilla/5.0 http docs.python.org /lib/lib.html 200 Mozilla/5.0 > cat docs.python.org http.log /icons/up.png 304 Mozilla/5.0 docs.python.org /icons/next.png 304 Mozilla/5.0 #fields ts id.orig_h id.orig_p [...] host uri status_code user_agent [...] docs.python.org /icons/contents.png docs.python.org /lib/lib.css Mozilla/5.0 Mozilla/ docs.python.org /icons/previous.png 304 docs.python.org /icons/modules.png 304 Mozilla/ docs.python.org /lib/lib.html 200 Mozilla/ docs.python.org /icons/index.png docs.python.org /icons/up.png Mozilla/ docs.python.org /icons/next.png 304 Mozilla/ / docs.python.org /icons/contents.png Mozilla/5.0 Mozilla/ docs.python.org /icons/modules.png 304 Mozilla/ docs.python.org /icons/index.png 304 Mozilla/ / 200 Mozilla/5.0 7

25 Identifying HTTP Servers 8

26 Identifying HTTP Servers Server Addresses a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com nuq04s07-in-f27.1e100.net a deploy.akamaitechnologies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com nuq04s06-in-f27.1e100.net upload-lb.pmtpa.wikimedia.org nuq04s08-in-f27.1e100.net 8

27 Identifying HTTP Servers Server Addresses HTTP Host Headers a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com nuq04s07-in-f27.1e100.net a deploy.akamaitechnologies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com a deploy.akamaitechnolgies.com nuq04s06-in-f27.1e100.net upload-lb.pmtpa.wikimedia.org nuq04s08-in-f27.1e100.net ad.doubleclick.net ad.yieldmanager.com b.scorecardresearch.com clients1.google.com googleads.g.doubleclick.net graphics8.nytimes.com l.yimg.com liveupdate.symantecliveupdate.com mt0.google.com pixel.quantserve.com platform.twitter.com profile.ak.fbcdn.net s0.2mdn.net safebrowsing-cache.google.com static.ak.fbcdn.net swcdn.apple.com upload.wikimedia.org

28 File Content 9

29 File Content GET /skins-1.5/common/images/magnify-clip.png image/png GET /skins-1.5/monobook/external.png image/png GET /softw/90/update/avg9infoavi.ctf text/plain GET /softw/90/update/avg9infowin.ctf text/plain GET /softw/90/update/u7avi1777u1705ff.bin application/x-dosexec 0210a9516dd34abc481683f877bd GET /softw/90/update/u7avi1778u1705z7.bin application/x-dosexec 9bd8e3a274d8ada852bc3d bf GET /softw/90/update/u7iavi2511u2510ff.bin application/x-dosexec 5e63f63fd a56dbd89d8688f GET /softw/90/update/u7iavi2512u2511z7.bin application/x-dosexec a8e1ef490967ef7eb6641bef9eed GET /softw/90/update/x8xplsb2_118c8.bin application/x-dosexec e c5550e9fbf33ef15fed75e5a GET /softw/90/update/x8xplsc_149d148c8.bin application/x-dosexec db5b04f3c45da4c0686c678bfd0e241c GET /sports/ text/html - 9

30 Software Logging 10

31 Software Logging Windows-Update-Agent - - Windows-Update-Agent Microsoft-IIS 6 0 Microsoft-IIS/ ASP.NET - - ASP.NET Microsoft-IIS 7 0 Microsoft-IIS/ ASP.NET - - ASP.NET SCSDK 6 0 SCSDK Apache 2 2 Apache/2.2.9 (Debian) PHP/ lenny PHP 5 2 PHP/ lenny Apache 2 2 Apache/2.2.9 (Debian) PHP/ lenny PHP 5 2 PHP/ lenny Apache 2 2 Apache/2.2.9 (Debian) PHP/ lenny PHP 5 2 PHP/ lenny Apache 2 2 Apache/2.2.9 (Debian) PHP/ lenny PHP 5 2 PHP/ lenny Apache 2 2 Apache/2.2.9 (Debian) PHP/ lenny PHP 5 2 PHP/ lenny Apache 2 0 Apache/ (Debian GNU/Linux) PHP 4 3 PHP/ Apache 2 2 Apache/2.2.9 (Debian) PHP/ lenny PHP 5 2 PHP/ lenny3 10

32 SSL Certificate Logging 11

33 SSL Certificate Logging CN=Microsoft Secure Server Authority,DC=redmond,DC=corp,DC=microsoft,DC=com CN=Sun Microsystems Inc SSL CA,OU=Class 3 MPKI Secure Server CA,OU=VeriSign CN=Microsoft Secure Server Authority,DC=redmond,DC=corp,DC=microsoft,DC=com CN=Microsoft Secure Server Authority,DC=redmond,DC=corp,DC=microsoft,DC=com CN=VeriSign Class 3 Extended Validation SSL CA,OU=Terms of use at CN=Akamai Subordinate CA 3,O=Akamai Technologies Inc,C=US CN=Akamai Subordinate CA 3,O=Akamai Technologies Inc,C=US OU=Equifax Secure Certificate Authority,O=Equifax,C=US CN=VeriSign Class 3 Secure Server CA,OU=Terms of use at CN=Akamai Subordinate CA 3,O=Akamai Technologies Inc,C=US CN=Network Solutions Certificate Authority,O=Network Solutions L.L.C.,C=US OU=Equifax Secure Certificate Authority,O=Equifax,C=US OU=Equifax Secure Certificate Authority,O=Equifax,C=US CN=Microsoft Secure Server Authority,DC=redmond,DC=corp,DC=microsoft,DC=com 11

34 Brownian 12

35 Architecture Packets Network 13

36 Architecture Events Protocol Decoding Event Engine Packets Network 13

37 Architecture Logs Notification Analysis Logic Policy Script Interpreter Events Protocol Decoding Event Engine Packets Network 13

38 Architecture Logs Notification User Interface Analysis Logic Policy Script Interpreter Events Protocol Decoding Event Engine Packets Network 13

39 Event Model Web Client Request for /index.html Web Server /4321 Status OK plus data /80 14

40 Event Model Web Client Request for /index.html Web Server /4321 Status OK plus data /80... Stream of TCP packets... SYN SYN ACK ACK ACK ACK FIN FIN 14

41 Event Model Web Client Request for /index.html Web Server /4321 Status OK plus data /80... Stream of TCP packets... SYN SYN ACK ACK ACK ACK FIN FIN Event connection_established( / /80) 14

42 Event Model Web Client Request for /index.html Web Server /4321 Status OK plus data /80... Stream of TCP packets... SYN SYN ACK ACK ACK ACK FIN FIN Event connection_established( / /80) TCP stream reassembly for originator Event http_request( / /80, GET, /index.html ) 14

43 Event Model Web Client Request for /index.html Web Server /4321 Status OK plus data /80... Stream of TCP packets... SYN SYN ACK ACK ACK ACK FIN FIN Event connection_established( / /80) TCP stream reassembly for originator Event http_request( / /80, GET, /index.html ) TCP stream reassembly for responder Event http_reply( / /80, 200, OK, data) 14

44 Event Model Web Client Request for /index.html Web Server /4321 Status OK plus data /80... Stream of TCP packets... SYN SYN ACK ACK ACK ACK FIN FIN Event connection_established( / /80) TCP stream reassembly for originator Event http_request( / /80, GET, /index.html ) TCP stream reassembly for responder Event http_reply( / /80, 200, OK, data) Event connection_finished( /4321, /80) 14

45 Script Example: Matching URLs Task: Report all Web requests for files called passwd. 15

46 Script Example: Matching URLs Task: Report all Web requests for files called passwd. event http_request(c: connection, # Connection. method: string, # HTTP method. original_uri: string, # Requested URL. unescaped_uri: string, # Decoded URL. version: string) # HTTP version. { if ( method == "GET" && unescaped_uri == /.*passwd/ ) NOTICE(...); # Alarm. } 15

47 Script Example: Scan Detector Task: Count failed connection attempts per source address. 16

48 Script Example: Scan Detector Task: Count failed connection attempts per source address. global attempts: table[addr] of count &default=0; event connection_rejected(c: connection) { local source = c$id$orig_h; # Get source address. local n = ++attempts[source]; # Increase counter. if ( n == SOME_THRESHOLD ) # Check for threshold. NOTICE(...); # Alarm. } 16

49 Distributed Scripts 17

50 Distributed Scripts Bro comes with >10,000 lines of script code. Prewritten functionality that s just loaded. Scripts generate all the logs. Amendable to extensive customization and extension. 17

51 Bro Ecosystem Internet Tap Internal Network Bro 18

52 Bro Ecosystem Internet Tap Internal Network Bro Control Output BroControl User Interface 18

53 Bro Ecosystem Internet Tap Internal Network Contributed Scripts Functionality Bro Control Output BroControl User Interface 18

54 Bro Ecosystem Internet Tap Internal Network Contributed Scripts Functionality Bro Events State Other Bros Control Output BroControl User Interface 18

55 Bro Ecosystem Internet Tap Internal Network Contributed Scripts Functionality Bro Events State Other Bros Control Output BroControl User Interface Events Bro Client Communication Library Broccoli 18

56 Bro Ecosystem Internet Tap Internal Network Contributed Scripts Functionality Bro Events State Other Bros Control Output BroControl User Interface Events Bro Client Communication Library Broccoli Python Broccoli Broccoli Ruby (Broccoli Perl) 18

57 Bro Ecosystem Time Machine Internet Tap Tap Internal Network Contributed Scripts Functionality Bro Events State Other Bros Control Output BroControl User Interface Events Bro Client Communication Library Broccoli Python Broccoli Broccoli Ruby (Broccoli Perl) 18

58 Bro Ecosystem Time Machine Internet Tap Tap Internal Network Contributed Scripts Functionality Bro Events State Other Bros Control Output bro-aux BinPAC capstats BroControl Events Bro Client Communication Library Broccoli Python BTest tracesummary Broccoli Broccoli Ruby User Interface (Broccoli Perl) 18

59 Bro Ecosystem Time Machine Bro Distribution bro-2.1.tar.gz Internet Tap Tap Internal Network Contributed Scripts Functionality Bro Events State Other Bros Control Output bro-aux BTest BinPAC tracesummary capstats BroControl Events Bro Client Communication Library Broccoli Python Broccoli Broccoli Ruby User Interface (Broccoli Perl) 18

60 Bro Ecosystem Time Machine Bro Distribution bro-2.1.tar.gz Internet Tap Tap Internal Network Contributed Scripts Functionality Bro Events State Other Bros Control Output bro-aux BTest BinPAC tracesummary capstats BroControl Events Bro Client Communication Library Broccoli Python Broccoli Broccoli Ruby User Interface (Broccoli Perl) git://git.bro-ids.org 18

61 Bro Cluster Ecosystem Internet Tap Internal Network Contributed Scripts Functionality Bro Events State External Bro Control Output bro-aux BinPAC capstats BroControl Events Bro Client Communication Library Broccoli Python BTest tracesummary Broccoli Broccoli Ruby User Interface (Broccoli Perl) 19

62 Bro Cluster Ecosystem Internet Tap Internal Network Contributed Scripts Functionality Bro Events State External Bro Control Output bro-aux BinPAC capstats BroControl Events Bro Client Communication Library Broccoli Python BTest tracesummary Broccoli Broccoli Ruby User Interface (Broccoli Perl) 19

63 Bro Cluster Ecosystem Internet Tap Load- Balancer Internal Network Contributed Scripts Functionality Bro Events State External Bro Control Output bro-aux BinPAC capstats BroControl Events Bro Client Communication Library Broccoli Python BTest tracesummary Broccoli Broccoli Ruby User Interface (Broccoli Perl) 19

64 Bro Cluster Ecosystem Internet Tap Internal Network Packets Load- Balancer Contributed Scripts Functionality Bro Bro Bro Bro Bro Events State External Bro Control Output bro-aux BTest BinPAC tracesummary capstats BroControl Events Bro Client Communication Library Broccoli Python Broccoli Broccoli Ruby User Interface (Broccoli Perl) 19

65 Bro Cluster Ecosystem Internet Tap Internal Network Packets Load- Balancer Contributed Scripts Functionality Bro Bro Bro Bro Bro Events State External Bro bro-aux BTest BinPAC tracesummary capstats Control Control Output BroControl Output Events Bro Client Communication Library Broccoli Python Broccoli Broccoli Ruby User Interface (Broccoli Perl) 19

66 Bro Cluster Ecosystem Internet Tap Internal Network Packets Load- Balancer Frontend Contributed Scripts Functionality Bro Bro Bro Bro Bro Workers Events State External Bro bro-aux BTest BinPAC tracesummary capstats Control Control Output Manager BroControl Output Events Bro Client Communication Library Broccoli Python Broccoli Broccoli Ruby User Interface (Broccoli Perl) 19

67 A Production Load-Balancer cflow: 10GE line-rate, stand-alone load-balancer 10 Gb/s in/out Web & CLI Filtering capabilities Available from cpacket 20

68 A Production Load-Balancer cflow: 10GE line-rate, stand-alone load-balancer 10 Gb/s in/out Web & CLI Filtering capabilities Available from cpacket 20

69 Indiana University Indiana University OpenFlow Deployment v.1.0 Bloomington CIC Chicago Chicago Testlab 2 Nodes 10 Gig via 10 Gig via DWDM System via 8 OpenFlow Switches Test Servers 2 Nodes 5 Nodes Indianapolis ICTC Testpoint 2 Nodes IU Production Deployment IU Wireless SSID: OpenFlow Layer 3 router on OpenFlow switches InterOp lab Workshop Lindley Hall Telcom Bldn Informatics East Informatics West Monitoring Indianapolis VM Server 4 OpenFlow switches IU Core Network 6 x 10G 12 x 10G OpenFlow load balancer IDS Cluster 12 servers Source: Indiana University 21

70 Indiana University Indiana University OpenFlow Deployment v.1.0 Bloomington CIC Chicago Chicago Testlab 2 Nodes 10 Gig via 10 Gig via DWDM System via 8 OpenFlow Switches Test Servers 2 Nodes 5 Nodes Indianapolis ICTC Testpoint 2 Nodes IU Production Deployment IU Wireless SSID: OpenFlow Layer 3 router on OpenFlow switches InterOp lab Workshop Lindley Hall Telcom Bldn Informatics East Informatics West Monitoring Indianapolis VM Server 4 OpenFlow switches IU Core Network 6 x 10G 12 x 10G OpenFlow load balancer IDS Cluster 12 servers Source: Indiana University 21

71 External Events: Broccoli 22

72 External Events: Broccoli Auditing SSHD 22

73 External Events: Broccoli Solu5on&Overview& Auditing SSHD PARENT' SSHD' CHILD' SSHD' STUNNEL' SSLOGMUX' BROPIPE' Source: Scott Campbell / NERSC 5& 22

74 NERSC reserves the right to remove any data at any time and/or transfer data to NERSC Computer Use Policies Form other individuals working on the same or similar project once a user account is deleted or a person no longer has a business association with NERSC. Account Usage Users are not allowed to share their accounts with others. Monitoring and Privacy Users have no explicit or implicit expectation of privacy. NERSC retains the right to monitor the content of all activities on NERSC systems and networks and access any computer files without prior knowledge or consent of users, senders or recipients. NERSC may retain copies of any network traffic, computer files or messages indefinitely without prior knowledge or consent. NERSC personnel and users are required to address, safeguard against and report misuse, abuse and criminal activities. Misuse of NERSC resources can lead to temporary or permanent disabling of accounts, loss of DOE allocations, and administrative or legal actions. revision 1.1 date: 2007/October/11 20:06:56 23

75 ons for the Classroom The Security Fence Presented by NIEonline.com and the Association of American Editorial Cartoonists (AAEC) vs. y lines g civil me of as been n the ttacks the d peoe up otecttion ts in ction toon illustrates the problem of Cartoon Courtesy Clay Bennett / The Christian Science Monitor 24

76 Version 2.0 (Jan 2012) 25

77 Version 2.0 (Jan 2012) Default scripts rewritten from scratch. Focus on ease of use and operational deployment. New logging infrastructure. New build and packaging system. New auto-documentation system (Broxygen). Lots of bugs fixed. Obsolete code removed. New development infrastructure. New regression testing framework. New web server. New mailing lists. New logo. 25

78 Just released... 26

79 Just released... Bro 2.1 Comprehensive IPv6 support. Tunnel decapsulation. New logging formats (DataSeries / ElasticSearch) Input Framework 26

80 Input Framework Example: Blacklists IP Reason Timestamp Connected to honeypot Too many DNS requests Sent spam

81 User Interface 28

82 User Interface type Index: record { ip: addr; }; type Value: record { reason: string; timestamp: time; }; global blacklist: table[addr] of Value; Input::add_table(source="blacklist.tsv", idx=index, val=value, destination=blacklist); (Syntax simplified.) 28

83 User Interface type Index: record { ip: addr; }; type Value: record { reason: string; timestamp: time; }; global blacklist: table[addr] of Value; Input::add_table(source="blacklist.tsv", idx=index, val=value, destination=blacklist); (Syntax simplified.) event connection_established(c: connection) { if ( c$id$orig_h in blacklist ) alarm(...) } 28

84 Current Research 29

85 Performace: 100 Gb/s Now these sites need a monitoring solution... Working with cpacket on a 100GE loadbalancer! DOE/ESNet 100G Advanced Networking Initiative Source: ESNet Source: ESNet 30

86 Production Backbone in Planing 31

87 100 Gb/s Load-balancer

88 100 Gb/s Load-balancer 100Gbps cflow 100G 10Gb/s Bro Cluster

89 100 Gb/s Load-balancer 100Gbps cflow 100G API 10Gb/s Control Bro Cluster

90 Concurrent Analysis Logs Notification Analysis Logic Policy Script Interpreter Events Protocol Decoding Event Engine Packets Network 33

91 Concurrent Analysis Logs Notification Single Thread Analysis Logic Policy Script Interpreter Events Protocol Decoding Event Engine Packets Network 33

92 Architecture Notification Detection Logic Scripting Language Script Threads Events Packet Analysis Event Engine Event Engine Threads Packets Dispatcher Packet Dispatcher (NIC) Network 34

93 Architecture Notification Detection Logic Scripting Language Script Threads Events Packet Analysis Event Engine Event Engine Threads Cluster in a Box Packets Dispatcher Packet Dispatcher (NIC) Network 34

94 Architecture How to parallelize a scripting language? Notification Detection Logic Scripting Language Script Threads Events Packet Analysis Cluster in a Box Event Engine Packets Dispatcher Event Engine Threads Packet Dispatcher (NIC) Network 34

95 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue 35

96 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A http_request 35

97 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A http_request http_reply Conn A 35

98 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A http_request http_reply Conn A Conn B http_request 35

99 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A Conn A Conn B Orig X http_request http_reply http_request conn_rejected 35

100 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A Conn A Conn B Orig X Orig Y http_request http_reply http_request conn_rejected conn_rejected 35

101 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A Conn A Conn B Orig X Orig Y Orig X http_request http_reply http_request conn_rejected conn_rejected conn_rejected 35

102 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A Conn A Conn B Orig X Orig Y Orig X Conn B http_request http_reply http_request conn_rejected conn_rejected conn_rejected http_reply 35

103 Parallel Event Scheduling Threaded Script Interpreter Thread 1 Thread 2 Thread 3 Thread 4 Thread n Queue Queue Queue Queue Queue Queue Conn A Conn A Conn B Orig X Orig Y Orig X Conn B Conn A http_request http_reply http_request conn_rejected conn_rejected conn_rejected http_reply http_request 35

104 Improving Bro s Performance Bottlenecks: Single-thread structure & Script interpretation 36

105 Improving Bro s Performance Bottlenecks: Single-thread structure & Script interpretation A High-Level Intermediary Language for Traffic Inspection Host Application HILTI Machine Environment OS Toolchain Application Core C Interface Stubs Analysis Specification Analysis Compiler HILTI Machine Code HILTI Compiler Native Object Code System Linker Native Executable LLVM Runtime Library hiltic hilti-build 36

106 BinPAC: Yacc for Network Protocols 37

107 BinPAC: Yacc for Network Protocols type SMB_header = record { protocol : bytestring &length = 4; command : uint8; status : SMB_error(err_status_type); flags : uint8; flags2 : uint16; pad : padding[12]; tid : uint16; pid : uint16; uid : uint16; mid : uint16; } &let { err_status_type = (flags2 >> 14) & 1; unicode = (flags2 >> 15) & 1; } &byteorder = littleendian; type SMB_error (err_status_type: int) = case err_status_type of { 0 -> dos_error: SMB_dos_error; 1 -> status: int32; }; type SMB_dos_error = record { error_class : uint8; reserved : uint8; error : uint16; }; 37

108 Next-generation BinPAC 38

109 Next-generation BinPAC type Message = unit(body_default: bool) { headers : list<header(self)>; end_of_hdr: /\r?\n/; body : Body([...]) }; HTTP Message 38

110 Next-generation BinPAC type Message = unit(body_default: bool) { headers : list<header(self)>; end_of_hdr: /\r?\n/; body : Body([...]) const HeaderName = /[^:\r\n]+/; const HeaderValue = /[^\r\n]*/; type Header = unit(msg: Message) { name : HeaderName; : /:[\t ]*/; content: HeaderValue; : NewLine; }; HTTP Message }; HTTP Header 38

111 Next-generation BinPAC type Message = unit(body_default: bool) { headers : list<header(self)>; end_of_hdr: NewLine; body : Body(self, self.delivery_mode) if ( self.has_body ); on end_of_hdr { if ( self?.content_length ) self.delivery_mode = DeliveryMode::Length; } if ( self.content_type.startswith("multipart/") ) [... Parse boundary...] [...] var content_length: uint64; var content_type: bytes; var delivery_mode: DeliveryMode; var has_body: bool; var multipart_boundary: bytes; var transfer_encoding: bytes; }; HTTP Message const HeaderName = /[^:\r\n]+/; const HeaderValue = /[^\r\n]*/; type Header = unit(msg: Message) { name : HeaderName &convert=to_lower; : /:[\t ]*/; content: HeaderValue; : NewLine; }; on content { if ( self.name == "content-length" ) { msg.content_length = to_uint(self.content); msg.has_body = True; } if ( self.name == "transfer-encoding" ) { msg.transfer_encoding = self.content; msg.has_body = True; } if ( self.name == "content-type" ) msg.content_type = self.content; HTTP Header 39

112 Next-generation BinPAC type Message = unit(body_default: bool) { headers : list<header(self)>; end_of_hdr: NewLine; body : Body(self, self.delivery_mode) if ( self.has_body ); on end_of_hdr { if ( self?.content_length ) self.delivery_mode = DeliveryMode::Length; } if ( self.content_type.startswith("multipart/") ) [... Parse boundary...] [...] var content_length: uint64; var content_type: bytes; var delivery_mode: DeliveryMode; var has_body: bool; var multipart_boundary: bytes; var transfer_encoding: bytes; }; BinPAC++ Streamlined usage. const HeaderName = /[^:\r\n]+/; const HeaderValue = /[^\r\n]*/; Adding semantics to syntax. type Header = unit(msg: Message) { name : HeaderName &convert=to_lower; : /:[\t ]*/; content: HeaderValue; : NewLine; Decoding layers of protocols. Robust error handling. Fully usable outside of Bro. Compiles to HILTI. }; on content { if ( self.name == "content-length" ) { msg.content_length = to_uint(self.content); msg.has_body = True; } if ( self.name == "transfer-encoding" ) { msg.transfer_encoding = self.content; msg.has_body = True; } if ( self.name == "content-type" ) msg.content_type = self.content; HTTP Message HTTP Header 39

113 Outlook & Conclusion 40

114 More Things in the Bro Queue... 41

115 More Things in the Bro Queue... Comprehensive File Analysis Intelligence Framework Metrics Framework Database interface Packet Filter Framework New/improved protocol analyzers SMB/GridFTP/Modbus/DNP3 Reaction Framework Load-balancer Interface 41

116 The Curse of Success... 42

117 The Curse of Success... Success can be kind of problematic in research... Bro is now used operationally by many sites. Demands of operations community hard to meet for small team. 42

118 The Curse of Success... Success can be kind of problematic in research... Bro is now used operationally by many sites. Demands of operations community hard to meet for small team. Aiming to establish sustainable development model. Modernize the system to make usage and contributions easier. Develop a community around the project. 42

119 The Curse of Success... Success can be kind of problematic in research... Bro is now used operationally by many sites. Demands of operations community hard to meet for small team. Aiming to establish sustainable development model. Modernize the system to make usage and contributions easier. Develop a community around the project. NSF supports work through a 3-year engineering grant. Bro changed a lot over the couples years. Collaboration with National Center for Supercomputing Applications. 42

120 Target: Blue NCSA 43

121 Target: Blue NCSA 10 PF/s peak performance >1 PF/s sustained on applications >300,000 cores >1 Petabyte memory >10 Petabyte disk storage >0.5 Exabyte archival storage Hosted in 88,000-square-foot facility 43

122 Summary 44

123 Summary Bro will keep bridging the research/operations gap. We have plenty more ideas... 44

124 Summary Bro will keep bridging the research/operations gap. We have plenty more ideas... Long-term goal is a sustainable development model. We are planing to offer commercial services and support. 44

125 Summary Bro will keep bridging the research/operations gap. We have plenty more ideas... Long-term goal is a sustainable development model. We are planing to offer commercial services and support. blog.bro-ids.org git.bro-ids.org on Twitter 44

126 Summary Bro will keep bridging the research/operations gap. We have plenty more ideas... Long-term goal is a sustainable development model. We are planing to offer commercial services and support. blog.bro-ids.org git.bro-ids.org on Twitter 44