NetFlow Security Monitoring with Cisco Threat Defense Matthew Robertson, Security Technical Marketing Engineer BRKSEC-2073
The world is full of obvious things which nobody by any chance observes. Sherlock Holmes, The Hound of the Baskervilles
Evolution of Cyber Conflict Manual Attacks (1980s) War Dialing, Phone Phreaking Mechanized Attacks (1988) Viruses, Worms Google, RSA Talented Human / Mechanized Attackers (2009) APT, Multi-Step Attacks Target, Neiman Marcus DIY Human / Mechanized Attackers (2011) Cyrptocurrency Ransoms, Store-bought Credentials... Manual Defenses Unplug Mechanized Defenses Firewall, IDS/IPS Targeted Human/Mechanized Defenders Reputation, App-aware Firewall Intelligence Driven Human Defenders
Agenda Introduction Understanding the Landscape Introduction to NetFlow Adding Context Flow Collection Flow Export Design and Deployment Working with NetFlow Discovery Identifying IOC s Responding Summary
About this session http://www.cisco.com/go/threatdefense http://www.cisco.com/go/securedatacenter https://learningnetwork.cisco.com/community/certifications/s ecurity/cybersecurity/scyber_exam
About the Speaker Matthew Robertson Security Technical Marketing Engineer Partner Product Team Development and Technical Marketing Focused on advanced threat detection Author of 3 CVD s I am Canadian!
Thinking Beyond the Perimeter Allen Pace Dunbar Armored Facility Robbery: 1997 - $18M
Case Study: Retailer
What do these stories have in common? The Insider Threat
Three Kinds of Insider Threats Negligent Insiders: Employees who accidentally expose data Malicious Insiders: Employees who intentionally expose data Compromised Insiders: Employees whose access credentials or devices have been compromised by an outside attacker
Managing the Insider Threat Data
Managing the Insider Threat Access Controls Control who and what is on the network Segmentation Define what they can do SGT
Managing the Insider Threat Control movement of malicious content through inspection points Content Controls Deep contextual visibility at inspection points
Once the walls are built monitor for security visibility
Agenda Introduction Understanding the Landscape Introduction to NetFlow
eth0/1 eth0/2 NetFlow 10.2.2.2 port 1024 10.1.1.1 port 80 Start Time Interface Src IP Src Dest IP Dest Proto Pkts Bytes SGT DGT TCP Flags Port Port Sent Sent 10:20:12.221 eth0/1 10.2.2.2 1024 10.1.1.1 80 TCP 5 1025 100 1010 SYN,ACK,PSH 10:20:12.871 eth0/2 10.1.1.1 80 10.2.2.2 1024 TCP 17 28712 1010 100 SYN,ACK,FIN
NetFlow = Visibility A single NetFlow Record provides a wealth of information Router# show flow monitor CYBER-MONITOR cache IPV4 SOURCE ADDRESS: 192.168.100.100 IPV4 DESTINATION ADDRESS: 192.168.20.6 TRNS SOURCE PORT: 47321 TRNS DESTINATION PORT: 443 INTERFACE INPUT: Gi0/0/0 FLOW CTS SOURCE GROUP TAG: 100 FLOW CTS DESTINATION GROUP TAG: 1010 IP TOS: 0x00 IP PROTOCOL: 6 ipv4 next hop address: 192.168.20.6 tcp flags: 0x1A interface output: Gi0/1.20 counter bytes: 1482 counter packets: 23 timestamp first: 12:33:53.358 timestamp last: 12:33:53.370 ip dscp: 0x00 ip ttl min: 127 ip ttl max: 127 application name: nbar secure-http
NetFlow Analysis can help: Discovery Identify business critical applications and services across the network Identify additional IOCs Policy & Segmentation Network Behaviour & Anomaly Detection (NBAD) Better understand / respond to an IOC: Audit trail of all host-to-host communication
Agenda Introduction Understanding the Landscape Introduction to NetFlow Flow Export Design and Deployment
NetFlow Deployment Architecture Management/Reporting Layer: Run queries on flow data Centralize management and reporting Flow Collection Layer: Collection, storage and analysis of flow records NetFlow Flow Exporting Layer: Enables telemetry export As close to the traffic source as possible
Considerations: Flow Exporting Layer 1. NetFlow support 2. Which version of NetFlow to use 3. How to configure/what to measure 4. Where in the network to enable NetFlow export
Cisco NetFlow Support Cisco 2800 Cisco 7200 VXR Cisco 2900 Cisco Catalyst 6500 Cisco 3560/3750-X/3850 Cisco NGA Cisco Nexus 1000v Cisco 1700 Cisco Catalyst 4500 Cisco ISR G2 Cisco XR 12000 Cisco 7600 Hardware Supported Cisco Nexus 7000 Cisco ASR Cisco ASA
NetFlow Version 5 Fixed format
Versions of NetFlow Version Major Advantage Limits/Weaknesses V5 V9 Flexible NetFlow (FNF) IP Flow Information Export (IPFIX) AKA NetFlow V10 NSEL (ASA only) Defines 18 exported fields Simple and compact format Most commonly used format Template-based IPv6 flows transported in IPv4 packets MPLS and BGP nexthop supported Defines 104 fields, including L2 fields Reports flow direction Template-based flow format (built on V9 protocol) Supports flow monitors (discrete caches) Supports selectable key fields and IPv6 Supports NBAR data fields Standardized RFC 5101, 5102, 6313 Supports variable length fields, NBAR2 Can export flows via IPv4 and IPv6 packets Built on NetFlow v9 protocol State-based flow logging (context) Pre and Post NAT reporting IPv4 only Fixed fields, fixed length fields only Single flow cache IPv6 flows transported in IPv4 packets Fixed length fields only Uses more memory Slower performance Single flow cache Less common Requires more sophisticated platform to produce Requires more sophisticated system to consume Even less common Only supported on a few Cisco platforms Missing many standard fields Limited support by collectors
Configuring Flexible NetFlow 1. Configure the Exporter Router(config)# flow exporter my-exporter Where do I want my data sent? Router(config-flow-exporter)# destination 1.1.1.1 2. Configure the Flow Record Router(config)# flow record my-record Router(config-flow-record)# What data match do I want ipv4 to destination meter? address Router(config-flow-record)# match ipv4 source address Router(config-flow-record)# collect counter bytes 3. Configure the Flow Monitor Router(config)# flow monitor my-monitor How do I want to cache information Router(config-flow-monitor)# exporter my-exporter Router(config-flow-monitor)# record my-record 4. Apply to an Interface Router(config)# interface s3/0 Which interface do I want to monitor? Router(config-if)# ip flow monitor my-monitor input Best Practice: include all v5 fields
NetFlow Deployment Each network layer offers unique NetFlow capabilities Access Distribution & Core Edge Catalyst 3560/3750-X ISR Catalyst 4500 Catalyst 4500 ASA Catalyst 3650/3850 Catalyst 6500 ASR
NetFlow Deployment Access Catalyst 3560/3750-X Catalyst 4500 Catalyst 3650/3850 Access: New network edge Detect threats as the enter the network Detect threats inside the switch east-west Layer 2 traffic Fewer false positives Higher-granular visibility Identify the endpoint collect MAC Address
Catalyst 3650-X,3750-X Flow Record! flow record CYBER_3KX_FLOW_RECORD match datalink mac sourceaddress match datalink mac destination-address match datalink mac source-vlan-id match ipv4 tos match ipv4 ttl match ipv4 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port collect interface input snmp collect interface output snmp collect counter bytes collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last!
Catalyst 4500 Flow Record! flow record cts-cyber-4k match ipv4 tos match ipv4 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port match interface input match flow direction collect flow cts source group-tag collect flow cts destination group-tag collect flow cts switch derived-sgt collect transport tcp flags collect interface output collect counter bytes collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last!
NetFlow Deployment - Converged Access Converged Access: NetFlow for the first time on Wireless Visibility in BYOD environments Consistent configuration for wired and wireless Single flow monitor can be applied to wired ports and SSID Natively available in the UADP ASIC Can monitor East-West and North-South flows 48k flows on the 48 port model
Considerations: 3850! Ingress: SGT Sources: Derived from packet header DGT Sources: Derived based on destination IP lookup SGACL enforcement must be enabled Trunk link only Egress: SGT Sources: Incoming packet header Port configured SGT IP to SGT mapping DGT Sources: Derived based on destination IP lookup Requires SGACL enforcement to be enabled Trunk link only flow monitor cts-cyber-monitor-in exporter StealthWatch-FC cache timeout active 60 record cts-cyber-3k-in!! flow monitor cts-cyber-monitor-out exporter StealthWatch-FC cache timeout active 60 record cts-cyber-3k-out! interface GigabitEthernet1/0/1 ip flow monitor cts-cyber-monitor-in input ip flow monitor cts-cyber-monitor-out output! vlan configuration 100 ip flow monitor cts-cyber-monitor-in input ip flow monitor cts-cyber-monitor-out output!
Catalyst 3850/3650 Flow Record! flow record cts-cyber-3k-in match datalink mac source address input match datalink mac destination address input match ipv4 tos match ipv4 ttl match ipv4 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port match interface input match flow direction match flow cts source group-tag match flow cts destination group-tag collect counter bytes long collect counter packets long collect timestamp absolute first collect timestamp absolute last!! flow record cts-cyber-3k-out match ipv4 tos match ipv4 ttl match ipv4 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port match flow direction match flow cts source group-tag match flow cts destination group-tag collect counter bytes long collect counter packets long collect timestamp absolute first collect timestamp absolute last!
NetFlow Deployment Distribution & Core Catalyst 4500 Distribution & Core: Traditional deployment Minimal recommended deployment Enable at critical points/bottle necks Typically done on a Layer 3 boundary Detect threats internal to the VLAN When deployed on an SVI interface Detect threats as they traverse the internal network Move between subnets Catalyst 6500
Catalyst 6500 (Sup 2T) Flow Record! flow record cts-cyber-6k match ipv4 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port match flow cts source group-tag match flow cts destination group-tag collect transport tcp flags collect interface output collect counter bytes collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last!
NetFlow Deployment Edge ISR ASA Edge: Detect threats as they enter and leave the network Monitor communication between branches Gain context from edge devices Application - NBAR Events, NAT & User-ID - NSEL ASR
NetFlow Deployment: Edge with ASA NetFlow Security Event Logging: Provides visualization into policy enforcement points Monitor communication between branches Efficient event reporting mechanism: Syslog - Verbose, text based, single event per packet: ~30% processing overhead NetFlow - Compact, binary, multiple events per packet: ~7-10% processing overhead Context rich: Event driven: Flow Created, Denied, tear-down Network Address Translations User-ID
ISR Flow Record! flow record cts-cyber-ipv4 match ipv4 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port match interface input match flow direction match flow cts source group-tag match flow cts destination group-tag collect routing next-hop address ipv4 collect ipv4 dscp collect ipv4 ttl minimum collect ipv4 ttl maximum collect transport tcp flags collect interface output collect counter bytes collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last collect application name! Enable NBAR
ASA NSEL Configuration! flow-export destination management <ip-address> 2055! policy-map global_policy class class-default flow-export event-type all destination <ip-address>! flow-export template timeout-rate 2 logging flow-export syslogs disable!
Flow Monitor Configuration! flow monitor CYBER_MONITOR exporter CYBER_EXPORTER cache timeout active 60 cache timeout inactive 15! record CYBER_RECORD Inactive Timeout: How long a flow can be inactive before being removed from cache Recommended 15 seconds All exporters should have the same timeout Active Timeout: Longest amount of time a flow can be in cache without exporting a Flow Record Recommended 60 seconds All exporters should have the same timeout
Aside: Myths about NetFlow Generation Myth #1: NetFlow impacts performance Hardware implemented NetFlow has no performance impact Software implementation is typically significantly <15% processing overhead Myth #2: NetFlow has bandwidth overhead NetFlow is a summary protocol Traffic overhead is typically significantly <1% of total traffic per exporting device
Agenda Introduction Understanding the Landscape Introduction to NetFlow Flow Collection Flow Export Design and Deployment
Components for NetFlow Security Monitoring StealthWatch Management Console Management and reporting Up to 25 FlowCollectors Up 6 million fps globally StealthWatch FlowCollector Collect and analyze Up to 2000 sources Up to sustained 240,000 fps UDP Director UDP Packet copier Forward to multiple collection systems NetFlow Cisco Network Best Practice: Centralize collection globally StealthWatch FlowSensor (VE) Generate NetFlow data Additional contextual fields (ex. App, URL, SRT, RTT)
eth0/1 eth0/2 NetFlow Collection: Flow Stitching Uni-directional flow records 10.2.2.2 port 1024 Start Time Interface Src IP Src Port Dest IP Dest Port Proto 10.1.1.1 port 80 Pkts Sent Bytes Sent 10:20:12.221 eth0/1 10.2.2.2 1024 10.1.1.1 80 TCP 5 1025 100 1010 10:20:12.871 eth0/2 10.1.1.1 80 10.2.2.2 1024 TCP 17 28712 1010 100 SGT DGT Start Time Client IP Client Port Server IP Server Port Proto Client Bytes Client Pkts Server Bytes Server Pkts Client SGT Server SGT Interfaces 10:20:12.221 10.2.2.2 1024 10.1.1.1 80 TCP 1025 5 28712 17 100 1010 eth0/1 eth0/2 Bi-directional: Conversation flow record Allows easy visualization and analysis
NetFlow Collection: De-duplication Start Time 10.2.2.2 port 1024 Sw1 ASA 10.1.1.1 port 80 Client IP Client Port Server IP Server Port Prot o Client Bytes Client Pkts Server Bytes Server Pkts App Client SGT Server SGT Exporter, Interface, Direction, Action 10:20:12.221 10.2.2.2 1024 10.1.1.1 80 TCP 1025 5 28712 17 HTTP 100 1010 Sw1, eth0, in Sw1, eth1, out Sw2, eth0, in Sw2, eth1, out ASA, eth1, in ASA, eth0, out, Permitted ASA eth0, in, Permitted ASA, eth1, out Sw3, eth1, in Sw3, eth0, out Sw1, eth1, in Sw1, eth0, out Sw2 Sw3
Conversational Flow Record Who What Who When Where How More context Highly scalable (enterprise class) collection High compression => long term storage Months of data retention
Conversational Flow Record: Exporters Path the flow is taking through the network
Agenda Introduction The Insider Threat Concepts and Attribution Adding Context Flow Collection Flow Export NetFlow Design and Deployment
Context is Critical
Host Groups: Applied Situational Awareness Virtual container of multiple IP Addresses/ranges that have similar attributes Lab servers Best Practice: classify all known IP Addresses in one or more host groups
ISE as a Telemetry Source Monitor Mode Open Mode, Multi-Auth Unobstructed Access No impact on productivity Profiling, posture assessment Gain Visibility StealthWatch Management Console Maintain historical session table Correlate NetFlow to username Build User-centric reports syslog Cisco ISE Authenticated Session Table
Configuration: Logging on ISE 1 1. Create Remote Logging Target on ISE 2. Add Target to Logging Categories 2 Required Logging categories: Passed Authentications RADIUS Accounting Profiler Administrative and Operational Audit
Configuration: Add ISE to SMC 1. (Not Shown) Create Admin User on ISE 2. (Not Shown) Configure ISE or CA certificate on SMC 3. Add Cisco ISE nodes to SMC Configuration
Global Intelligence Known C&C Servers Tor Entrance and Exits
Conversational Flow Record NBAR Geo-IP mapping ISE Telemetry Applied situational awareness Threat feed FlowSensor
Agenda Introduction The Insider Threat Concepts and Attribution Adding Context Flow Collection Flow Export NetFlow Design and Deployment Working with NetFlow
Agenda Introduction The Insider Threat Concepts and Attribution Adding Context Flow Collection Flow Export NetFlow Design and Deployment Working with NetFlow Discovery
There is nothing like first hand evidence Sherlock Holmes, A Study in Scarlett
Flow Query Basics The Flow Table Filter Filter conditions Details More details
Flow Query Basics - Filtering Select host to investigate All flows in which this host was a client or server
Flow Query Basics - Filtering All flows for 10.10.200.79 in the last hour
Flow Table: Visibility across NAT User Inside local Outside global Server
Host Groups Application Report Applications inbound Applications outbound
Host Groups Targeted Reporting Geo-IP-based Host Group Summary chart of traffic inbound and outbound from this Host Group
Host Groups Targeted Reporting Traffic inbound Traffic outbound
Host Groups Discovering Rogue Hosts Catch All: All unclassified RFC1918 addresses Table of all individual hosts
Host Groups Discovering Rogue Hosts Rogue Hosts
Agenda Introduction The Insider Threat Concepts and Attribution Adding Context Flow Collection Flow Export NetFlow Design and Deployment Working with NetFlow Discovery Identifying IOC s
Concept: Indicator of Compromise an artifact observed on a network or in operating system that with high confidence indicates a computer intrusion http://en.wikipedia.org/wiki/indicator_of_compromise IDS/IPS Alert IP Addresses File hashes Log analysis (SIEM) Raw flow analysis Outside notification Anomaly detection Behavioural analysis Activity monitoring
Attack Lifecycle Model Exploratory Actions Theft Initial Recon Initial Compromise Infiltration (C&C) Footprint Expansion Execution Staging Disruption
IoC s from Traffic Analysis Behavioural Analysis: Leverages knowledge of known bad behaviour Policy and segmentation Anomaly Detection: Identify a change from normal
Behaviour Analysis Leverages knowledge of known bad behaviour
Segmentation Monitoring Forbidden relationship Host Groups Relationship
Unauthorized Access Attempted communication in violation of policy Flow denied by firewall rule
Custom Security Events and Host Locking Object conditions Peer conditions Connection conditions Time range
Policy Violations Communication in violation of policy Active alarm monitoring adherence to policy
Anomaly Detection Identify an change from normal
Anomaly Detection Identify an change from normal
Anomaly Detection Identify an change from normal
Anomaly Detection Identify an change from normal
Anomaly Detection Identify an change from normal This is weird. Very curious. What up, dude?
StealthWatch NBAD Model Track and/or measure behaviour/activity Notification of security event generated Algorithm Security Event Alarm Suspicious behaviour observed or anomaly detected
Alarm Categories Each category accrues points.
Example Alarm Category: Concern Index Concern Index: Track hosts that appear to compromising network integrity Security events. Over 80 different algorithms.
StealthWatch: Alarms Alarms Indicate significant behaviour changes and policy violations Known and unknown attacks generate alarms Activity that falls outside the baseline, acceptable behaviour or established policies
Policy Tuning Policies can be created for individual host groups Tune alarm thresholds Default policy for Inside and Outside hosts
Internal Reconnaissance Concern Index Events Scanning on TCP-445 across multiple subnets
High Concern Index Baseline deviated by 2,432%!
Watching for Data Theft Data Exfiltration Identify suspect movement from Inside Network to Outside Single or multiple destinations from a single source Policy and behavioral
Data Hoarding
Data Hoarding Suspect Data Hoarding: Unusually large amount of data inbound from other hosts Target Data Hoarding: Unusually large amount of data outbound from a host to multiple hosts
Suspect Data Hoarding Data Hoarding Unusually large amount of data inbound to a host from other hosts Policy and behavioral
Agenda Introduction The Insider Threat Concepts and Attribution Adding Context Flow Collection Flow Export NetFlow Design and Deployment Working with NetFlow Discovery Identifying IOC s Responding
Responding Exploratory Actions Theft Initial Recon Initial Compromise Infiltration (C&C) Footprint Expansion Execution Staging Disruption IOC Found: Investigate forwards and backward
The Science of Deduction. Chapter 1: The Sign of the Four
The Science of Deduction Gathering Evidence What did they get? IOC Where did they go? Data Element When did they get it? Who is they? Are they still here?
Responding to an IOC IOC: Security vendor publishes list of IP addresses identified as BlackPOS servers Create a Host Group for BlackPOS Servers IP Addresses
BlackPOS Host Locking Violation Alarm Set client hosts to POS terminals Create a Host Lock Violation Alarm for communication to BlackPOS servers Set server hosts to BlackPOS Servers Alarm on FTP traffic Trigger alarm on unsuccessful connections
BlackPOS - Investigate You know today what you didn t know yesterday Run a Flow Query Over the last 90 days Configure application to be FTP Server or client includes the known bad BlackPOS IP Addresses
BlackPOS Returned Flows Infected hosts FTP Transfers BlackPOS Servers
Investigating a Host Host report for 10.201.3.59 Summary information Behavior alarms Quick view of host group communication IOC: IDS Alert indicating a known worm operating inside your network
Investigating: Host Drilldown User information Applications
Investigating: Applications A lot of applications. Some suspicious!
Investigating: Behaviour Alarms Significant network activity
Investigating: Security Events associated with host Touched hosts.
Investigating: View all Flows Network behavior retroactively analyzed
It Could Start with a User Username View Flows Active Directory Details Alarms Devices and Sessions
Audit Trails Network behavior retroactively analyzed
Agenda Introduction The Insider Threat Concepts and Attribution Adding Context Flow Collection Flow Export NetFlow Design and Deployment Working with NetFlow Discovery Identifying IOC s Responding Summary
Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings Related sessions
Related Sessions BRKSEC-2026 Network as a Sensor and Enforcer Darrin Miller, Matt Robertson Monday, Jun 8 1:00-3:00 BRKCRS-1449 - Threat Defense for Enterprise Networks with Unified Access Vaibhav Katkade Tuesday, Jun 9, 3:30-5:00 PCSZEN-1003 - Network as a Sensor: Using NetFlow for Incident Response Gavin Reid, Matt Valites Wednesday, Jun 10, 9:15 9:45 BRKSEC-3010 - Detecting Adversarial Threats - Tools, Techniques, and Infrastructure to Find the Bad Guys Matt Healy, Paul Eckstein Monday, Jun 8, 1:00 3:00 BRKSEC-3068 Intermediate - Red Team, Blue Team: Lessons Learned for Real World Attacks Jamey Heary, Nick Hitchcock Monday Jun 8, 10:00-12:00
Links and Recommended Reading More about the Cisco Cyber Threat Defense Solution: http://www.cisco.com/go/threatdefense http://www.lancope.com Recommended Reading Cyber Threat Defense Cisco Validated Design Guide: http://www.cisco.com/en/us/solutions/collateral/ns1015/ns1238/cyber_threat_defense_design_guide.pdf Cyber Threat Defense for the Data Center Cisco Validated Design Guide: http://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise/design-zone-security/ctd-first-look-design-guide.pdf Securing Cisco Networks with Threat Detection and Analysis (SCYBER) https://learningnetwork.cisco.com/community/certifications/security/cybersecurity/scyber_exam
Participate in the My Favorite Speaker Contest Promote Your Favorite Speaker and You Could Be a Winner Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress) Send a tweet and include Your favorite speaker s Twitter handle @mattrobertson25 Two hashtags: #CLUS #MyFavoriteSpeaker You can submit an entry for more than one of your favorite speakers Don t forget to follow @CiscoLive and @CiscoPress View the official rules at http://bit.ly/cluswin
Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
Key Takeaways Insider threats are operating on the network interior Threat detection and response requires visibility and context into network traffic NetFlow and the Lancope StealthWatch System provide actionable security intelligence
Q & A
The game is afoot! Sherlock Holmes, The Adventure of the The Abbey Grange
Thank you