Total Visibility 1 1
What Is Meant by Telemetry? Te lem e try a technology that allows the remote measurement and reporting of information of interest to the system designer or operator. The word is derived from Greek roots tele = remote, and metron = measure! 2
Check List Check SNMP. Is there more you can do with it to pull down security information? Check RMON. Can you use it? Check Netflow. Are you using it, can you pull down more? Check Passive DNS See addendum for lots of links. 3
Holistic Approach to System-Wide Telemetry Holistic Approach to Patient Care Uses a system-wide approach, coordinating with various specialists, resulting in the patient s better overall health and wellbeing. Cardiologist Podiatrist Ophthalmologist Neurologist Hematologist Nephrologist 4
Broadband, Wireless (3G, 802.11), Ethernet, FTTH, Leased Line, ATM, Frame- Relay Holistic Approach to System-Wide Telemetry CPE/ACCESS/AGGREGATION CORE DATA/SVC PEERING Center CPE(s) PE(s) PE Listen L2 Agg. Listen P P P P Listen P P Listen ISP / Alt. Carrier Customer Edge: Shared resources and services should be available Core: Performance must not be affected Data/Service Center Data Center: Inter as well as Intra Data Center traffic SP Peering: Ability to trace through asymmetric traffic 5
Open Source Tools for NetFlow Analysis Visualization Investigate the spike FlowScan Source: University of Wisconsin An identified cause of the outage 6
What s NetFlow? NetFlow is a form of telemetry pushed from the network devices. Netflow is best used in combination with other technologies: IPS, vulnerability scanners, and full traffic capture. Traffic capture is like a wiretap NetFlow is like a phone bill We can learn a lot from studying the network phone bill! Who s talking to whom? And when? Over what protocols & ports? How much data was transferred? At what speed? For what duration? 7 7
Elements of a Netflow Packet Ingress i/f Netflow is our #1 tool Data Flow Data Flow Egress i/f Usage! Packet Count! Byte Count! Source IP Address! Destination IP Address! From/To! Time! of Day! Start sysuptime! End sysuptime! Source TCP/UDP Port! Destination TCP/UDP Port! Application! Port! Utilization! QoS! Input ifindex! Output ifindex! Type of Service! TCP Flags! Protocol! Next Hop Address! Source AS Number! Dest. AS Number! Source Prefix Mask! Dest. Prefix Mask! Routing and! Peering! 8
Netflow Setup Don t have a copy of netflow data b/c IT won t share? Many products have the ability to copy flow data off to other destinations Regionalized collection to minimize WAN impact Export netflow data to OSU Flowtools Collector Storage Collector Netflow data copied to other destinations with flow-fanout Peakflow NetQoS 9 9
NetFlow Collection at Cisco DMZ Netflow Collection (4 servers) Data Center Netflow Collection (20+ servers) Query/Reporting tools (OSU Flowtools, DFlow, Netflow Report Generator) 200K pps 3 ISP gateways 600GB ~ 3 months 10
OSU Flowtools Netflow Collector Setup Tool: OSU FlowTools! - Free!! - Developed by Ohio State University! Examples of capabilities! Did 192.168.15.40 talk to 216.213.22.14?! What hosts and ports did 192.168.15.40 talk to?! Who s connecting to port TCP/ 6667?! Did anyone transfer data > 500MB to an external host?! 11 11
OSU Flowtools Example Who s Talking? Scenario: New botnet, variant undetected You need to identify all systems that talked to the botnet C&C Luckily you ve deployed netflow collection at all your PoPs flow.acl file uses familiar ACL syntax. create a list named bot [mynfchost]$ head flow.acl ip access- list standard bot permit host 69.50.180.3 ip access- list standard bot permit host 66.182.153.176 concatenate all files from Feb 12, put in specific 2007 then query filter syntax for src the or dest example! of bot acl we ve got a host in the botnet! [mynfchost]$ flow- cat /var/local/flows/data/2007-02- 12/ft* flow- filter - Sbot - o -... Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP 0213.08:39:49.911 0213.08:40:34.519 58 10.10.71.100 8343 98 69.50.180.3 31337 0213.08:40:33.590 0213.08:40:42.294 98 69.50.180.3 31337 58 10.10.71.100 83 12 12
Custom NetFlow Report Generator Query by IP 13
Know Thy Subnets Critical to providing context to an incident Is the address in your DMZ? lab? remote access? desktop? data center? Make the data queryable Commercial & open source products available Build the data into your security devices SIMS - netforensics asset groups SIMS - CS-MARS network groups IDS - Cisco network locale variables variables DC_NETWORKS address 10.2.121.0-10.2.121.255,10.3.120.0-10.3.127. 255,10.4.8.0-10.4.15.255 variables DMZ_PROD_NETWORKS address 198.133.219.0-198.133.219.255 variables DMZ_LAB_NETWORKS 172.16.10.0-172.16.11.255 eventid=1168468372254753459 eventtype=evidsalert hostid=xxx- dc- nms- 4appName=sensorApp appinstanceid=6718 tmtime=1178426525155 severity=1 vlan=700 Interface=ge2_1 Protocol=tcp riskratingvalue=26 sigid=11245 sigdetails=nick...user" src=10.2.121.10 srcdir=dc_networks srcport=40266 dst=208.71.169.36 dstdir=out dstport=6665 data center host! 14 14
Network Telemetry - MRTG/RRDTool! Not just netflow, can also use SNMP to grab telemetry! Shows data volumes between endpoints! You must understand your network traffic volume! 15
Blanco Wireless: Network Network traffic data Based on our design, environment, and these aggregate traffic levels with spikes above 400Mbps, We need an IPS 4260 Subnet information - IP address management data» 10.10.0.0/19 A (Active) Data Centers!» -- 10.10.0.0/20 A (Active) Building 3 Data Center!» -- 10.10.0.0/25 S (Active) Windows Server Subnet!» -- 10.10.0.128/25 S (Active) Oracle 10g Subnet!» -- 10.10.1.0/26 S (Active) ESX VMWare Farm!» -- 10.10.1.64./26 S (Active) Web Application Servers!» 10.10.0.0/16 A (Active) Indiana Campus!» -- 10.10.0.0/19 A (Active) Data Centers!» -- 10.10.32.0/19 A (Active) Site 1 Desktop Networks!» -- 10.10.32.0/24 S (Active) Building 1 1st floor!» -- 10.10.33.0/25 S (Active) Building 1 2nd floor!» -- 10.10.33.128/25 S (Active) Building 2! 16 16
NetFlow - Stager Source: UNINETT 17
Other Visualization Techniques Using SNMP Data with Anomaly for DNS Queries RRDTool Thru put Spike Source: http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/ RTT Spike 18
Displaying RMON ntop Examples Source: http://www.ntop.org Detailed Analysis i.e. TTL 19
BGP Example SQL Slammer 20
Correlating NetFlow and Routing Data Matching data collected from different tools 21
Syslog De facto logging standard for hosts, network infrastructure devices, supported in all most routers and switches Many levels of logging detail available choose the level(s) which are appropriate for each device/situation Logging of ACLs is generally contraindicated due to CPU overhead NetFlow provides more info, doesn t max the box Can be used in conjunction with Anycast and databases such as MySQL (http://www.mysql.com) to provide a scalable, robust logging infrastructure Different facility numbers allows for segregation of log info based upon device type, function, other criteria Syslog-ng from http://www.balabit.com/products/syslog_ng/ adds a lot of useful functionality HOW-TO located at http://www.campin.net/newlogcheck.html 22
Benefits of Deploying NTP Very valuable on a global network with network elements in different time zones Easy to correlate data from a global or a sizable network with a consistent time stamp NTP based timestamp allows to trace security events for chronological forensic work Any compromise or alteration is easy to detect as network elements would go out of sync with the main clock Did you there is an NTP MIB? Some think that we may be able to use NTP Jitter to watch what is happening in the network. 23
Packet Capture Examples Source: http://www.ethereal.com Wealth of information, L1-L7 raw data for analysis 24
Total Visibility Addendum 25 25
NetFlow More Information Cisco NetFlow Home http://www.cisco.com/ warp/public/732/tech/nmp/netflow Linux NetFlow Reports HOWTO http:// www.linuxgeek.org/netflow-howto.php Arbor Networks Peakflow SP http://www.arbornetworks.com/ products_sp.php 26
More Information about SNMP Cisco SNMP Object Tracker http://www.cisco.com/pcgi-bin/support/ Mibbrowser/mibinfo.pl?tab=4 Cisco MIBs and Trap Definitions http://www.cisco.com/public/sw-center/ netmgmt/cmtk/mibs.shtml SNMPLink http://www.snmplink.org/ SEC-1101/2102 give which SNMP parameters should be looked at. 27
RMON More Information IETF RMON WG http://www.ietf.org/html.charters/ rmonmib-charter.html Cisco RMON Home http://www.cisco.com/ en/us/tech/tk648/tk362/tk560/ tech_protocol_home.html Cisco NAM Product Page http://www.cisco.com/en/us/products/ hw/modules/ps2706/ps5025/index.html 28
BGP More Information Cisco BGP Home http://www.cisco.com/en/us/tech/tk365/ tk80/tech_protocol_family_home.html Slammer/BGP analysis http://www.nge.isi.edu/~masseyd/pubs/ massey_iwdc03.pdf Team CYMRU BGP Tools http://www.cymru.com/bgp/index.html 29
Syslog More Information Syslog.org - http://www.syslog.org/ Syslog Logging w/postgres HOWTO http://kdough.net/projects/howto/ syslog_postgresql/ Agent Smith Explains Syslog http://routergod.com/agentsmith/ 30
Packet Capture More Information tcpdump/libpcap Home http://www.tcpdump.org/ Vinayak Hegde s Linux Gazette article http://www.linuxgazette.com/issue86/ vinayak.html 31
Remote Triggered Black Hole Remote Triggered Black Hole filtering is the foundation for a whole series of techniques to traceback and react to DOS/DDOS attacks on an ISP s network. Preparation does not effect ISP operations or performance. It does adds the option to an ISP s security toolkit. 32
More Netflow Tools NfSen - Netflow Sensor http://nfsen.sourceforge.net/ NFDUMP http://nfdump.sourceforge.net/ FlowCon http://www.cert.org/flocon/ 33