Total Visibility 1 1

Transcription

1 Total Visibility 1 1

2 What Is Meant by Telemetry? Te lem e try a technology that allows the remote measurement and reporting of information of interest to the system designer or operator. The word is derived from Greek roots tele = remote, and metron = measure! 2

3 Check List Check SNMP. Is there more you can do with it to pull down security information? Check RMON. Can you use it? Check Netflow. Are you using it, can you pull down more? Check Passive DNS See addendum for lots of links. 3

4 Holistic Approach to System-Wide Telemetry Holistic Approach to Patient Care Uses a system-wide approach, coordinating with various specialists, resulting in the patient s better overall health and wellbeing. Cardiologist Podiatrist Ophthalmologist Neurologist Hematologist Nephrologist 4

5 Broadband, Wireless (3G, ), Ethernet, FTTH, Leased Line, ATM, Frame- Relay Holistic Approach to System-Wide Telemetry CPE/ACCESS/AGGREGATION CORE DATA/SVC PEERING Center CPE(s) PE(s) PE Listen L2 Agg. Listen P P P P Listen P P Listen ISP / Alt. Carrier Customer Edge: Shared resources and services should be available Core: Performance must not be affected Data/Service Center Data Center: Inter as well as Intra Data Center traffic SP Peering: Ability to trace through asymmetric traffic 5

6 Open Source Tools for NetFlow Analysis Visualization Investigate the spike FlowScan Source: University of Wisconsin An identified cause of the outage 6

7 What s NetFlow? NetFlow is a form of telemetry pushed from the network devices. Netflow is best used in combination with other technologies: IPS, vulnerability scanners, and full traffic capture. Traffic capture is like a wiretap NetFlow is like a phone bill We can learn a lot from studying the network phone bill! Who s talking to whom? And when? Over what protocols & ports? How much data was transferred? At what speed? For what duration? 7 7

8 Elements of a Netflow Packet Ingress i/f Netflow is our #1 tool Data Flow Data Flow Egress i/f Usage! Packet Count! Byte Count! Source IP Address! Destination IP Address! From/To! Time! of Day! Start sysuptime! End sysuptime! Source TCP/UDP Port! Destination TCP/UDP Port! Application! Port! Utilization! QoS! Input ifindex! Output ifindex! Type of Service! TCP Flags! Protocol! Next Hop Address! Source AS Number! Dest. AS Number! Source Prefix Mask! Dest. Prefix Mask! Routing and! Peering! 8

9 Netflow Setup Don t have a copy of netflow data b/c IT won t share? Many products have the ability to copy flow data off to other destinations Regionalized collection to minimize WAN impact Export netflow data to OSU Flowtools Collector Storage Collector Netflow data copied to other destinations with flow-fanout Peakflow NetQoS 9 9

10 NetFlow Collection at Cisco DMZ Netflow Collection (4 servers) Data Center Netflow Collection (20+ servers) Query/Reporting tools (OSU Flowtools, DFlow, Netflow Report Generator) 200K pps 3 ISP gateways 600GB ~ 3 months 10

11 OSU Flowtools Netflow Collector Setup Tool: OSU FlowTools! - Free!! - Developed by Ohio State University! Examples of capabilities! Did talk to ?! What hosts and ports did talk to?! Who s connecting to port TCP/ 6667?! Did anyone transfer data > 500MB to an external host?! 11 11

12 OSU Flowtools Example Who s Talking? Scenario: New botnet, variant undetected You need to identify all systems that talked to the botnet C&C Luckily you ve deployed netflow collection at all your PoPs flow.acl file uses familiar ACL syntax. create a list named bot [mynfchost]$ head flow.acl ip access- list standard bot permit host ip access- list standard bot permit host concatenate all files from Feb 12, put in specific 2007 then query filter syntax for src the or dest example! of bot acl we ve got a host in the botnet! [mynfchost]$ flow- cat /var/local/flows/data/ /ft* flow- filter - Sbot - o -... Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP :39: :40: :40: :40:

13 Custom NetFlow Report Generator Query by IP 13

14 Know Thy Subnets Critical to providing context to an incident Is the address in your DMZ? lab? remote access? desktop? data center? Make the data queryable Commercial & open source products available Build the data into your security devices SIMS - netforensics asset groups SIMS - CS-MARS network groups IDS - Cisco network locale variables variables DC_NETWORKS address , , variables DMZ_PROD_NETWORKS address variables DMZ_LAB_NETWORKS eventid= eventtype=evidsalert hostid=xxx- dc- nms- 4appName=sensorApp appinstanceid=6718 tmtime= severity=1 vlan=700 Interface=ge2_1 Protocol=tcp riskratingvalue=26 sigid=11245 sigdetails=nick...user" src= srcdir=dc_networks srcport=40266 dst= dstdir=out dstport=6665 data center host! 14 14

15 Network Telemetry - MRTG/RRDTool! Not just netflow, can also use SNMP to grab telemetry! Shows data volumes between endpoints! You must understand your network traffic volume! 15

16 Blanco Wireless: Network Network traffic data Based on our design, environment, and these aggregate traffic levels with spikes above 400Mbps, We need an IPS 4260 Subnet information - IP address management data» /19 A (Active) Data Centers!» /20 A (Active) Building 3 Data Center!» /25 S (Active) Windows Server Subnet!» /25 S (Active) Oracle 10g Subnet!» /26 S (Active) ESX VMWare Farm!» /26 S (Active) Web Application Servers!» /16 A (Active) Indiana Campus!» /19 A (Active) Data Centers!» /19 A (Active) Site 1 Desktop Networks!» /24 S (Active) Building 1 1st floor!» /25 S (Active) Building 1 2nd floor!» /25 S (Active) Building 2! 16 16

17 NetFlow - Stager Source: UNINETT 17

18 Other Visualization Techniques Using SNMP Data with Anomaly for DNS Queries RRDTool Thru put Spike Source: RTT Spike 18

19 Displaying RMON ntop Examples Source: Detailed Analysis i.e. TTL 19

20 BGP Example SQL Slammer 20

21 Correlating NetFlow and Routing Data Matching data collected from different tools 21

22 Syslog De facto logging standard for hosts, network infrastructure devices, supported in all most routers and switches Many levels of logging detail available choose the level(s) which are appropriate for each device/situation Logging of ACLs is generally contraindicated due to CPU overhead NetFlow provides more info, doesn t max the box Can be used in conjunction with Anycast and databases such as MySQL ( to provide a scalable, robust logging infrastructure Different facility numbers allows for segregation of log info based upon device type, function, other criteria Syslog-ng from adds a lot of useful functionality HOW-TO located at 22

23 Benefits of Deploying NTP Very valuable on a global network with network elements in different time zones Easy to correlate data from a global or a sizable network with a consistent time stamp NTP based timestamp allows to trace security events for chronological forensic work Any compromise or alteration is easy to detect as network elements would go out of sync with the main clock Did you there is an NTP MIB? Some think that we may be able to use NTP Jitter to watch what is happening in the network. 23

24 Packet Capture Examples Source: Wealth of information, L1-L7 raw data for analysis 24

25 Total Visibility Addendum 25 25

26 NetFlow More Information Cisco NetFlow Home warp/public/732/tech/nmp/netflow Linux NetFlow Reports HOWTO Arbor Networks Peakflow SP products_sp.php 26

27 More Information about SNMP Cisco SNMP Object Tracker Mibbrowser/mibinfo.pl?tab=4 Cisco MIBs and Trap Definitions netmgmt/cmtk/mibs.shtml SNMPLink SEC-1101/2102 give which SNMP parameters should be looked at. 27

28 RMON More Information IETF RMON WG rmonmib-charter.html Cisco RMON Home en/us/tech/tk648/tk362/tk560/ tech_protocol_home.html Cisco NAM Product Page hw/modules/ps2706/ps5025/index.html 28

29 BGP More Information Cisco BGP Home tk80/tech_protocol_family_home.html Slammer/BGP analysis massey_iwdc03.pdf Team CYMRU BGP Tools 29

30 Syslog More Information Syslog.org - Syslog Logging w/postgres HOWTO syslog_postgresql/ Agent Smith Explains Syslog 30

31 Packet Capture More Information tcpdump/libpcap Home Vinayak Hegde s Linux Gazette article vinayak.html 31

32 Remote Triggered Black Hole Remote Triggered Black Hole filtering is the foundation for a whole series of techniques to traceback and react to DOS/DDOS attacks on an ISP s network. Preparation does not effect ISP operations or performance. It does adds the option to an ISP s security toolkit. 32

33 More Netflow Tools NfSen - Netflow Sensor NFDUMP FlowCon 33