Challenges in High Performance Network Monitoring

Outline Challenges in High Performance Network Monitoring How to monitor networks that become faster and faster Fulvio Risso (fulvio.risso@polito.it) http://staff.polito.it/fulvio.risso/ Introduction What is Network Monitoring Why you need Network Monitoring What to monitor Technologies How to get data Active Network Monitoring Ping, traceroute, pathchar, RIPE TT Passive Network Monitoring Polling, event reporting Sniffing, SNMP, RMON, Flow-based technologies Challenges in High Speed Networks Speed Information overload (e.g. storage) N E T G R O U P P O L I T E C N I C O D I T O R I N O 1/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 2/75 What is Network Monitoring Why Network Monitoring? Network monitoring relates to the observation and the analysis of the status and behaviour of the following managed objects: network devices end systems network links network traffic network applications Network statistics (for optimization and planning) Network mapping/inventory Network monitoring Traffic statistics Identification of routers and servers (DNS, ) Security bandwidth usage Mapping client characteristics (opened ports, ) service usage Troubleshooting Identifying unofficial services or servers Detection traffic of distribution network security (e.g. local violations vs. remote) Accounting Network Faulty Hardware Intrusion optimization Detection and hardening (to achieve responsiveness (No) Connectivity to change and growth) Keep Compromised Hosts Bottlenecks Resource logs of users and activities service availability Protecting your network from the world Throughput N E T G R O U P P O L I T E C N I C O D I T O R I N O 3/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 4/75

Why you need Network Monitoring (1) Why you need Network Monitoring (2) Network statistics (for optimization and planning) Network monitoring Traffic statistics (bandwidth usage, service usage, traffic distribution (e.g. local vs. remote)) Network optimization and hardening (to achieve responsiveness to change and growth) Bottlenecks Throughput Network mapping/inventory: Identification of routers and servers (DNS, ) Mapping client characteristics (opened ports, ) Security Identifying unofficial services or servers Detection of network security violations Intrusion Detection Compromised Hosts Protecting your network from the world Troubleshooting Faulty Hardware (No) Connectivity Resource and service availability Accounting Keep logs of users activities N E T G R O U P P O L I T E C N I C O D I T O R I N O 5/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 6/75 What to monitor? By far the most important! Traffic Measurements When you already know what to measure E.g. get the amount of IP traffic Generic monitors When you do not know exactly what to measure E.g. get the distribution of the network-layer protocols Traffic characterization When you want to create a model (mathematical, maybe?) of the traffic E.g. extract some valuable data from the current traffic Probes When you want to probe your network Availability (links, network resources, services, etc) Events and Alerts (e.g. traffic thresholds) Example: ntop Ntop is a simple, open source (GPL), portable traffic measurement and monitoring tool, which supports various management activities, including network optimization and planning and detection of security violations N E T G R O U P P O L I T E C N I C O D I T O R I N O 7/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 8/75

What ntop does (1) What ntop does (2) Traffic Measurement Data sent/received: Volume and packets, classified according to network/ip protocol Multicast Traffic TCP Session History Bandwidth Measurement and Analysis Traffic Characterisation and Monitoring Network Flows Protocol utilisation (# req, peaks/storms, positive/negative repl.) and distribution Network Traffic Matrix ARP, ICMP Monitoring Network Optimisation and Planning Passive network mapping/inventory: identification of Routers and Internet Servers (DNS, Proxy) Traffic Distribution (Local vs. Remote) Service Mapping: service usage (DNS, Routing) Anomalies Detection through some common traffic parameters ICMP ECHO request/response ratio ICMP Destination/Port Unreachable # SYN Pkts vs. # Active TCP Connections Suspicious packets (e.g. out of sequence) Fragments percentage Traffic from/to diagnostic ports TCP connections with no data exchanged N E T G R O U P P O L I T E C N I C O D I T O R I N O 9/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 10/75 What ntop does (3) Possible approaches to NM TCP/IP Stack Verification Network mapping: improper TCP three way handshaking (e.g. queso/nmap OS Detection) Portscan: stealth scanning, unexpected packets (e.g. SYN/FIN) DOS: synflood, invalid packets (ping of death, WinNuke), smurfing IDS/Firewall elusion: overlapping fragments, unexpected SYN/ACK (sequence guessing) Intruders: peak of RST packets Intrusion Detection Trojan Horses (e.g. traffic at know ports) Spoofing: Local (more MAC addresses match the same IP address) and Remote (TTL!) Network discovery (via ICMP, ARP) Viruses: # host contacts in the last 5 minutes (warning: in this respect P2P apps behave as viruses/trojans!) Active The system under monitor is probed periodically with some external signal Passive A probe (silently) collects data and infers some properties from it N E T G R O U P P O L I T E C N I C O D I T O R I N O 11/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 12/75

Active Network Monitoring Passive Network Monitoring Often based on specific traffic / packet patterns, generated specifically for monitoring purposes Usually ICMP packets Sometimes other probes (e.g. TCP connections) Used for: Delay measurement One way, End-to-end Remote devices availability Services Examples RIPE Test Traffic Measurement Service PingER (Ping End-to-end Reporting) at Stanford University nmap The most widely used approach Preferred for its lack of intrusiveness Used for: Traffic measurement, monitoring, characterization E.g. network traffic is examined to generate alerts or statistics E.g. full packet decoding (e.g. for troubleshooting) Status and parameters of network links, network devices, E.g. traffic load on interface, link-layer signals Available technologies Packet-based approach: Packet Sniffing Generic statistics and network status: SNMP Aggregate statistics approach: RMON Flow-based approach: NetFlow, sflow, IPFIX N E T G R O U P P O L I T E C N I C O D I T O R I N O 13/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 14/75 Sniffing Passive NM: packet-based approach Sniffing: architectural choices Fast " Expensive (niche market) " Difficult to move / duplicate " ASIC: cannot be reprogrammed / updated (FPGA can, but it is not very simple) We want to capture exactly the frames that are being transferred on a wire or on some specific network segment Very detailed view (e.g. for debugging) May have limited knowledge of link-layer issues (e.g. Ethernet collisions, ) Very large amount of data to be processed Privacy concerns Performance Hardwarebased Systems Optimized Operating System May be very fast " Requires custom OS Software-based Systems Standard Operating System Cheap Easy to move / duplicate Easily updated " May be slow Very easy to setup (e.g. just install WinPcap) " Rather slow Versatility N E T G R O U P P O L I T E C N I C O D I T O R I N O 15/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 16/75

Sniffing: where to capture traffic (1) Sniffing: where to capture traffic (2) Old Ethernet Shared Ethernet Passive Tap Switched Network Network device-based Mirror port (per port, per port group, per vlan,) Captures everything, even physical signals Precise timestamping " Practical issues (you need an old Ethernet) Captures everything, even part of physical signals Precise timestamping " Some physical signals are not captured (e.g. collisions) " Practical issues (you need a shared Ethernet) Captures everything, even physical signals Precise timestamping " Practical issues (need a tap) " Need a faster interface (2x for tx and rx) Captures all the traffic, even from several ports, even from remote locations (such as Cisco RSPAN) " Requires a dedicated port on the switch " May need faster interfaces (at least 2x for tx and rx) " Timestamps not precise " May be problems for correlating traffic (which port originates this packet?) " Unable to detect link-layer problems Captures all the traffic, even from several ports Precise timestamps Traffic correlation easier " Requires a dedicated port on the device " May need faster interfaces (at least 2x for tx and rx) " Unable to detect link-layer problems " Technology in the early stage, not widely supported - Cisco Catalyst 9000 and some other proprietary examples - RMON is hardly usable - PSAMP is still ongoing N E T G R O U P P O L I T E C N I C O D I T O R I N O 17/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 18/75 What about sniffing in network devices? How not go to jail with Sniffing Difficult to get exactly the wanted packet trace SNMP does not allow packet capture RMON allows packet capture, but only within some standard templates E.g. poor filtering options Cisco NetFlow does not allow packet capture sflow allows packet capture, but it cannot be customized; not widely supported A new header contains the packet; however often key information are missing (e.g. originating interface, ) IETF PSAMP should be helpful Standardization rather show (began in 2000) Requires ad-hoc hardware, otherwise resources are stolen from the router main objective (forwarding and routing) Ascertain compliance with regulatory procedures Check the regulation in your country You can use sniffing for National security To prevent or detect crime To prevent or detect unauthorised use To ensure effective systems operation You have to make sure that: The identity of the sender/receiver cannot be inferred from the captured data Addresses masquerading Aggregate data N E T G R O U P P O L I T E C N I C O D I T O R I N O 19/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 20/75

Passive NM: the SNMP approach Architecture of SNMP components Allows retrieving generic statistics, network status, Not widely used for network configuration (although supported) Defines mechanism for remote management of network devices (routers, bridges, etc.) Fundamental principle: all device management done by simple variable value manipulation Approach: standard means for specifying quantities recognized by devices protocol for requesting, returning, notifying of changes of values An SNMP network consists of three main components: Managed Devices Agents Network Management Systems (NMS) The managed device is a node in the SNMP network and it contains the SNMP agent The NMS makes a virtual connection to the SNMP agent The agent serves the information to the NMS regarding the network status N E T G R O U P P O L I T E C N I C O D I T O R I N O 21/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 22/75 Components of the SNMP world Structure of Manag. Information (SMIv2) Protocol for exchanging data between Agents and Management Entity SNMP Definition of the objects that can be read / modified Must be know on both side (Agents and ME) MIB Syntax used to specify the Management Information Base SMIv2 SMIv2 defines the rules for creating MIBs and it is based on simple typed variables SMIv2 is based on extended subset of ASN.1 (1998) Characteristics of the variables defined by SMI Each variable has an ASN.1 datatype INTEGER, OCTET STRING, OBJECT IDENTIFIER, NULL, It does not implement complex data structures and operations on the variables Variables are either scalars (exactly one instance) or columns in a conceptual two dimensional table (zero or several variables) N E T G R O U P P O L I T E C N I C O D I T O R I N O 23/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 24/75

Management Information Base (1) Management Information Base (2) "The set of managed objects within a system, together with their attributes, constitutes that system's management information base." (ISO 7498-4) MIBs are created using the SMIv2 syntax MIBs are controlled by the SNMP agent The information in the MIB is organized hierarchically MIB consists of managed objects Managed objects that are identified by two names: Object Name Object Identifier MIB have private branches Variables recognized by device supplied in MIB (Management Information Base) text file giving variables and data structures defined using ASN.1 standard variable sets often provided as RFC s device-specific sets provided by vendors Management stations parse MIB s to determine variables available for management obtain both data structure and management information Example -- the Interfaces group ifnumber OBJECT-TYPE SYNTAX INTEGER ACCESS read-only STATUS mandatory DESCRIPTION "The number of network interfaces present on this system." ::= { interfaces 1 } N E T G R O U P P O L I T E C N I C O D I T O R I N O 25/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 26/75 ASN.1 Object Identifiers SNMP Message Encoding Variables identified by globally unique strings of digits Example: 1.3.6.1.4.1.3.5.1.1 name space is hierarchical in above, 1 stands for iso, 3 stands for org, 6 stands for dod, 1 stands for internet, 4 stands for private, etc. Variable names are aliases for digit strings (within MIB) Example: ifnumber ::= { interfaces 1 } interfaces was previously defined in MIB as 1.3.6.1.2.1.2, so: ifnumber = 1.3.6.1.2.1.2.1 Encode message as byte stream using ASN.1 BER (Abstract Syntax Notation 1 Basic Encoding Rules) Quantities encoded as Type, Length, Value triples Types Subset of basic ASN.1 types used in SNMP: integer, octet string, object identifier ( variable name ), sequence SNMP-defined types: gauge, counter, IP address, etc. Values weirdly encoded!! (see ASN.1 specs) N E T G R O U P P O L I T E C N I C O D I T O R I N O 27/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 28/75

SNMP Encapsulation SNMPv1 Protocol UDP Manager Get Agent It can be used for reading one or more variables Agent: port 161 Management Entity: port 162 (for traps) Response Delivery of management information is particularly important in moment of high loss Congestion Improper operation GetNext Response It retrieves the object name and the value of the next instance. This operation is used to discover MIB structures and read tables Using multiple/successive GetNext operations it is possible to read the complete MIB without knowing its structure TCP is not suitable (although supported, particularly for SNMPv3 due to its write operations) Set It writes values in one or more MIB instances Response It is the only operation Agent # Manager; it is an asynchronous event Trap With the trap operation an agent can emit an event and inform a manager. However, the receipt of a trap operation is not acknowledged, thus the message can be lost Therefore, even if traps are used, polling is still necessary (for instance the agent might be down) N E T G R O U P P O L I T E C N I C O D I T O R I N O 29/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 30/75 SNMPv3 SNMP and Network Monitoring SNMP increasingly used for CONTROL In addition to monitor Write operations (SET) SNMPv3 adds security Scarcely deployed, mostly due to security concerns and implementation problems Possibility to capture and create data values from properly targeted and formatted traps; the information gathered using SNMP can be used for network monitoring E.g. packet arrival and departure rates, packet drop rates, packet error rates, system load, modem availability etc. Examples of network monitoring tools: MRTG HP OpenView (not only monitoring) MRTG uses the data collected from SNMP agents to generate graphical representations of it almost real time N E T G R O U P P O L I T E C N I C O D I T O R I N O 31/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 32/75

Some SNMP Issues Passive NM: RMON Often, the most valuable data is exported only through proprietary MIBs Often, units are differents (Kbps for one vendor, bps for another, ) Difficult to manage a multivendor network Cannot add a new MIB within an agent Cannot customize the variable which are needed to monitor the network The opposite (add a new MIB in the Management Station) is pretty simple Defines a remote network monitoring MIB Is an addition to the basic set of SNMP standards Why RMON? With MIB-II the network manager can obtain information that is purely local to the individual devices What about information pertaining to traffic on the LAN as a whole? Collision domain concept Features Is used to passively monitor data transmitted over LAN segments Provides interoperability between SNMP-based management consoles and remote monitors N E T G R O U P P O L I T E C N I C O D I T O R I N O 33/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 34/75 RMON Goals RMON-1 MIB (RFC 1757, RFC 1513) (1) Off-line operation RMON MIB allows a probe to be configured to perform diagnostics even in the absence of communication with the management station Proactive monitoring A monitor can continuously run diagnostics and log network performance. In the event of a failure, the monitor can supply this information to the management station Problem detection and reporting The monitor can be configured to recognize error conditions, continuously check for them and notify the management station in the event of one Value added data A remote monitoring device can add value to the data it collects by highlighting those hosts that generate the most traffic or errors Multiple Managers An organization can have multiple management stations for different units. The monitor can be configured to deal with more than one management station concurrently Not all implementations fulfill all these goals Statistics (1) Contains extent of utilisation and error statistics for the Ethernet and Token Ring network segments. It shows packets, collisions, octets, broadcasts, multicasts, errors, and keeps track of packet size distribution (< 64, 64-1518, > 1518 octets) History (2) Enables to copy periodically the values from the Statistics group into a circular buffer Alarm (3) Implements the monitoring of MIB instances threshold values, based on the ASN.1 datatype INTEGER. An alarm (SNMP Trap) is produced when a threshold is exceeded Host (4) Maintains the association of IP, MAC addresses, bytes sent/received (and more) for the observed traffic N E T G R O U P P O L I T E C N I C O D I T O R I N O 35/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 36/75

RMON-1 MIB (RFC 1757, RFC 1513) (2) RMON-1 MIB (RFC 1757, RFC 1513) (3) hosttopn (5) Analyzes (i.e. sorts) the data entered in the Hosts group Matrix (6) Contains data over communication relations which are defined by pairs by MAC addresses. Useful for what if analysis, and for detecting intruders Filter (7) Used to select individual packets. A filter expression (bit patterns only) assigns packages to a channel. The channel determines whether the packet is only counted or whether an event is produced on packet receipt Capture (8) Provides a scratchpad memory where are stored all the packets received by a channel Event (9) The Event group regulates the handling of internal events: it defines the various events that cause the emission of SNMPv1 traps sent to management applications or be stored in a log. tokenring (10) Historical All the groups on RMON MIB are optional There are some dependencies: The Alarm group requires the implementation of Event group The HostTopN group requires the implementation of Host group The packet Capture group requires the implementation of Filter group N E T G R O U P P O L I T E C N I C O D I T O R I N O 37/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 38/75 RMONv1 vs. RMON v2 RMON-2 MIB (RFC 2021, RFC 2074) (1) RMONv1 has been designed for low level protocols below IP RMONv2 has been designed to monitor high layer protocols RMONv2 extends RMONv1 by adding nine new groups Protocol directory group Describes the protocols detected by the probe including the protocol parameter (e.g. UDP port numbers). All protocols above the network layer are supported (e.g. http, ftp) Protocol distribution group Produces basic statistics for selected protocols (number of byte, number of packages) Address mapping group Provides a mapping of MAC addresses (flown through the probe) in network addresses Network layer host group Provides statistics for the network layer classified according to network addresses Network layer matrix group Supplies statistics for communication relations (host communications matrix) at network level N E T G R O U P P O L I T E C N I C O D I T O R I N O 39/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 40/75

RMON-2 MIB (RFC 2021, RFC 2074) (2) RMONv2 Time Filter Application layer host group Provides statistics for an application layer protocol according to network addresses Application layer matrix group Is similar to Network Layer Matrix group with the exception that in this case statistics are calculated on an application layer protocol layer User history group Permits an automatic generation of statistics stored into so-called Buckets. The number of available buckets is configurable Probe configuration group Enables the configuration of the probe and covers among other things: Configuration of serial access (Modems) IP network configuration Configuration of serial connections (SLIP) for Trap delivery Configuration of parameters for Traps delivery A table can contain a very large number of values E.g. traffic from each host to any other host on the network Retrieving the whole table can be expensive The TimeFilter allows getting only the values that changed after time T (specified in the GET operation) N E T G R O U P P O L I T E C N I C O D I T O R I N O 41/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 42/75 Some RMON Issues Passive NM: Flow-based approaches Implementation of RMON agents and management station is very complex RMON is usually done through ad-hoc blades in high-end network devices Customizability Cannot add new feature to the existing MIBs Often, users need just some simple functions, but they are forced to but expensive equipment to get them done, althoug the most part of the features are useless in their view Not widely used The most part of the data trasfer in a data network involves some transport-layer protocol TCP, UDP The flow-based approach analyzes transport-layer sessions, and uses this data as the basis for the network monitor Flow information IP source, destination Transport protocol Port source, destination Additional fields, not strictly related to the session E.g. IP flags, N E T G R O U P P O L I T E C N I C O D I T O R I N O 43/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 44/75

Mostly used architecture Flow-based NM: characteristics FlowInfo 1.1.1.1(80) $ 2.2.2.2(1081), TCP 1.1.1.1(80) $ 2.2.2.3(2163), TCP Exporter Captures packets, processes them and creates a flow table internally The flow table is (partially) periodically exported to the collector Exporting modes depend on the technology involved Very high requirements in terms of CPU and memory Flow Table TimeFirst 10:12.123 10:12.445 TimeLast 10:15.729 10:19.992 Bytes 88643 11321 Collector Minimal processing requirement Problems may arise if the flow table must be saved for future reference (e.g. in a database) Advantages Reduces the amount of information to process (flow information are smaller than packet information) More scalable Problems Cannot deal with some of the aspects related to packet level E.g. ICMP probes, routing protocols, Most important technologies Cisco NetFlow Uses data (partially) available for CEF (Cisco Express Forwarding) IETF IPFIX sflow N E T G R O U P P O L I T E C N I C O D I T O R I N O 45/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 46/75 Cisco NetFlow Exporting Flows Open standard for network traffic measurement defined by Cisco Systems By far, the most used technology Very small interaction between collector and exporter SNMP may be used to configure the probe and (occasionally) to get data back Data is exported by means of a UDP stream, with proper headers Packet sampling in order to decrease the processing Flows are exported to collector when: the flow ends (e.g. a TCP packet with the FIN or RST bits) the flow has been inactive for a certain period of time, i.e. if no packets belonging to it have been observed for a given timeout (usually 15 sec) the flow is still active, but a given timeout (usually 30 min) is expired; this is useful for exporting long-lasting flows at regular basis the probe experiences internal constraints (e.g. counters wrapping or low memory); in this case, a flow may be forced to expire prematurely N E T G R O U P P O L I T E C N I C O D I T O R I N O 47/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 48/75

NetFlow problems IETF IPFIX Different methods for exporting a flow Makes processing harder Flow records span several bins The concept of bins is not well defined in NetFlow (at least, bins are 30min) The collector cannot now, at time T, which are the flows seen, because some active flows may have not been exported (yet) Targeted for TCP/IP networks only No support for link-layer headers Impossible to add new information (e.g. protocol fields) in the exported flow record Packet Sampling Unsuitable for some kind of applications IP Flow Information Export Basically, NetFlow with the IETF stamp Limited differences Transport protocol (SCTP; optional TCP or UDP) Limited customizability of the fields that are exported within each flow record (e.g.. MPLS label, BGP Autonomous System, ) N E T G R O U P P O L I T E C N I C O D I T O R I N O 49/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 50/75 Realtime Traffic Flow Measurement sflow IETF Working Group (RTFM) Proposal is more advanced than NetFlow Simple Ruleset Language Provides a way to customize flow definition (which can be a generic group of packet with some common characteristics, e.g. the packets from source A to destination B) action (byte count, and more) Flows are bidirectional makes easier to check the two directions of a connection Interaction between probe and collector is done through SNMP queries Probe must store flow records in memory until the collector ask for them Not supported in commercial devices Only the public-domain NeTraMet tool Packet Sampling (like Cisco NetFlow) Can export either: Sampled packets (although limited to the first few hundred bytes) Flow information Excellent technology, but not supported by Cisco N E T G R O U P P O L I T E C N I C O D I T O R I N O 51/75 N E T G R O U P P O L I T E C N I C O D I T O R I N O 52/75

Scalability of the proposed approaches Scalability SNMP RMON Flowbased Packetbased SNMP and RMON show excellent scalability properties But they usually work on traffic aggregate RMON may need to compute more precise statistics (e.g. traffic sent by each host, or traffic matrix) Flow-based and Packet-based are the most critical technologies from this point of view So, let s investigate how to mitigate the problems of flow-based and packetbased technologies N E T G R O U P P O L I T E C N I C O D I T O R I N O 53/75