}w!"#$%&'()+,-./012345<ya

Transcription

1 }w!"#$%&'()+,-./012345<ya MASARYK UNIVERSITY FACULTY OF INFORMATICS Network Traffic Collection with IPFIX Protocol MASTER S THESIS Radek Krejčí Brno, 2009

2 Místo této strany bude vložena kopie zadání. i

3 Declaration Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Advisor: Ing. Pavel Čeleda, Ph.D. ii

4 Acknowledgement I would like to thank Pavel Čeleda for his valuable advice and great help when supervising this thesis. I also have to thank all people participating on the Liberouter project and especially its FlowMon group. iii

5 Abstract In this thesis I analyze current needs and possibilities of IP flow network monitoring. I compare usability of currently most widely used NetFlow protocols with new IPFIX protocol, which was recently prepared as a common successor of all previous proprietary flow export protocols. As part of this comparison, I describe available solutions for collecting flow data. Finally I implement an IPFIX collecting process as integrated part of the NfSen/NFDUMP open source collector tool set. The presented implementation is prepared for the GNU/Linux operating system under BSD license. Particular implementation challenges as well as practical experiences with the proposed solution, tested with the hardware-accelerated FlowMon passive monitoring probe, are also discussed. iv

6 Keywords IPFIX, NetFlow, IP flow, NFDUMP, NfSen, network monitoring, collector, Liberouter, FlowMon, COMBO v

7 Contents Contents vii List of Figures viii 1 Introduction IP Flow Monitoring FlowMon Probe Flexible FlowMon Probe NetFlow Protocols NetFlow Version NetFlow Version NetFlow Version NetFlow Version NetFlow Version NetFlow Version Flexible NetFlow IP Flow Information Export (IPFIX) Protocol IPFIX Export Packet Format Transport Protocol Packet Sampling Information Timestamps Available IPFIX Collectors AURORA ntop SiLK NfSen/NFDUMP Collector NFDUMP NFDUMP tool set overview NfSen NFDUMP Binary File Format NfSen/NFDUMP Modifications for IPFIX Processing Overall System Changes FlowMon Exporter Extension IPFIX Processing Notes vi

8 6.3.1 Transport Protocols Timestamps Sampling Using and Testing IPFIX Collector SCTP Support IP Flow Monitoring Use Cases Accounting/Billing Network Monitoring and Security Analysis Top Statistics Pattern Matching Profiling and Network Planning Network Maps and Data Path Reconstruction Conclusion Bibliography A NetFlow Export Packet Format A.1 NetFlow Packet Version A.2 NetFlow Packet Version A.3 NetFlow Packet Version A.4 NetFlow Packet Version A.5 NetFlow Packet Version A.6 NetFlow Packet Version B NFDUMP File Format Records B.1 Common Record Structure B.1.1 Extension B.1.2 Extension B.1.3 Extension 2 - Input Packet Counter B.1.4 Extension 3 - Byte Counter B.2 Optional Record Extensions B.2.1 Extension 4, 5 - Interface Record B.2.2 Extension 6, 7 - Autonomous System Record B.2.3 Extension 8 - Multiple Fields Record B.2.4 Extension 9, 10 - Next-Hop IP Address Record B.2.5 Extension 11, 12 - BGP Next-Hop IP Record B.2.6 Extension 13 - VLAN Record B.2.7 Extension 14, 15 - Out Packet Counter Record B.2.8 Extension 16, 17 - Out Byte Counter Record B.2.9 Extension 18, 19 - Aggregated Flows Record B.2.10 Extension 20, 21 - MAC Record B.3 Extension Map Record Structure C Content of the CD-ROM vii

9 List of Figures 1.1 Scheme of using network router to report traffic statistics Scheme of a passive network monitoring probe connection COMBO hardware accelerator Example of NetFlow v9 export packet layout Options Template example NetFlow v9 Format Scheme IPFIX export packet layout The form of IPFIX information elements Systematic count-based sampling Systematic time-based sampling Random sampling Comparison of export packet headers in NetFlow v9 and IPFIX protocols AURORA GUI with a graph of application utilization ntop traffic statistics Scheme of collecting and processing flow data by NFDUMP tools Flowchart of flow data processing in nfcapd(1) Example of NFDUMP standard output Homepage of the NfSen interface NFDUMP file layout Data block format 2 layout Modifications of NFDUMP tool set Scheme of DDoS attack using IP spoofing Network map generated by Scrutinizer 45 viii

10 Chapter 1 Introduction Computer networks become a key part of the business processes in most companies. They are one of the critical resources needed for business success of the company. Therefore understanding what is happening on the network or how the network is used and which threats can compromise its features and services is necessary for further effectiveness of the network as well as entire company. Satisfaction of these requirements is achieved by combining several network monitoring tools including intrusion detection/prevention systems (IDS/IPS), active network monitors, periodically checking availability of key network services, and last but not least the monitoring devices watching network traffic in a real-time. Acquired data can be then used as a reason for operator s immediate responses (to e.g. security threads or to satisfy regular users needs) as well as for later complex data analysis utilizing big amount of collected historical data. Figure 1.1: Scheme of using network router to report traffic statistics Sometimes network monitoring is thought as a synonymous for SNMP (Simple Network Management Protocol). But SNMP is able to give only general network traffic statistics. Network administrator is unable to retrieve information needed e.g. for billing purposes or to identify potentially dan- 1

11 1. INTRODUCTION gerous traffic on the network. Current needs for network traffic information are about to know who is (has been) communicating with whom, how, when, for how long and how much data is transferred. And SNMP is not sufficiently suitable for these needs. Situation on high speed networks is usually much more critical, because SNMP statistics are retrieved from active network elements like routers. But especially on high speed networks this is a quite bad idea. These devices should do their work (and they are usually really busy with it) and nothing else. In these situations the independent device, passively retrieving network traffic information (fig. 1.2), is more suitable solution than using active network device for reporting traffic information (fig. 1.1). Besides the combination of passive connection to the monitored link and dedicated link for exporting flow records enables probe to stay absolutely invisible for devices in the monitored network and avoid possible attacks on the probe. Figure 1.2: Scheme of a passive network monitoring probe connection Monitoring devices can be used as necessary base for several more appliances. It is possible to use data collected by these devices for the accounting purposes as well as for further improving and extending provided network services. Data can be also successfully used for planning future development of the network. User demands for increased bandwidth or other network resources and services can be solved by overdesigning (and further extending) network resources or by improving network performance and effectiveness. The second solution can present an additional cost for monitoring devices which doesn t make a profit directly. On the other hand and in the light of the economic situation of past months this approach offers a way to higher cost effectiveness of investment spent on currently built network infrastructure. 2

12 1. INTRODUCTION Furthermore, investments in monitoring framework are also needed in some countries (including Czech Republic) to satisfy local laws requiring retention of traffic data for specified time. These requirements are mainly aimed to facilitate judicial co-operation in criminal matters. In the states of EU this area is covered by the data retention directive [11]. This directive obliges EU Member States to adopt legislative requiring retention of data, necessary to identify source, destination, type, date, time and duration of a communication, for periods not less than six months and not more than two years from the date of the communication. All described requirements can be satisfied by IP flow monitoring process. 1.1 IP Flow Monitoring Network traffic monitoring based on a concept of IP flows was developed by Cisco Systems, Inc and widespread as Cisco NetFlow technology [21]. This proprietary protocol for collecting IP traffic information was widely accepted as a de facto standard and adopted in devices of other vendors (sometimes with different names and minor changes like Jflow from Juniper Networks). The general definition of the IP flow describes it as a stream of IP packets with defined common properties. If another observed packet has at least one different property than previous packet(s) it belongs to a new flow. There are several definitions what are those common properties. For the original (non-flexible) NetFlow protocols, this set of common properties defining a unique flow consists of these key fields: Source IP address Destination IP address Source port number (only for TCP or UDP, zero for other protocols) Destination port number (only for TCP or UDP) or type and code for ICMP and zero for other protocols Layer 3 protocol type Type of Service (ToS) Input logical interface (ifindex) 3

13 1. INTRODUCTION Sometimes the last two fields are not observed as important and a unique flow is defined only by first five key fields. Besides key fields, the flow record also contains non-key fields containing additional information. Values of these fields are taken from the first packet of the flow and they are not used to decide packet assignment into the flow. Typical non-key fields include: Flow timestamps to understand the life of a flow (and used to calculate packets and bytes per second) Next hop IP addresses including BGP routing autonomous systems Subnet mask for the source and destination addresses to calculate prefixes The last type of the flow record fields can be items with information about whole flow such as packet or byte counters or TCP flags. Value of TCP flags field is stored as a result of bitwise OR of TCP flags taken from all packets in a flow. It can be used to examine TCP handshake and session termination. The last NetFlow v9 protocol [6] came with a concept of flexibility using templates to define all flow fields directly by user. This conception is followed (and extended) by IPFIX (IP Flow Information export) protocol [4] supposed to be new common standard for IP flow information data. This approach is described in detail in the following two chapters. 1.2 FlowMon Probe The FlowMon probe [24] is a passive network monitoring device developed as part of the Liberouter project [25]. Liberouter project is a programmable hardware activity of CESNET (Czech National Research and Education Network operator). With own developed COMBO card family [23] utilizing technology of FPGA (Field-Programmable Gate Arrays). Liberouter profits from hardware/software co-design approach. Time critical operations are performed by COMBO card and other operations that can wait or that are too complicated for hardware are performed by software application. The FlowMon probe is a network device fully satisfying needs of highspeed network monitoring, described in first paragraphs of this thesis. It passively observes network traffic passing its interface(s), completes flow records information and exports them to the collector(s). The device can 4

14 1. INTRODUCTION Figure 1.3: COMBO6X with COMBO-2XFP2 add-on card with two XFP cages for 10GE and one SFP cage for 1GE interfaces be used on all networks from 10 Mbps to 10 Gbps. According to hardware/software co-design the hardware card of the FlowMon probe only parses packets incoming from high-speed link(s). Acquired information is further passed to the software application that completes export protocol packets and sends them to a collector. Current version of an exporting application is able to send flow information in NetFlow format v5, v9 and in new IPFIX format. This solution, developed by Liberouter project, was lately adopted and modified for commercial needs by spin-off company INVEA-TECH a.s. [1] Flexible FlowMon Probe The Flexible FlowMon probe concept further develops original idea of the FlowMon probe. It again gains maximum from the hardware/software codesign but improves original concept by enabling user to define what information is observed in the network traffic (by COMBO card). It actually follows changes made in the flow export protocols (NetFlow v9 and IPFIX) that also become more flexible. Flexible FlowMon probe is currently still under development but first functional versions are already available. 5

15 Chapter 2 NetFlow Protocols Cisco NetFlow technology is embedded feature of the Cisco IOS (Internetwork Operating System) devices. It is used to record and deliver flow information from a measurement device to a collector for further data analyses. Originally the NetFlow was developed as a caching technique of Cisco routers to speed-up routing. The side effect of using these caches, based on a flow information, was a possibility to collect useful network statistics. Net- Flow become valuable tool of network administrators to understand what is happening on the network. It is a beneficial tool for analyzing network traffic, used for wide range of application from analyzing impact of new network services through security and anomaly detection to QoS (Quality of Services) control. NetFlow equipped device processes all incoming packets and compares its key flow properties with active flow records stored in device s cache to determine if the packet is a continuing of any previous flow or a beginning of a new one. Each flow record contains items common for all packets of the flow so these values are stored as soon as the first packet of a flow appears and they don t change ever. But flow record also contains non-key items, with values obtained from the first packet of the flow, and items describing flow as a whole whose content must be updated with any incoming packet belonging to the flow. The NetFlow data observed by measurement device are sent to the collector device as soon as a flow records expire. Flow expiration can arise for the following four reasons: Any new packet of the flow didn t appear in a specific time (inactive timeout). Flow is long lived and it is expired before its end due to active timeout. Flow cache becomes full (or some other internal constraint appears) so some flow records are expired prematurely. Measurement device detects regular end of a flow (TCP packet with end of byte stream (FIN) or reset (RST) flag). 6

16 2. NETFLOW PROTOCOLS Cisco devices usually set 15 seconds for inactive and 30 minutes for active timeout. FlowMon probe default values are 10 seconds for inactive and 5 minutes for active timeout. All these timeouts are configurable. Expired flow records are completed into the export packet in the form of the specific version of the NetFlow protocol. The general form of the export packet is the same for all NetFlow versions. Each packet begins with header carrying general information about its content. Header of the export packet begins with NetFlow version identifier followed by count of flow records (or flow sets in v9) enclosed in the export packet. Next header fields as well as its order differ in specific versions, but in general they contain information about packet export time including export device system uptime. Beginning with version 5, header also includes a flow sequence number to allow detection of lost packets sent from exporting device. Packet loss can happen due to using unreliable transport over UDP (User Datagram Protocol). Full description of the export packet form in all NetFlow versions can be found in appendix A. Cisco NetFlow protocol is available in several versions. The last version is labeled as NetFlow v9. But not all previous versions are used in real. Versions 2, 3 and 4 were not released ever and versions 1 and 6 are used scarcely ever. On the contrary, versions 5 and 9 achieved wide acceptance and they are used by the most of current NetFlow devices. Here follows a description of all NetFlow versions. 2.1 NetFlow Version 1 It is an initial NetFlow version, which is rarely used today. 2.2 NetFlow Version 5 The most widely used NetFlow version today. It extends version 1 to contain information about source and destination autonomous systems (AS). The second significant change is an addition of the flow sequence number into the export packet header. This number can be used by collecting application to detect and report count of lost flow records. NetFlow protocols use unreliable UDP as its transport protocol so any packet loss detection must be done by target application. NetFlow v1 didn t enable this detection but v5 changed that. The loss detection is done by sequence checking. The sequence number in the currently received packed should be equal to the sum of previous packet sequence number and flow count in the previous packet. Nonzero value of this comparison matches count of lost flow records (equation 2.1). 7

17 2. NETFLOW PROTOCOLS lostrecords = curseqnumber (prevseqnumber + prevrecordcount) (2.1) NetFlow v5 provides simple and clear way how to get sight into the network. It is able to give answers to questions like who is (has been) communicating with whom, how, when, for how long and how much data they are transferring. So NetFlow v5 covers the most network trouble shooting situations when administrator have to identify, trace-back and react to problems like security threads or inappropriate traffic consuming bandwidth. But sometimes it would be useful to be able to change what information is provided by measurement device. For these situations Cisco prepared last NetFlow v9 (see section 2.6). 2.3 NetFlow Version 6 Not supported version today. It was kind of development version not released in public. In general this version is very similar to version NetFlow Version 7 In comparison to previous versions, this one adds NetFlow support for Cisco Catalyst 5000 series switches. 2.5 NetFlow Version 8 Thanks to added router-based aggregation schemes, this NetFlow version enables to summarize NetFlow data before exporting them from a measurement device. This approach can reduce bandwidth requirement. On Cisco routers this feature is implemented by set of specialized aggregation caches. Using this set of aggregation schemes is actually the first step on the way to the flexible flow export protocol because each aggregation cache detects flows according to different key fields. 2.6 NetFlow Version 9 NetFlow v9 continues on the way to the flow export flexibility started in the version 8. NetFlow v9 introduces template-based approach enabling future adding of new flow record fields without any impact to the protocol and export packet structure. In all previous NetFlow versions adding new field implied a change in the structure of the export packet and new NetFlow version. 8

18 2. NETFLOW PROTOCOLS Templates contain structural information about flow record fields. They are used by collector to determine where and what information is stored in the received flow record. Thanks to this flexibility, NetFlow v9 newly supports IP addresses in both IPv4 and IPv6 formats. If collector doesn t understand semantics of some field (possibly new defined or proprietary field types), collector is able to skip such field and continue with processing of followed known field type. This feature makes protocol resistant to incompatibilities in supported field sets between exporters and collectors. The flexibility of the template approach enables reducing exported data volume and possible disk space savings for collectors. NetFlow v9 export packet begins by packet header (see A.9) but followed by one or more FlowSets (fig. 2.1) instead direct flow records as in previous NetFlow versions. Figure 2.1: Example of NetFlow v9 export packet layout FlowSet is one of these three types: Template FlowSet NetFlow v9 templates provide protocol flexibility and extensibility. Template FlowSet contains concrete templates with description of data records structure in Data FlowSet. Template s fields define type and length (and therefore a position) of an appropriate field in corresponding Data FlowSet identified by Template ID. Full description of this FlowSet type is available in appendix table A.12. Options Template FlowSet This special kind of template set represents a way how exporter can declare that some field values are valid for all records within specified scope. Usually, this information describes some behavior of exporting device like sampling parameters (fig. 2.2) or switching engine. The scope represents a part of the exporter to which the option fields refers. The scope can be System, Interface, Line Card, Cache or Template. For the full description of this FlowSet type, see appendix table A.13. 9

19 2. NETFLOW PROTOCOLS Figure 2.2: Example of Options Template with sampling parameters common to specified interface Data FlowSet This FlowSet type actually represents the export packet payload from previous NetFlow versions. Data FlowSet contains a group of flow records referring to the common template or options template that describes the structure of these records. Full description of this Flow- Set type can be found in appendix A.14. All three FlowSet types can appear after export packet header in any order. Export packet can contain only template FlowSets, only Data Flow- Sets or any combination of both (fig. 2.1). Collector identifies FlowSet type according to FlowSet ID at the beginning of each FlowSet. All templates must be stored by collector and used for Data FlowSets referring them by template ID. There is also one discrete change in the export packet header that must be mentioned. Flow sequence field known since NetFlow v5 is changed to package sequence field representing number of all export packets sent by the export device. It s hard to evaluate strengths and weaknesses of the static and dynamic approaches presented by the old-style NetFlow (v5 like) and new NetFlow v9, because it truly depends on specific user demands. The strength of the static approach is in its simplicity. Each export packet consists of the packet header followed by 1-30 flow record(s) in exactly same length and form. Therefore exporting as well as collecting processes can be really simple and they haven t much to do. But sometimes the static form can be con- 10

20 2. NETFLOW PROTOCOLS Figure 2.3: NetFlow v9 Format Scheme sidered as a disadvantage. NetFlow v9 enables to define content of the flow records so user is able to dynamically change what information is exported from a measurement device. But this feature is on the other hand reflected in increased requirements to the exporting and collecting applications to be able to process user defined form of the flow records Flexible NetFlow Considering vital changes in NetFlow v9 Cisco has been forced to modify the traditional static NetFlow technology inside its devices. This nextgeneration flow technology is labeled as Cisco IOS Flexible NetFlow. It is a 11

21 2. NETFLOW PROTOCOLS feature of the Cisco IOS Software enabling user to take advantage of the NetFlow v9 facilities. Flexible NetFlow is allowed to track network packet information from layer 2 to layer 7 according to user specifications. User is allowed to export limited part of a packet payload. The same idea of the Flexible NetFlow is shared by the Flexible Flow- Mon (see section 1.2.1) developed by Liberouter. 12

22 Chapter 3 IP Flow Information Export (IPFIX) Protocol The IPFIX protocol is supposed to be a common standard of exporting IP flow information from network measurement devices to collectors that are processing and storing these data. It has been prepared by IETF s (Internet Engineering Task Force) IP Flow Information Export Working Group [15] as a successor of various proprietary protocols. Preparation of the IPFIX protocol started in 2002 and final specification was released in January 2008 as RFC 5101 [4]. IPFIX Working Group is currently still concerned with several topics related to IPFIX, such as configuration model for IPFIX devices or a framework for IPFIX mediation. In the beginning of the IPFIX development, working group defined its requirements for new protocol in RFC 3917 [18]. It contains e.g. demand for reliability of the metering process, which means that each packet passing observation point must be recorded. If metering process is not able to process each packet, it must be able to detect such situations and report them. Reliability in conjunction with security requirements and congestion awareness are included in requests for the data transfer. Important requirement, for the new protocol, was also extensibility of the data model concerning to possible future changes in the data model without necessity to change protocol itself. On the basis of these requirements, IPFIX Working Group had been examining applicability of several candidate protocols. The result of this process was the recommendation to use NetFlow v9 protocol as a base for new IPFIX protocol [16]. Following sections describes main protocol differences between Net- Flow v9 and IPFIX. 3.1 IPFIX Export Packet Format Based on NetFlow v9, IPFIX assumed general structure of its export packet. A slightly changed header is followed by any combination of three types of FlowSets Template FlowSet, Options Template FlowSet and Data FlowSet. 13

23 3. IP FLOW INFORMATION EXPORT (IPFIX) PROTOCOL Figure 3.1: IPFIX export packet layout Templates An endeavor to allow operator flexibly react to immediate needs lead in NetFlow v8 to the introduction of aggregation schemes. NetFlow v9 improved this approach and began to use templates. They allow defining a collection of fields which are exported from the observation point. It actually separates exporting protocol from the information model of carried data. Therefore new fields can be defined without any impact to the protocol itself. The IPFIX protocol continues in using templates and evolves them. NetFlow v9 generally enables changes in its information model and definition of entirely new fields. Unfortunately these changes could be done only by Cisco or some (limited number) vendor proprietary fields could be used for these purposes. But in that case, using collector and exporter with different understanding of the same vendor proprietary field can lead to misunderstanding of collected data. Figure 3.2: The form of IPFIX information elements The IPFIX improves template approach in this point of view. Global IP- FIX information fields are still administered centrally, in this case by IANA (Internet Assigned Numbers Authority [14]). But IPFIX also comes with the possibility of defining enterprise-specific information elements so any organization is allowed to define its own field specifications. To guarantee that the field is globally unique, the information element ID (which is sufficient for global information elements on its own) is combined with the enterprise identifier to be able fully identify enterprise-specific fields. These enterprise identifiers conform to IANA administered SMI (Structure of Management 14

24 3. IP FLOW INFORMATION EXPORT (IPFIX) PROTOCOL Information) Network Management Private Enterprise Codes (available at Transport Protocol The NetFlow protocols use primarily UDP (User Datagram Protocol) as its transport protocol. Although UDP enables preserving efficiency while exporters handle high volumes of export packets, it is not able to meet IP- FIX requirements for the data transfer (reliability, security and congestion awareness). Therefore any compliant IPFIX implementation must allow using SCTP (Stream Control Transmission Protocol) with the PR-SCTP (Partially- Reliable SCTP) extension. The TCP (Transmission Control Protocol) support may be also implemented. Exporters must be able to send data to multiple collectors using different transport protocols. 3.3 Packet Sampling Information Sampling is a statistical method of selecting suitable sample of the population for further study. Selected sample is then used to get information that can be generalized to the whole population. In the context of the IP networks monitoring, sampling is used by monitoring devices to select subset of packets out of all watched packets. The information carried by such sample still describes the whole traffic, but monitoring device reduces this way its system load. This method is usually used as a reaction to the situation when the monitoring device is not able to process all incoming packets. For analysis of measured data, the information about sampling is critical. The NetFlow protocol is able to provide only basic information about sampling. Therefore sometimes sflow [20] is used besides traditional Net- Flow to allow devices conserving its resources. Analogously to NetFlow, sflow is a network traffic reporting technology, but data obtained from sflow equipped device are based on packet sampling. IPFIX protocol is not concerned with sampling at all. It even doesn t support sampling information fields defined in the NetFlow v9 (fields number 34 and 35) which are marked as RESERVED in the IPFIX protocol. In the IPFIX point of view the packet sampling questions are solved by the Packet Sampling Protocol (PSAMP [5] [10]) extending IPFIX information model for information related to the sampling process. 15

25 3. IP FLOW INFORMATION EXPORT (IPFIX) PROTOCOL Therefore, in comparison to the NetFlow v9, sampling process performed by monitoring devices can be described in more details. PSAMP defines information elements describing several types of the packet sampling. Systematic count-based sampling each N th packet is selected. Figure 3.3: Systematic count-based sampling for N = 5 Systematic time-based sampling each packet appeared after N milliseconds is selected. Figure 3.4: Systematic time-based sampling Random n-out-of-n sampling n random packets out of set of N packets are selected. Figure 3.5: Random sampling for n = 1 and N = 5 Uniform probabilistic sampling for each packet a sampling probability is calculated and if it is higher than specified limit, packet is selected. Property match filtering observed packet is selected if a specific field in the packet equals a predefined value. Hash based filtering fields from packet IP header are used as input for the hash function. In case the result of a hash function fall into specified range, packet is selected. Thanks to more detailed packet sampling description, the flow analysis is able to interpret data in a much more proper way. 16

26 3.4 Timestamps 3. IP FLOW INFORMATION EXPORT (IPFIX) PROTOCOL The main difference between NetFlow v9 and IPFIX export packet headers is a missing sysuptime field in the IPFIX header (fig. 3.6). The challenge is that NetFlow v9 timestamps (of capturing the first or the last packet of the flow) have a value relative to the device boot time. So, collectors have to recalculate timestamps values using sysuptime and UNIXSecs fields. Then the time values can be displayed in a standard form (since epoch, i.e. 00:00 UTC, January 1, 1970). absolutet ime = bootrelativet ime + (UNIXSecs sysupt ime) (3.1) Unfortunately, this approach can cause increased system load on both exporters (metering/exporting processes) and collectors. And on the exporter side this feature increases possibility of packet dropping. Figure 3.6: Comparison of export packet headers in NetFlow v9 and IPFIX protocols The IPFIX protocol has removed sysuptime field from the header to give exporters more freedom in using timestamps. It enables selection of the most suitable timestamp form for exporters and operators. This way, exporters using absolute time as its internal format are allowed to export timestamps without any conversion. The same thing applies for devices working with the system uptime in relative time format, which is used in NetFlow protocols. In a most cases operator can also choose a time precision best corresponding to his needs. On the other hand collectors have to be prepared to handle all possible alternatives. This is a list of IPFIX information elements that can be used to define flow start and flow end timestamps. flowendsysuptime, flowstartsysuptime These are NetFlow v9 compatible information elements containing timestamp (in milliseconds) of capturing the last/first packet of the flow relatively to the last IPFIX device (re)initialization. 17

27 3. IP FLOW INFORMATION EXPORT (IPFIX) PROTOCOL flowstartseconds, flowendseconds These are IPFIX information elements containing absolute timestamp when the first/last packet of the flow was captured. IPFIX also defines other flowstart /flowend information elements for different time units (milliseconds, microseconds and nanoseconds). flowstartdeltamicroseconds, flowenddeltamicroseconds These fields contain negative time offset of the first/last observed packet of the flow relative to the value of the Export Time field from the export packet header. systeminittimemilliseconds It is IPFIX information element to replace NetFlow v9 sysuptime export packet header field. It is needed to calculate absolute time value from flowendsysuptime and flowstartsysuptime information elements. 18

28 Chapter 4 Available IPFIX Collectors Final specification of IPFIX protocol is still new and it is not widely supported by collectors. But there are several solutions that declare IPFIX support. This chapter describes some of these solutions and level of IPFIX implementation. Liberouter s FlowMon probe was used as a source of Net- Flow and IPFIX data during tests of all following tools. 4.1 AURORA Name AURORA - Network Performance Analyzer Version (release ) Developed by IBM Corp. Homepage AURORA is commercially available research project focused on network traffic analysis and visualization. AURORA is supported on Linux systems and it should be able to collect data using all NetFlow protocols, sflow and IPFIX. AURORA provides web-based user interface (UI). AURORA is really a visualization tool. Its UI provides many different types of charts with nice features like zooming. Also sophisticated user management system and status display with detailed information must be appreciated. On the other hand, AURORA is intended for high speed networks, but UI sometimes seems to have problems generating charts from higher volume of flow data. AURORA declares DNS address resolving, but in all lists containing IP addresses only numeric address form is displayed. Domain names are displayed only in a detail views of specific flow. But really frustrating problem of UI is refreshing and (non)displayed time range of a flow. UI sometimes displays obsolete charts (e.g. half an hour old) and even UI s refresh button is not able to regenerate charts from current data (already successfully stored on the disk). The next problem is that UI doesn t display start and end time of the flow. This information is not included in a flow details either. Further data processing also seems to be a challenging issue due to missing export from the database. 19

29 4. AVAILABLE IPFIX COLLECTORS Figure 4.1: AURORA GUI with a graph of application utilization As AURORA declares, it supports NetFlow protocols (NetFlow v5 and NetFlow v9 were tested). But IPFIX data exported by FlowMon probe are received by capture daemon (supporting UDP, TCP as well as SCTP transport protocols), stored into buffers and disk files, but not displayed in UI. Due to closed AURORA s source codes, it is hard to say what is wrong. 4.2 ntop Name ntop - network top Version Developed by Luca Deri Homepage ntop is an open-source network traffic probe able to observe network traffic (using libpcap) and collect captured data as well as collect data received from remote network probes. ntop runs under nix systems as well as Win32 and supports NetFlow protocols, sflow and IPFIX. Due to ntop s ability to work as a local probe running on a common PC, it is generally used for non-high speed networks. ntop provides quite simple and really intuitive web based user interface. The main advantage of the UI seems to be its interactivity. DNS resolving 20

30 4. AVAILABLE IPFIX COLLECTORS Figure 4.2: ntop traffic statistics is performed automatically to all IP addresses. User is also able to locate hosts through Geotool service with one simple click. Really useful feature is a data dump enabling flow data export in several formats including plain text or perl, php, python and xml structures, that can be accessed online (using e.g. wget(1)) by external third-party applications. ntop is primarily a local network probe. Processing flow data from remote devices is available as a set of plugins. Plugin for processing IPFIX is integrated to a NetFlow processing plugin. ntop is able to listen for NetFlow and IPFIX packets on UDP and SCTP ports, TCP is not implemented. IP- FIX packets are processed in a same way as NetFlow v9 data. It means that ntop is able to process only information stored in a NetFlow v9 compatible information elements. In addition, ntop defines its own specific NetFlow v9 extensions (e.g. for VoIP information [9]), that are unfortunately defined as fields numbered equally to some IPFIX information elements. This means, that if ntop receives e.g. IPFIX information element with ID 130, it ll be represented as a SIP_CALL_ID instead of correct exporteripv4address information element. Non-compatible fields to NetFlow v9 format are not implemented at all. During testing ntop features, there were some problems with enabled NetFlow plugin (several Segmentation faults). So NetFlow and IPFIX support is not on a sufficient level and should be improved in a future. 21

31 4. AVAILABLE IPFIX COLLECTORS 4.3 SiLK Name SiLK - System for Internet-Level Knowledge Version Developed by CERT Network Situational Awareness group Homepage SiLK is a part of open source CERT NetSA (Network Situational Awareness) Security Suit for monitoring networks using flow data. SiLK serves as a collector storing flow data from a variety of observation points. The main idea of SiLK is based on a cooperation between plenty of little commandline utilities, each doing a part of a complex job. This approach is useful for network operators who are able to write own scripts for frequently repeated tasks. On the other hand, it is quite difficult to familiarize with a huge set of tools and to start using it effectively. The collecting process is provided by a group of programs performing conversion between flow export protocols and a specific SiLK file format for long-term data storage. Collected data can be then resent (yet in SiLK format) to another host for further analysis. For reading flow export data, SiLK uses libfixbuf [2] library able to process NetFlow v5, NetFlow v9 and IPFIX. Other NetFlow versions are also processed, but transformed into the NetFlow v5 format and only common fields are used. IPFIX processing is limited to non-flexible format of SiLK files. So, despite of libfixbuf s ability to process any IPFIX information element, including newly defined enterprise information elements, SiLK stores only a discrete set of flow information. Depending on SCTP support in kernel, SiLK is able to retrieve flow data through UDP, TCP and SCTP transport protocols. SiLK tools provides only command line interface. Further analysis and visualization can be done using other tools from the CERT NetSA Security Suit. Unfortunately a big system complexity and its quite wide dependency on third party tools make using the suit fairly difficult. 22

32 Chapter 5 NfSen/NFDUMP Collector The NfSen/NFDUMP collector refers to a several open source tools. NF- DUMP tools serve as collector backend receiving, processing and storing IP flow information. Data can be received through different flow export protocols. NfSen is used as an easy to use user friendly web-based graphical frontend. All tools are developed by Peter Haag, network security engineer at SWITCH-CERT. Within SWITCH-CERT he is in charge of incident handling, computer forensics, malware analysis and security tools design. With Peter s real work practices, NfSen/NFDUMP collector is developed with aim for usability and stability needed by network professionals. 5.1 NFDUMP NFDUMP refers to a group of cooperating independent programs (according to "do one thing and do it well" philosophy) to collect and process IP flow data received from exporting devices. NFDUMP tools currently support sflow [20] and NetFlow version 5, 7 and 9 protocols. Figure 5.1: Scheme of collecting and processing flow data by NFDUMP tools 23

33 5. NFSEN/NFDUMP COLLECTOR NFDUMP tool set overview nfcapd(1) This is one of two main backend programs. nfcapd(1) is used to capture incoming NetFlow data, process them and store flow information to the NF- DUMP files. These files are automatically rotated and renamed after specified amount of time (typically after 5 minutes) to enable better and faster further work with the stored data. Figure 5.2: Flowchart of flow data processing in nfcapd(1) 24

34 5. NFSEN/NFDUMP COLLECTOR nfcapd(1) serves as a typical network server daemon listening on specified port (by default on port number 9995). Daemon is used for listening on not only unicast addresses attached to a host interfaces, but it is also able to join multicast group for listening. On the other hand, it can be also used as a packet repeater resending incoming packets to another host. NetFlow data are stored in the proprietary NFDUMP extensible file format which is described in the section 5.3. sfcapd(1) sfcapd(1) is an analogy of the nfcapd(1) for processing IP flow information exported in sflow format. Received sflow data are also stored in NFDUMP binary file format, which is independent of the source format of the flow data. nfdump(1) Together with nfcapd(1), nfdump(1) makes up a base of the NFDUMP tool set. It is used to display and analyze stored IP flow information. Stored flow records are processed according to given filtering options based on the Berkeley Packet Filter (BPF) syntax, which is used also by tcpdump(1). It is also possible to use nfdump(1) to filter specific information from flow records stored by nfcapd(1) and then resave these data in the NFDUMP file format for further processing. Due to decreased size of data files, this approach can lead to a significant speed up of time critical processing. Figure 5.3: Example of NFDUMP standard output Program output can be displayed in several formats from the most detailed raw format displaying all record information through a pipe format suitable for further processing to a simple line format with the most common information. User is also able to define own output format by specifying required fields to display. nfdump(1) program can be used not only to display and filter stored flow information but it is also able to calculate statistics across multiple data files or to sort flow records. This feature is e.g. 25

35 5. NFSEN/NFDUMP COLLECTOR used to identify a host(s) causing congestion of the network link by displaying the top n IP addresses with the biggest amount of incoming/outgoing traffic. nfdump(1) is also directly used by NfSen frontend to access detailed flow information and statistics. More detailed information about filter syntax or program output formats are available in the nfdump(1) man page. nfprofile(1) nfprofile(1) program connects NFDUMP with NfSen. It reads NFDUMP files and creates corresponding input files for every NfSen source channel. These output files are then used by NfSen (through the RRD tools) to create network traffic graphs. nfreplay(1) This utility is used to resend flow data, previously collected by NFDUMP collector backend, through the selected NetFlow protocol. It is a kind of exporter program that doesn t measure network traffic on a physical network interface but resends information previously measured and exported by another flow exporter. nfexpire(1) nfexpire(1) manages collector data stores. It is used to set expiration limits for nfcapd(1) auto expiration mode or to expire (and delete) data files directly. Limits can be set as maximum disk space used by all files in specified data directory or as a maximum lifetime of these files. Both limit types can be set and files are removed when any of specified limits is reached. The nfcapd(1) auto expiration mode means that nfcapd(1) automatically checks expiration limits after each capturing cycle and expires data files reached any of the limits. To enable auto expiration mode, the nfcapd(1) must be started with -e option. Another way, how to automatically expire collected data, is to start nfexpire(1) periodically by cron(8). ft2nfdump(1) This is a program to convert NetFlow data stored by flow-tools [12] to the NFDUMP file format. This tool is only optional and it is built and installed only with -enable-ftconv option given to configure program during the installation process. $./configure --enable-ftconv 26

36 5. NFSEN/NFDUMP COLLECTOR 5.2 NfSen NfSen is a graphical web based frontend for the NFDUMP tools. The NfSen interface is what user in most cases see and work with while using Nf- Sen/NFDUMP collector. It preserves advantages of command line based tools using directly nfdump(1) program but in addition it gives easy to understand graphical overview of the network utilization. Figure 5.4: Homepage of the NfSen interface NfSen allows operator to easily navigate through graphs of flow information (showing amount of flows, packets or bytes) over time. Graphs simplifies identification of suspicious traffic for further analysis. NfSen contains a feature to create alerts. Operator can specify a filter that is applied to collected flow information. Satisfaction of defined conditions (based on total number of flows, packets or bytes) triggers specified action (sending an or running a plugin). This way, operator is able to (semi- )automatically react to known situations such as link congestion or some kind of attack to the network resources. 27

37 5.3 NFDUMP Binary File Format 5. NFSEN/NFDUMP COLLECTOR Network traffic information, in the form of flow records, is processed by capture daemon programs (nfcapd(1) and sfcapd(1)) and stored into NFDUMP proprietary file format. The format was significantly changed in NFDUMP version 1.5 (released in March 2006) to enable processing and storing data received through various flow information export protocols (versions and older support only NetFlow v5/7). Performed changes are not backward compatible, so tools from version and older are not able to read data stored by follow-up versions. On the contrary, new versions are able to read old style data files but they always store data in a new format. Note that this feature is only optional and available after compiling the NFDUMP toolset with -enable-compat14 option given to the configure script. Changed file format consists of a file header, stat record and one or more data block(s). Each NFDUMP file begins with the file header identifying the NFDUMP data file, file format version and a way how data are stored (e.g. compressed or non-compressed file and number of data blocks in the file). All NFDUMP data files are identified by a magic 16b integer with value of 0xA50C at the beginning of the file. Figure 5.5: NFDUMP file layout NFDUMP tools included in version 1.5 and newer can process and independently store traffic information received through several flow export protocols. But with incoming flexibility of new protocols (NetFlow v9 and consequently IPFIX) the problem how to store data exported in a user defined form appeared. Records of the 1.5 file format version are static. It means that all defined items of the data blocks must be stored including those that were not exported. Such items are filled by zeros in the output file but they still occupy disk space and slowdown following file processing. To enable flexible storing of processed data, some next changes in a file format had to be done. It is done as part of development of the last NFDUMP version, which is currently marked as nfdump-snapshot-1.5.7, but finally it seems to be la- 28

38 5. NFSEN/NFDUMP COLLECTOR beled as version 1.6. Concerning about NFDUMP file format, the new version extends a list of supported NetFlow v9 fields and modifies file data blocks. These data blocks format 2 are extension based. It enable user to specify which information from received export packets is processed and stored. Snapshot programs are able to read data from previous data block format 1 (used by 1.5. versions) transparently. To the contrary, 1.5. versions are able to read only file header and statistic record from snapshot data files. Data blocks format 2 are skipped. Data Block Format 2 Figure 5.6: Data block format 2 layout Each data block begins with a header that is common for both format 1 and 2. The header identifies type of data block and its size. Data block format 1 contains after the header only a set of common records with flow information. Data block format 2 currently defines three types of records and the list can be further extended in the future: 1. Common record containing flow information. 2. Extension map describing used record extensions in common records. 3. Exporter record with exporter meta information. Common record basically contains only basic flow information like protocol type, source and destination port or flow start and flow end timestamps. Other flow information is stored as record extensions. In opposite to NetFlow templates, extension maps don t describe a structure (nor size of items) of information stored in common records. They only identify extensions used in common records and their order. The size and internal structure of each extension is defined statically as a structure in C language. There are three special extensions that must be always concatenated with each common record. These record extensions contains source and destination IP addresses, packet counter and byte counter. These items are stored in 29

39 5. NFSEN/NFDUMP COLLECTOR a record extension to enable flexible and effective storing of IPv4 or IPv6 addresses and 32b or 64b counters according to data contained in flow export records. There are 21 currently defined extensions including mentioned 3 mandatory extensions. More detailed description of the NFDUMP common record as well as record extensions can be found in appendix B. 30

40 Chapter 6 NfSen/NFDUMP Modifications for IPFIX Processing FlowMon probe is currently able to export flow information in a form of three protocols: NetFlow v5, NetFlow v9, IPFIX. During development of the FlowMon probe, NfSen/NFDUMP collector has been using for storing data exported by probes. After using this collector for several years with good experiences, the NfSen/NFDUMP collector is now recommended by Liberouter as a FlowMon probe collector. Data sent through NetFlow v5 or v9 protocols can be processed and stored by the NfSen/NFDUMP collector without any problems. But after adding support for the IPFIX protocol in the exporter, the problem with collecting such data had to be solved, because original NFDUMP tools are not able to process IPFIX data. Some existing solutions (described in chapter 4) were tested and partially used for development. But finally, with knowledge of a great reputation of NfSen/NFDUMP between network operators, the decision to stay with NfSen/NFDUMP was made. The possibility of adding IPFIX support was discussed with Peter Haag, main developer and author of the NfSen/NFDUMP collector. His own plans with NfSen/NFDUMP were mainly about improving support for the Net- Flow v9 protocol, not about adding IPFIX processing. But he encouraged us to prepare our own modification of the NfSen/NFDUMP, which could be a special part of the collector project. Currently, there are already such modifications for Packeteer devices and for using Cisco NetFlow Secure Event Logging (NSEL) from Cisco ASA devices. Peter also provided to us useful information mainly about latest changes in the NFDUMP file format (see section 5.3). 31

41 6. NFSEN/NFDUMP MODIFICATIONS FOR IPFIX PROCESSING 6.1 Overall System Changes NFDUMP tool set is basically divided in two parts. Tools storing data into NFDUMP binary file format and tools reading and processing data. To add IPFIX processing feature to the NFDUMP tool set, only storing the data part has to be modified. There are two possible options how (respectively where) to process IPFIX data. The first one is to implement IPFIX processing as part of an existing capture daemon program. The support of NetFlow v5/7 and v9 is implemented this way. There is a common program (nfcapd(1)) that is able to identify protocol of currently processed packet and use appropriate functions to process its content. So this way an identification of IPFIX protocol would be added and IPFIX processing functions would be created. Modified nfcapd(1) would be able to process NetFlow v5, NetFlow v9 and IPFIX protocols at once. The second way is to create entirely new program processing IPFIX packets. Both ways have some advantages as well as disadvantages. The kind of middle way was finally chosen. To avoid code duplication (and creating new bugs) due to writing entirely new program, only a modification of original nfcapd(1) source codes were made. All IPFIX specific parts are in addition bounded by #ifdef IPFIX preprocessor directive. Naturally IP- FIX processing functions had to be written separately but following the borders given by NetFlow v9 processing functions. This follows the first way of implementation described above. But finally the source codes are compiled as two independent programs (thanks to mentioned preprocessor directive). nfcapd(1) is used for NetFlow protocols and new ipfixcapd(1) for IPFIX. This approach allows using of TCP and SCTP transport protocols only with IPFIX traffic (these transport protocols are not officially specified for using with NetFlow protocols). Additionally, this approach preserves original nfcapd(1) as simple as possible. NFDUMP file format (described in section 5.3) stores data in defined record extensions. Therefore if some kind of information should be stored, there must be defined a record extension for such information. The record extension is actually a structure in C language. Due to this feature, NF- DUMP is not able to dynamically process and store unknown (not implemented or enterprise information elements) information. Nor presented IP- FIX extension of NFDUMP tool set is able to process all defined IPFIX information elements [17]. Only NetFlow v9 compatible fields and its IPFIX equivalents are implemented and supported. 32

42 6. N F S EN /NFDUMP M ODIFICATIONS FOR IPFIX P ROCESSING Figure 6.1: Modifications (highlighted in red) of NFDUMP tool set 6.2 FlowMon Exporter Extension Collector is usually realized as dedicated computer collecting data delivered through network connections. Sometimes a combination of both network probe and collector is placed on the same host computer. But exporters are usually implemented with only possibility to connect to the collector over network. In described example with shared host, exporter sends data through loopback interface to a collector. This way of communication unnecessarily loads host system. Therefore, as part of presented modifications of NFDUMP tool set, a flowmon_nfsen(1) program was created. It combines functionality of FlowMon s exporters and NFDUMP s capture daemons. This program reads flow information from COMBO card and directly stores it in the NFDUMP binary file format. The savings of system load are about a half of earlier solution sending data through the loopback interface. 33

43 6. NFSEN/NFDUMP MODIFICATIONS FOR IPFIX PROCESSING 6.3 IPFIX Processing Notes Following paragraphs describe specific parts of IPFIX processing that significantly differ from processing of NetFlow v9 implemented by original nfcapd(1) Transport Protocols All IPFIX compliant implementations must include support for carrying data over SCTP. UDP and TCP transport protocols may be also implemented according to RFC 5101 [4]. Presented implementation supports all mentioned protocols thanks to implemented support in the Linux kernel. User chooses required protocol by a program parameter together with a port where program is listening. ipfixcapd(1) is then able to read data delivered only over specified protocol Timestamps As described in section 3.4, the IPFIX collector must be prepared to handle all possible ways how exporter can express capture time of the first and last packet of the flow. The presented IPFIX implementation is able to do so. The resolution of the timestamps displayed by collector is in milliseconds. If received timestamps are defined in different units (seconds, microseconds or nanoseconds), the collector transforms them into the milliseconds. This behavior can be changed by using NFDUMP file record extension for precise timestamps. This extension was implemented within the presented modifications and will be discussed later in this section. Information model for IPFIX [17] does not specify exact size of the datetime values its size is defined only in the corresponding template. Presented modification is able to handle datetime value sizes from 2 to 8 bytes. Collector then stores these values in NFDUMP file format as combination of 4B integer representing timestamp s seconds and 2B integer corresponding to the timestamp s milliseconds. This resolution is enough for displaying timestamps in the human readable UTC format which is printed by collector. On the other hand, it is not enough for precise timestamps applications. They use a special NFDUMP file record extension that stores timestamps in their original precision. NFDUMP tool set includes nfreplay(1) program to resent collected data to another collector. So created IPFIX part of this program sends flow start and flow end timestamps as a pair of flowstartmilliseconds and flowendmillisecond information elements. In the contrast with Net- 34

44 6. NFSEN/NFDUMP MODIFICATIONS FOR IPFIX PROCESSING Flow v9 processing the IPFIX functions resend exactly same timestamps stored in the collected data. NetFlow v9 functions modify these timestamps relatively to the time of the nfreplay(1) execution. Precise Timestamps Record Extension As mentioned above, the way how NFDUMP tools store timestamps is not enough for precise timestamps, used e.g. for data path reconstruction described in section 7.4. For such purposes, a new NFDUMP file record extension is defined. Timestamps are stored as an 8B integer. But only 62 bits are used for a timestamp itself. The highest two bits are used to distinguish time units of a timestamp. There are four time units supported by IPFIX. time units highest bits value seconds 00 milliseconds 01 microseconds 10 nanoseconds 11 Using count of nanoseconds since epoch 1, 62 bits are enough for approximately 146 years since epoch Sampling Information about packet sampling (see section 3.3) is critical for the analysis of measured data. There are three possible ways how to deal with this information. The first, and the worst way is to ignore these information. Knowledge of sampling information is left to user who has to know what data are collected and how it is sampled. Unfortunately, this approach is used by original NfSen/NFDUMP collector. The second, middle way, means that sampling information are stored with other flow information and displayed to user. Final interpretation of measured data is then left to the user. This approach can be the most accurate way due to possible additional user knowledge about relations between collected data and sampling. On the other hand, this approach leads to increased demands for the user knowledge and the space of stored data. 1. Epoch refers to a time 00:00:00 UTC on January 1,

45 6. NFSEN/NFDUMP MODIFICATIONS FOR IPFIX PROCESSING The last way is to let collector to try to correct sampled information (packet and byte counters) according to received sampling process information. In the presented modification of the NfSen/NFDUMP collector, the first and the last described possibility can be chosen by user. By default, the collector ignores all sampling process information. If user wants to correct sampled information, there is -s option to enable this feature. The collector is able to correct packet and byte counters sampled in the monitoring device by following sampling methods. Description of the sampling methods can be found in section 3.3. Systematic count-based sampling is described by number of consecutively sampled packets (parameter Interval) and by number of not sampled packets between two successive Intervals (parameter Space). Original counter value (before sampling) is approximated as follows: Counter = Space + Interval sampledcounter (6.1) Space Systematic time-based sampling is similar to the previous count-based sampling with difference in measurement units where, in place of packets, microseconds are used. This sampling type is also described by a couple of parameters Interval and Space. The method of original counter values approximation is also the same. Random n-out-of-n sampling represents the random selection of n packets (parameter Size) out of set of N packets (parameter Population). The original counter values are approximated this way: Counter = P opulation sampledcounter (6.2) Size Uniform probabilistic sampling is described by only one parameter. It determines the probability, that a packet is sampled, as a value between 0 and 1. This probability is equal for every packet. The approximation of the original counter values are calculated in this way: Counter = 1 sampledcounter (6.3) P robability Other sampling methods (property match filtering and hash based filtering) are not supported because they don t make a correction possible. 36

46 6. NFSEN/NFDUMP MODIFICATIONS FOR IPFIX PROCESSING All used calculations of original counter values are based on the theory of probabilities. The results of the sampling correction are not the same as if a monitoring device wouldn t sample, but it provides a good approximation of original values. Without complying with the IPFIX information model specification, presented IPFIX implementation supports sampling parameters defined by NetFlow v9, because SAMPLING_INTERVAL and SAMPLING_ALGORITHM parameters are still used by some IPFIX exporters. The information element SAMPLING_INTERVAL describes the rate at which packets are sampled. This sampling can be described as 1-out-of-N sampling where N is the value of the SAMPLING_INTERVAL. The selection of the one representing packet can be done deterministically or randomly (according to field SAMPLING_ALGORITHM). But the way of packet selection doesn t have effect on the approximation method used to reconstruct original counter values. Following expression is based on equation 6.2. Counter = SAMP LING_INT ERVAL sampledcounter (6.4) Using and Testing IPFIX Collector Presented modification of the NfSen/NFDUMP collector is mainly intended for using with FlowMon probes. During the whole implementation process, newly added features were tested using FlowMon probe as a source of IP- FIX data. During different tests, the probe was monitoring dedicated link with a specific traffic as well as high-speed link with live traffic. Tests with specific and known traffic were used to check collector correctness of e.g. storing IPv6 data or correcting sampled information. On the other hand, receiving high volume of flow data, during high-speed network monitoring, was used to ensure preservation of performance qualities of original NFDUMP tools. Running these tests helped to detect bugs in collector, but several bugs were also found and corrected in FlowMon exporter. NFDUMP modifications were aimed at IPFIX data processing and storing. Some little changes of the NfSen frontend had to be done to enable starting new IPFIX capture daemon through the NfSen s user interface. In contrast to the former capture daemons nfcapd(1) and sfcapd(1), daemon ipfixcapd(1) appears as three different programs according to used transport protocol. From the NfSen s point of view, the type of collector can be newly ipfix-udp, ipfix-tcp or ipfix-sctp. 37

47 6. NFSEN/NFDUMP MODIFICATIONS FOR IPFIX PROCESSING The example of NfSen s source configuration follows. %sources = ( s1 => { port => 9994, col => #0000ff, type => netflow }, s2 => { port => 9995, col => #00ff00, type => sflow }, s3 => { port => 9996, col => #ff0000, type => ipfix-udp }, s4 => { port => 9997, col => #ffff00, type => ipfix-tcp }, s5 => { port => 9998, col => #00ffff, type => ipfix-sctp }, ); SCTP Support Using SCTP transport also revealed some possible problems with using Linux kernel SCTP (LKSCTP) support [7]. SCTP support is in most Linux distributions built as a standalone module. This module is not automatically loaded by default, so it must be loaded manually or added to the list in /etc/modprobe.conf file to enable automatic loading of the module. Following series of commands is used to load SCTP module and to check result. # modprobe sctp # cat /proc/modules grep sctp sctp Live 0xfd ipv sctp, Live 0xf8b81000 During implementation of the SCTP support into the collector, LKSCTP s sctp_darn testing tool was found as very useful sample implementation. This tool can serve as both client or server and used for testing developed application. Thanks to the open source codes, this tool can be (and it was) used as an example source code using SCTP transport. 38

48 Chapter 7 IP Flow Monitoring Use Cases Understanding of network behavior and its utilization is necessary for all network service providers. Records of who is communicating with whom, at what time and how long, using what type of application or how much data were transferred can be used for many different purposes. This chapter discusses several applications of IP flow monitoring like network profiling and planning, security analysis or billing. Suitability of selected flow export protocols is also discussed. 7.1 Accounting/Billing Flow information collection seems to be invented for accounting purposes. It provides exactly information needed for billing that can be based on several different characteristics. Bandwidth usage billing, that depends on transferred data from/to specific client. Application usage billing, based on purposes of transferred data (like VoIP or web services). Time-of-day based billing when the price of transferred data is flexible according to time when the communication happened.... and many others or any combination of them. Not to be mistaken, that world is a perfect place and everything is easy, IP flow information collecting process is only the first step of all needed in billing procedure. Following requirements of the specific situation, data must be aggregated and de-duplicated, due to their origin in different observation points. Customers also requires a data conversion into a common (human understandable) format with translated port numbers to names of services (e.g. port 80 means web service) or IP addresses translated according to DNS lookup. 39

49 7. IP FLOW MONITORING USE CASES Using collected data for accounting/billing purposes enables all Net- Flow protocols. sflow is inapplicable due to its sampling features, which makes accounting extremely inaccurate. Sampling can be a problem as well as in NetFlow protocols when measurement device is not able to process all incoming traffic and it has to start sampling. Therefore some kind of profit can be made of using template based protocols (NetFlow v9 or IP- FIX). These protocols can be set to process and export minimum of information required for billing, according to specific billing type. This can reduce system load of measurement device and prevent sampling. There is also another problem with all NetFlow protocols. They use UDP as a transport protocol, that doesn t guarantee reliability which is usually required for billing purposes. IPFIX solves this problem by mandatory support of SCTP protocol (and optional TCP support). Flow information data (especially for billing purposes) carry sensitive information that should be secured. But none of flow export protocols define anything like encryption. These security threads are usually solved by using dedicated links to connect measurement device(s) with collector. In some cases, both devices are placed at the same host keeping possible performance problems in view. 7.2 Network Monitoring and Security Analysis Understanding network behavior and ability to detect changes in the behavior patterns are essential for network anomaly detection and security analysis. Flow monitoring enables operators to watch network parameters and their changes in almost real-time and react to a security threats. On the other hand, warehoused flow data can be lately used to understand and replay the history of realized security incidents. In comparison to traditional IDS, flow export systems miss ability to analyze packet payload. This situation slightly changes with using Flexible NetFlow (or similar) technology (see 2.6.1). Despite of this, there are still techniques to analyze flow data and to make valuable conclusions based on the analysis results Top Statistics Technique of analyzing network top statistics is based on comparison of current network activity (amount of connections, transferred data, type of traffic, etc.) with a normal network activity profile known from historical data. Traffic with some high volume characteristics may stand e.g. for: 40

50 7. IP FLOW MONITORING USE CASES Network scan characterized by a high volume of outgoing connections to a block of addresses discovering vulnerable services. This approach can be improved by analyzing TCP flags. Due to random generation of target addresses (and their physical absence), the large number of connections have set only SYN bit in TCP flags field. DoS/DDoS attack which is, in contrast to network scan, characterized by an extremely high volume of incoming connections to a single host. These connections can be generated by a single attacker (DoS) or by a high amount of attackers (DDoS). In fact, these attackers are usually some kind of victims abused by real attacker to perform attack. There are many methods for DoS/DDoS attacks, mentioning at least SYN flood, IP spoofing (fig. 7.1) or using botnets. Figure 7.1: Scheme of DDoS attack using IP spoofing Presence of a worm in monitored network becomes evident by a significant change in a behavior of infected host. Worm usually attempts to infect the next batch of victims (or perform its malicious activity like spam distribution), that comes to light with significant growth of outgoing connections. Worm presence can be also detected by a high amount of data transferred to/from unexpected host(s). Hosts transferring the most traffic make a relatively fixed group (they are usually enterprise servers located in network). If a new host suddenly appears in this group, operator should be noticed and host should be checked for misconfiguration or worm presence. 41

51 7. IP FLOW MONITORING USE CASES Pattern Matching Pattern matching is another technique how to identify abnormal network traffic. Flow records are, in this case, searched for fields with specific (suspicious) values. The most commonly used fields for these purposes are source/destination addresses and ports and its suspicious content is identified according to (in)formal network communication rules or known behavior of security threats (especially worms). Common rules for pattern matching follow. Filter out all traffic with specific destination ports. Most of worms target specific ports (e.g. Vampire trojan uses always ports 1020 or 6669, etc.) [8]. Operators can allow such ports in combination with specific address range and set an alarm trigger for other combinations. The disadvantage of this approach is possible high volume of false positives. As some worms are known for using specific destination port(s), some other worms are known for using covert channels. Covert channels are a type of hidden communication between compromised systems. It uses empty fields in specific packets that seem to be a regular traffic. For example, ICMP packets have a fixed length (64 bytes on Linux, 40 bytes on Windows) but the rest of such packet can be used to carry suspicious data. Although covert channels can be hidden in a much more sophisticated way (e.g. in specific packet following specific sequence of regular packets), pattern matching can be a way to discover them. Matching IANA reserved addresses. The IANA has reserved several blocks of Internet address space (including both IPv4 as well as IPv6 address space) for private networks [19]. So appearance of such addresses on network interface connecting public networks indicates abnormal traffic that should be analyzed. Besides private network addresses, there are also another special addresses like loopback ( in IPv4 address space or ::1 in IPv6 address space) or IPv4 broadcast address, whose appearance in source IP address field usually indicates IP spoofing and DoS attack. Matching specific prohibited addresses for the network. If the network doesn t provide transit service for other networks (typical case of transit autonomous systems), source addresses of outgoing traffic 42

52 7. IP FLOW MONITORING USE CASES have to belong to the network domain and vice versa source addresses of incoming traffic mustn t be part of the network domain. There is also possible to check source addresses according to list of fixed (or periodically refreshed) addresses. The IP address is placed on the list if it is recognized as e.g. a part of a botnet or as a spam source. There is no way to prevent network from all malicious data and network attacks. The key point is to properly set network usage policy and follow it. Flow data help to monitor observance of the rules and to early react to new security threats. New flexible flow protocols (NetFlow v9 and IPFIX) can better react to specific demands for changing pattern matching (including possibility to analyze part of packet payload) and they seem to be an improvement to ensure more protected networks. 7.3 Profiling and Network Planning Collected flow data are usually stored for a quite long time. This enables not only security analysis of network attacks behavior, but also analysis of long term trends of user behavior and network utilization. The results of such analysis are then used for planning and allocating network and application resources. These analyses are used to minimize cost while maximizing network performance and utilization. New applications, like Video On Demand (VOD) or Voice over IP (VoIP), increase demands for network resources. In addition, these applications are sensitive to latency and jitter so any network congestion is critical for them. Direct upgrade of link bandwidth is costly and very often ineffective solution. Understanding of network utilization is critical for such situations. Network congestion can be sometimes caused by misconfiguration or by unexpected combination of several factors (e.g. complete backup of newly added devices during working hours). In such cases only a correction of configuration or adding a specific network resource would be needed. For the traffic profiling, data can be categorized into the following three types. Regular (business-related) traffic contains all data connected to a purpose of network. If a link capacity is full and all traffic is legitimate, then a bandwidth upgrade seems to be necessary. 43

53 7. IP FLOW MONITORING USE CASES Inappropriate traffic refers to spam, worms traffic, DDoS/DoS attacks or irrelevant traffic not connected to a purpose of network (data downloading or P2P traffic in an enterprise network). Fighting with spam, worms and attackers is never-ending battle where flow data can help. On the other hand, irrelevant (not-business-related) traffic can be tolerated to a certain level. Some enterprise operators try to prevent such traffic entirely, including blocking web sites like Facebook, Youtube or Twitter. Using these (in most cases a kind of social) services could mean, that people are goofing off at work. But these services can be used e.g. for watching instructional videos (and saving time with learning from manual) or customer support on a social networks. So strict blocking of such services may not be a best solution, because sometimes it may be a regular business-related traffic. Irrelevant traffic must be solved when it overgrow a certain limit and when it can impact regular traffic. In most cases, the biggest problems with irrelevant traffic are caused due to breaking rules by individual person and should be solved individually. Unwise traffic is in general legitimate traffic at inappropriate time. Usually it refers to backups or synchronizations performed during working hours. Such traffic is necessary for network services as well as network itself, but it can be safely rescheduled to a non-peak hours. Flow export protocols provide operators a tool to classify traffic according to specific parameters. As well as in case of security analysis or billing, flexible flow protocols provide more freedom to specify required parameters to classify traffic. 7.4 Network Maps and Data Path Reconstruction These two features extends some flow analyze tools and they are used to easier understanding of network utilization. The first one, network maps, highlights information stored as a flow records in a more understandable form. It displays connections between particular hosts including a strength (how much data were transferred or how often they communicate) of a connection. That gives an easy to understand and fast overview of network utilization. The network map can be also connected with information about a geographical position of hosts (e.g. via Google Maps API). Data path reconstruction is a feature to track paths of flows from source to its destination using flow information. This feature seems to be very in- 44

54 7. IP F LOW M ONITORING U SE C ASES Figure 7.2: Network map generated by commercial program Scrutinizer ( teresting in virtual networks. In combination with VLAN information contained in flow data, the virtual network structure can be discovered. There are two algorithms to reconstruct data path. Interface number algorithm that combines information about numbers of passed interfaces with knowledge of a physical network structure. Timestamps algorithm detects time differences between timestamps of the same flow in different nodes. This way, ordered sequence of passed nodes can be discovered. This algorithm is sensitive to a time synchronization of all observation devices as well as to a precision of timestamps. Due to a speed of a transmission, minimal required precision must be in microseconds. On the other hand, this approach doesn t require any additional knowledge of a physical network structure. Results of both described features are dependent on a number of flow sources. The data path reconstruction actually requires flow data source in every node (usually router) that is supposed to be displayed as a part of a flow path. 45

55 7. IP FLOW MONITORING USE CASES Network maps need only basic flow information (source/destination address, packet/byte count), so they can be created based on any flow export protocol data. But timestamps precision, needed by timestamps based data path reconstruction algorithm, requires using IPFIX as a flow export protocol. All NetFlow protocols enable timestamps only in milliseconds, but IPFIX defines timestamps information elements up to nanosecond precision. 46

56 Chapter 8 Conclusion This thesis was aimed for flow data collection using IPFIX protocol. In the first part, flow export protocols are described successively from the oldest NetFlow versions through the NetFlow v9 to the latest IPFIX protocol, which is supposed to replace proprietary NetFlow and its alternatives. Therefore next parts of the thesis are focused on possibilities of using IP- FIX. The second part analyzes existing solutions for IPFIX data collection. There are summarized practical experiences in using such solutions with the FlowMon probe. Merits as well as shortcomings of each tested solution are mentioned. The common characteristic of all analyzed solutions is an inability to process and store newly defined IPFIX enterprise information elements. Although there is a possibility to process such information, none of tested solution is able to store data in a sufficiently flexible file format. Even NFDUMP/NfSen collector is not able to store newly defined information elements described only by incoming template. But, in comparison to mentioned solutions, the NFDUMP file format enables much more flexibility due to using record extensions. Therefore, the IPFIX data processing, presented in this thesis, was proposed and implemented as an extension to the existing NFDUMP/NfSen collector. The idea of extending NFDUMP by IPFIX processing was discussed with Peter Haag, the main developer of the NFDUMP/NfSen. We suppose future insertion of this extension into the original project of NFDUMP/NfSen collector. Presented solution was fully tested only with FlowMon exporter. Many tests using different IPFIX exporters must be still done. The thesis mentions using precise timestamps for the purposes of data path reconstruction. Due to availability of precise timestamps only in IPFIX protocol, special NFDUMP record extension was created. The format of this extension together with other implementation challenges and experiences from testing and using presented IPFIX implementation are mentioned in chapter 6. 47

57 8. CONCLUSION Together with further tests, storing of all possible IPFIX information elements seems to be a subject of the future work. IPFIX Working Group is preparing an IPFIX file format [22], which could be used for these purposes. But the possibility of storing NetFlow data in common format should be also considered. The presented extension of the NFDUMP/NfSen collector was successfully tested by Liberouter s Testing group. Detected bugs were fixed and the collector is ready for deployment with FlowMon probes. 48

58 Bibliography [1] INVEA-TECH a.s. INVEA-TECH: High-Speed Networking and FPGA Solutions [online]. Available from: [cited ]. [2] Carnegie Mellon University. libfixbuf [online]. Available from: http: //tools.netsa.cert.org/fixbuf [cited ]. [3] Inc. Cisco Systems. Cisco Web Pages [online] Available from: [cited ]. [4] Benoit Claise, Stewart Bryant, Simon Leinen, Thomas Dietz, and Brian H. Trammell. Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information. IETF, Network Working Group, Available from: http: //tools.ietf.org/html/rfc5101. [5] Benoit Claise, Juergen Quittek, and Andrew Johnson. Packet Sampling (PSAMP) Protocol Specifications. IETF, Network Working Group, Available from: draft-ietf-psamp-protocol-09. [6] Benoit Claise, Ganesh Sadasivan, Vamsi Valluri, and Martin Djernaes. Cisco Systems NetFlow Services Export Version 9. IETF, Network Working Group, Available from: html/rfc3954. [7] Linux community. Linux Kernel SCTP [online] Available from: [cited ]. [8] Simovits Consulting. Trojan list sorted on trojan port [online]. Available from: trojans.html [cited ]. [9] Luca Deri. Open Source VoIP Traffic Monitoring. In SANE 2006 proceedings, Available from: pdf. 49

59 8. CONCLUSION [10] Thomas Dietz, Benoit Claise, Paul Aitken, Falco Dressler, and Georg Carle. Information Model for Packet Sampling Exports. IETF, Network Working Group, Available from: html/draft-ietf-psamp-info-11. [11] European Parliament. Electronic communications: personal data protection rules and availability of traffic data for anti-terrorism purposes, Available from: cgi/sga_doc?smartapi!celexplus!prod!celexnumdoc&lg= EN&numdoc=32006L0024. [12] Mark Fullmer. flow-tools information [online]. Available from: http: // [cited ]. [13] Yiming Gong. Detecting Worms and Abnormal Activities with Net- Flow [online]. Available from: infocus/1802 [cited ]. [14] IANA. IP Flow Information Export (IPFIX) Information Elements [online]. Available from: ipfix/ipfix.xhtml [cited ]. [15] IETF. IP Flow Information Export Working Group [online]. Available from: ipfix-charter.html [cited ]. [16] Simon Leinen. Evaluation of Candidate Protocols for IP Flow Information Export (IPFIX). IETF, Network Working Group, Available from: [17] Juergen Quittek, Stewart Bryant, Benoit Claise, Paul Aitken, and Jeff Meyer. Information Model for IP Flow Information Export. IETF, Network Working Group, Available from: org/html/rfc5102. [18] Juergen Quittek, Tanja Zseby, Benoit Claise, and Sebastian Zander. Requirements for IP Flow Information Export (IPFIX). IETF, Network Working Group, Available from: html/rfc3917. [19] Yakov Rekhter, Robert G. Moskowitz, Daniel Karrenberg, Geert Jan de Groot, and Eliot Lear. Address Allocation for Private Internets. IETF, Network Working Group, Available from: ietf.org/html/rfc

60 8. CONCLUSION [20] sflow.org. sflow.org - Making the Network Visible [online]. Available from: [cited ]. [21] Caligare s.r.o. NetFlow Portal [online] Available from: http: //netflow.caligare.com/index.htm [cited ]. [22] Brian Trammell, Elisa Boschi, Lutz Mark, Tanja Zseby, and Arno Wagner. Specification of the IPFIX File Format. IETF, Network Working Group, Internet Draft. Available from: org/html/draft-ietf-ipfix-file-03. [23] CESNET z.s.p.o. Description of COMBO cards [online]. Available from: [cited ]. [24] CESNET z.s.p.o. FlowMon probe web page [online]. Available from: [cited ]. [25] CESNET z.s.p.o. Liberouter project web page [online]. Available from: [cited ]. 51

61 Appendix A NetFlow Export Packet Format A.1 NetFlow Packet Version 1 Bytes Contents Description Table A.1: NetFlow v1 Header Format 0-1 version NetFlow export format version number 2-3 count Number of flows exported in this packet (1-24) 4-7 sys_uptime Current time in milliseconds since the export device booted 8-11 unix_secs Current count of seconds since 0000 UTC unix_nsecs Residual nanoseconds since 0000 UTC 1970 Bytes Contents Description Table A.2: NetFlow v1 Record Format 0-3 srcaddr Source IP address 4-7 dstaddr Destination IP address 8-11 nexthop IP address of next hop router input SNMP index of input interface output SNMP index of output interface dpkts Packets in the flow doctets Total number of Layer 3 bytes in the packets of the flow first SysUptime at start of flow last SysUptime at the time the last packet of the flow was received srcport TCP/UDP source port number or equivalent dstport TCP/UDP destination port number or equivalent pad1 Unused (zero) bytes 38 prot IP protocol type 39 tos IP type of service 40 flags Cumulative OR of TCP flags pad2 Unused (zero) bytes 52

62 A.2 NetFlow Packet Version 5 A. NETFLOW EXPORT PACKET FORMAT Bytes Contents Description Table A.3: NetFlow v5 Header Format 0-1 version NetFlow export format version number 2-3 count Number of flows exported in this packet (1-30) 4-7 sys_uptime Current time in milliseconds since the export device booted 8-11 unix_secs Current count of seconds since 0000 UTC unix_nsecs Residual nanoseconds since 0000 UTC flow_sequence Sequence counter of total flows seen 20 engine_type Type of flow-switching engine 21 engine_id Slot number of the flow-switching engine sampling First two bits hold the sampling mode; remaining 14 bits hold value of sampling interval Bytes Contents Description Table A.4: NetFlow v5 Record Format 0-3 srcaddr Source IP address 4-7 dstaddr Destination IP address 8-11 nexthop IP address of next hop router input SNMP index of input interface output SNMP index of output interface dpkts Packets in the flow doctets Total number of Layer 3 bytes in the packets of the flow first SysUptime at start of flow last SysUptime at the time the last packet of the flow was received srcport TCP/UDP source port number or equivalent dstport TCP/UDP destination port number or equivalent 36 pad1 Unused (zero) bytes 37 tcp_flags Cumulative OR of TCP flags 38 prot IP protocol type 39 tos IP type of service src_as Autonomous system number of the source, either origin or peer 53

63 A. NETFLOW EXPORT PACKET FORMAT Table A.4: NetFlow v5 Record Format (continues) Bytes Contents Description dst_as Autonomous system number of the destination, either origin or peer 44 src_mask IP protocol type 45 dst_mask IP type of service pad2 Unused (zero) bytes A.3 NetFlow Packet Version 6 Bytes Contents Description Table A.5: NetFlow v6 Header Format 0-1 version NetFlow export format version number 2-3 count Number of flows exported in this packet (1-30) 4-7 sys_uptime Current time in milliseconds since the export device booted 8-11 unix_secs Current count of seconds since 0000 UTC unix_nsecs Residual nanoseconds since 0000 UTC flow_sequence Sequence counter of total flows seen 20 engine_type Type of flow-switching engine 21 engine_id Slot number of the flow-switching engine sampling First two bits hold the sampling mode; remaining 14 bits hold value of sampling interval Bytes Contents Description Table A.6: NetFlow v6 Record Format 0-3 srcaddr Source IP address 4-7 dstaddr Destination IP address 8-11 nexthop IP address of next hop router input SNMP index of input interface output SNMP index of output interface dpkts Packets in the flow doctets Total number of Layer 3 bytes in the packets of the flow 54

64 A. NETFLOW EXPORT PACKET FORMAT Table A.6: NetFlow v6 Record Format (continues) Bytes Contents Description first SysUptime at start of flow last SysUptime at the time the last packet of the flow was received srcport TCP/UDP source port number or equivalent dstport TCP/UDP destination port number or equivalent 36 pad1 Unused (zero) bytes 37 tcp_flags Cumulative OR of TCP flags 38 prot IP protocol type 39 tos IP type of service src_as Autonomous system number of the source, either origin or peer dst_as Autonomous system number of the destination, either origin or peer 44 src_mask IP protocol type 45 dst_mask IP type of service pad2 Unused (zero) bytes A.4 NetFlow Packet Version 7 Bytes Contents Description Table A.7: NetFlow v7 Header Format 0-1 version NetFlow export format version number 2-3 count Number of flows exported in this packet (1-30) 4-7 sys_uptime Current time in milliseconds since the export device booted 8-11 unix_secs Current count of seconds since 0000 UTC unix_nsecs Residual nanoseconds since 0000 UTC flow_sequence Sequence counter of total flows seen reserved Unused (zero) bytes 55

65 Bytes Contents Description A. NETFLOW EXPORT PACKET FORMAT Table A.8: NetFlow v7 Record Format 0-3 srcaddr Source IP address 4-7 dstaddr Destination IP address 8-11 nexthop IP address of next hop router input SNMP index of input interface output SNMP index of output interface dpkts Packets in the flow doctets Total number of Layer 3 bytes in the packets of the flow first SysUptime at start of flow last SysUptime at the time the last packet of the flow was received srcport TCP/UDP source port number or equivalent dstport TCP/UDP destination port number or equivalent 36 pad1 Unused (zero) bytes 37 tcp_flags Cumulative OR of TCP flags 38 prot IP protocol type 39 tos IP type of service src_as Autonomous system number of the source, either origin or peer dst_as Autonomous system number of the destination, either origin or peer 44 src_mask IP protocol type 45 dst_mask IP type of service flags Flags indicating, among other things, what flows are invalid router_sc IP address of the router that is bypassed by the Catalyst 5000 series switch. This is the same address the router uses when it sends NetFlow export packets. 56

66 A.5 NetFlow Packet Version 8 Bytes Contents Description A. NETFLOW EXPORT PACKET FORMAT Table A.9: NetFlow v8 Header Format 0-1 version NetFlow export format version number 2-3 count Number of flows exported in this packet (1-30) 4-7 sys_uptime Current time in milliseconds since the export device booted 8-11 unix_secs Current count of seconds since 0000 UTC unix_nsecs Residual nanoseconds since 0000 UTC flow_sequence Sequence counter of total flows seen 20 engine_type Type of flow switching engine 21 engine_id ID number of the flow switching engine 22 aggregation Aggregation method being used 23 agg_version Version of the aggregation export reserved Unused (zero) bytes Table A.10: NetFlow v8 Full Flow Record Format Bytes Contents Description 0-3 dstaddr Destination IP address 4-7 srcaddr Source IP address 8-9 dstport TCP/UDP destination port number srcport TCP/UDP source port number dpkts Packets in the flow doctets Total number of Layer 3 bytes in the packets of the flow first SysUptime, in seconds, at start of flow last SysUptime, in seconds, at the time the last packet of the flow was received output SNMP index of output interface input SNMP index of input interface 32 tos IP type of service 33 prot IP protocol type 34 marked_tos Type of Service of the packets that exceeded the contract 35 pad Unused (zero) bytes extrapkts Packets that exceed the contract router_sc IP address of the router that is bypassed by the Catalyst 5000 series switch. This is the same address the router uses when it sends NetFlow export packets. 57

67 A. NETFLOW EXPORT PACKET FORMAT NetFlow protocol version 8 supports several aggregation schemes with different flow record format. For the full description of the scheme flow record formats see A.6 NetFlow Packet Version 9 Bytes Contents Description Table A.11: NetFlow v9 Header Format 0-1 version NetFlow export format version number 2-3 count Number of flow sets exported in this packet, both template and data (1-30). 4-7 sys_uptime Current time in milliseconds since the export device booted 8-11 unix_secs Current count of seconds since 0000 UTC package_seq Sequence counter of all export packets sent by the export device source_id A 32-bit value that is used to guarantee uniqueness for all flows exported from a particular device (the Source ID field is the equivalent of the engine type and engine ID fields found in the NetFlow v5 and v8 headers). Table A.12: NetFlow v9 Template FlowSet Format bit 0-15 bit FlowSet ID = 0 Length Template ID 256 Field Count Field Type 1 Field Length 1 Field Type 2 Field Length Field Type N Field Length N Template ID 257 Field Count Field Type 1 Field Length Field Type M Field Length M Template ID K Field Count

68 A. NETFLOW EXPORT PACKET FORMAT FlowSet ID value of 0 is reserved for the Template FlowSet. Length of this FlowSet. Length value is used to determine the position of the next FlowSet record, which could be any type of FlowSet. Length is the sum of the lengths of the FlowSet ID, the Length itself, and all Template Records within this FlowSet. Template ID is a unique identifier of newly generated Template Records. This uniqueness is local to the measurement device that generated the Template ID. Templates that define data record formats begin numbering at 256 since are reserved for FlowSet IDs. Field Count specifies number of fields in this Template Record. Because a Template FlowSet usually contains multiple Template Records, this field allows the Collector to determine the end of the current Template Record and the start of the next. Field Type is a numeric value representing the type of the field. The list of currently defined field types can be found at com/en/us/technologies/tk648/tk362/technologies_white_ paper09186a00800a3db9.html Field Length of the corresponding Field Type, in bytes. Table A.13: NetFlow v9 Options Template FlowSet Format bit 0-15 bit FlowSet ID = 1 Length Template ID Option Scope Length Option Length Scope 1 Field Type Scope 1 Field Length... Scope N Field Length Option 1 Field Type Option 1 Field Length... Option M Field Length Padding FlowSet ID value of 1 is reserved for the Options Template FlowSet. Length of this FlowSet. Length value is used to determine the position of the next FlowSet record, which could be any type of FlowSet. Length is the sum of the lengths of the FlowSet ID, the Length itself, and all Options Template Records within this FlowSet. 59

69 A. NETFLOW EXPORT PACKET FORMAT Template ID is a unique identifier of newly generated Template Records. This uniqueness is local to the measurement device that generated the Template ID. Templates that define data record formats begin numbering at 256 since are reserved for FlowSet IDs. Option Scope Length in bytes of all Scope field definitions contained in this Options Template Record. Option Length (in bytes) of all options field definitions contained in this Options Template Record. Scope N Field Type specifies the relevant portion of the exporter process to which the Options Template Record refers. Currently defined values can be System (0x0001), Interface (0x0002), Line Card (0x0003), Cache (0x0004), or Template (0x0005). Scope N Field Length (in bytes) of the Scope field, as it would appear in an Options Data Record. Option M Field Type is a numeric value that represents the type of field that would appear in the Options Template Record. These numbers are the same as Field Type numbers Option M Field Length (in bytes) of the Option field. Padding that exporter should insert so that the subsequent FlowSet starts at a 4-byte aligned boundary. The Length field includes the padding bytes. Table A.14: NetFlow v9 Data FlowSet Format bit 0-15 bit FlowSet ID = Template ID (>255) Length Record 1 - Field Value 1 Record 1 - Field Value 2 Record 1 - Field Value 3... Record 2 - Field Value 1 Record 1 - Field Value 2 Record 2 - Field Value 3... Record 3 - Field Value Padding 60

70 A. NETFLOW EXPORT PACKET FORMAT FlowSet ID = Template ID is identifier associating Data FlowSet with appropriate template. Length of this FlowSet. Length value is used to determine the position of the next FlowSet record, which could be any type of FlowSet. Length is the sum of the lengths of the FlowSet ID, the Length itself, all Flow Records within this FlowSet and padding bytes, if any. Record N - Field Value M represents field values. The Type and Length of the fields have been previously defined in the (Options) Template Record referenced by the FlowSet ID or Template ID. Padding that exporter should insert so that the subsequent FlowSet starts at a 4-byte aligned boundary. The Length field includes the padding bytes. 61

71 Appendix B NFDUMP File Format Records B.1 Common Record Structure The record CommonRecordType (defined as numerical value 1) describes a data record including all optional extensions for this record. A data record contains common items (referred as extension 0) and at least the first 3 extensions (1-3). All other extensions are optional and described in the extension map. The common record contains a reference to the extension map which applies for this record. B.1.1 Extension 0 Table B.1: Extension 0 Byte 0 Byte 1 Byte 2 Byte 3 record type = 1 size flags exporter_ref extension map msec_first msec_last first last fwd_status tcpflags protocol src_tos srcport dstport / ICMP flags - bit map for description of required extensions type. Meanings of each bit is displayed in the following table flags bit 0 1 Extension 1 - src/dst addresses 0 IPv4 IPv6 Extension 2 - packet counter 1 32b 64b Extension 3 - byte counter 2 32b 64b exporter_ref - identification of a source of the flow information contained in this record 62

72 B. NFDUMP FILE FORMAT RECORDS extension map - identification of a ExtensionMapType record used for description of extensions used in this record msec_first, first - fields for storing timestamp of capturing the first packet of the flow msec_last, last - fields for storing timestamp of capturing the last packet of the flow fwd_status - NetFlow v9 field 89 - FORWARDING STATUS tcpflags - cumulative of all the TCP flags seen for this flow protocol - IP protocol src_tos - Type of Service srcport, dstport - source/destination ports ICMP - ICMP packet type in case of ICMP traffic (reported as (ICMP Type*256) + ICMP code) B.1.2 Extension 1 Table B.2: Extension 1 - IPv4 addresses Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 srcip dstip Table B.3: Extension 1 - IPv6 addresses Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 srcip srcip dstip dstip 63

73 B. NFDUMP FILE FORMAT RECORDS B.1.3 Extension 2 - Input Packet Counter Table B.4: Extension 2-32b counter size Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 in_packets Table B.5: Extension 2-64b counter size Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 in_packets B.1.4 Extension 3 - Byte Counter Table B.6: Extension 3-32b counter size Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 bytes Table B.7: Extension 3-64b counter size Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 bytes 64

74 B.2 Optional Record Extensions B. NFDUMP FILE FORMAT RECORDS B.2.1 Extension 4, 5 - Interface Record Input and output interfaces accepted as either 2 or 4 bytes numbers. Table B.8: Extension 4-2B values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 input output Table B.9: Extension 5-4B values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 input output B.2.2 Extension 6, 7 - Autonomous System Record Source and destination BGP autonomous system numbers accepted as either 2 or 4 bytes numbers. Table B.10: Extension 6-2B values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 src_as dst_as Table B.11: Extension 7-4B values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 src_as dst_as B.2.3 Extension 8 - Multiple Fields Record This record contains destination ToS, flow direction and source/destination address masks in slash notation fields grouped together in a 32b value. Table B.12: Extension 8 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 dst_tos direction srcmask dstmask 65

75 B. NFDUMP FILE FORMAT RECORDS B.2.4 Extension 9, 10 - Next-Hop IP Address Record IP address of next-hop router accepted as either IPv4 and IPv6. Table B.13: Extension 9 - IPv4 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 next_hop_ip Table B.14: Extension 10 - IPv6 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 next_hop_ip next_hop_ip B.2.5 Extension 11, 12 - BGP Next-Hop IP Record Next-hop router s IP in the BGP domain accepted as either IPv4 or IPv6. Table B.15: Extension 11 - IPv4 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 bgp_next_ip Table B.16: Extension 12 - IPv6 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 bgp_next_ip bgp_next_ip B.2.6 Extension 13 - VLAN Record Virtual LAN identifier associated with ingress and egress interfaces. Table B.17: Extension 13 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 src_vlan dst_vlan 66

76 B. NFDUMP FILE FORMAT RECORDS B.2.7 Extension 14, 15 - Out Packet Counter Record Table B.18: Extension 14-32b values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 out_packets Table B.19: Extension 15-64b values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 out_packets B.2.8 Extension 16, 17 - Out Byte Counter Record Table B.20: Extension 16-32b values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 out_bytes Table B.21: Extension 17-64b values Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 out_bytes B.2.9 Extension 18, 19 - Aggregated Flows Record Number of flows that were aggregated accepted as either 2 or 4 bytes numbers. Table B.22: Extension 18-4B value Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 aggr_flows Table B.23: Extension 19-8B value Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 aggr_flows 67

77 B.2.10 Extension 20, 21 - MAC Record B. NFDUMP FILE FORMAT RECORDS Information about incoming/outgoing source/destination MAC addresses. Table B.24: Extension 20 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 0 incoming source MAC address 0 outgoing destination MAC address Table B.25: Extension 21 Byte 0 Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 Byte 6 Byte 7 0 incoming destination MAC address 0 outgoing source MAC address B.3 Extension Map Record Structure The extension map replaces the individual flags in previous layout. With many possible extensions and combination of extensions an extension map is more efficient while reading and decoding the record. The structure of extension map record follows. Table B.26: Extension Map Record Byte 0 Byte 1 Byte 2 Byte 3 record type = 2 size map ID extension size Extension ID 1 Extension ID 2... Extension ID n 32b alignment (filled by 0) 68

78 Appendix C Content of the CD-ROM A CD-ROM is attached to this thesis. The disc contains: Directory text: L A TEX source code of this thesis including all pictures. Subdirectory output/ contains several standard output file formats. Directory sources: Source codes of the IPFIX processing extension to the NFDUMP/NfSen collector. Source codes are based on nfdump-snapshot development version. Build and install instructions: $ cd sources/ $./configure --enable-ipfix $ make # make install 69