qflow: a fast customer-oriented NetFlow database for accounting and data retention

Transcription

1 qflow: a fast customer-oriented NetFlow database for accounting and data retention Hallgrímur H. Gunnarsson Faculty of Industrial Engineering, Mechanical Engineering and Computer Science University of Iceland 2014

2

3 QFLOW: A FAST CUSTOMER-ORIENTED NETFLOW DATABASE FOR ACCOUNTING AND DATA RETENTION Hallgrímur H. Gunnarsson 60 ECTS thesis submitted in partial fulfillment of a Magister Scientiarum degree in Computer Science Advisors Snorri Agnarsson Helmut Neukirchen Faculty Representative Jón Ingi Einarsson Faculty of Industrial Engineering, Mechanical Engineering and Computer Science School of Engineering and Natural Sciences University of Iceland Reykjavik, September 2014

4 qflow: a fast customer-oriented NetFlow database for accounting and data retention 60 ECTS thesis submitted in partial fulfillment of a M.Sc. degree in Computer Science Copyright c 2014 Hallgrímur H. Gunnarsson All rights reserved Faculty of Industrial Engineering, Mechanical Engineering and Computer Science School of Engineering and Natural Sciences University of Iceland Hjarðarhagi Reykjavik, Reykjavik Iceland Telephone: Bibliographic information: Hallgrímur H. Gunnarsson, 2014, qflow: a fast customer-oriented NetFlow database for accounting and data retention, M.Sc. thesis, Faculty of Industrial Engineering, Mechanical Engineering and Computer Science, University of Iceland. Printing: Háskólaprent, Fálkagata 2, 107 Reykjavík Reykjavik, Iceland, September 2014

5 Abstract Internet service providers in Iceland must manage large databases of network flow data in order to charge customers and comply with data retention laws. The databases need to efficiently handle large volumes of data, often billions or trillions of records, and they must support fast queries of traffic volume per customer over time and extraction of raw flow data for given customers. Popular open-source tools for storing flow data, such as nfdump and flow-tools, are backed by flat binary files. They do not provide any type of indexing or summaries of customer traffic. As a result, flow queries for a given customer need to linearly scan through all the flow records in a given time period. We present a high-performance customer-oriented flow database that provides fast customer queries and compressed flow storage. The database is backed by indexed flow tablets that allow for fast extraction of customer flows and traffic volume per customer. Útdráttur Internetþjónustur á Íslandi þurfa að geyma mikið magn af netmælingargögnum til að gjaldfæra netnotkun og uppfylla lög um gagnageymd. Gagnagrunnskerfi fyrir netmælingargögn þurfa að ráða við margar færslur, oft á tíðum milljarða eða billjarða, og þau verða að styðja hraðvirka uppflettingu á gagnamagni og hráum netmælingarfærslum fyrir notendur. Vinsælar opnar lausnir, t.d. nfdump og flow-tools, geyma færslur í flötum skrám. Þau bjóða ekki upp á flýtivísa fyrir hraðvirka leit eða samantektir á gagnamagni notenda. Þar af leiðandi þarf að lesa allar færslurnar til að svara fyrirspurnum. Í þessu verkefni kynnum við til sögunar nýtt afkastamikið notendaskipt gagnagrunnskerfi fyrir netmælingargögn. Kerfið geymir gögnin á þjöppuðu sniði en býður samt upp á hraðvirkar fyrirspurnir um notendur. Gagnagrunnskerfið byggir ofan á safni af litlum töflusneiðum sem mynda eina heild. Hverri töflusneið fylgir flýtivísir sem gerir kleift að framkvæma hraðvirkar fyrirspurnir um færslur og gagnamagn notenda. v

6

7 Contents List of Figures List of Tables ix xi 1. Introduction Motivation Requirements Contribution Related work Structure of thesis Flow-based monitoring Network monitoring Flow probes Overview Flow export Packet sampling Cisco NetFlow History Version Version Storage requirements Observation points Edge deployment Ingress/egress monitoring Deployment strategies Customer traffic Design and implementation Architecture Collector Design Flow format Configuration Backend protocol vii

8 Contents 3.3. Database Design Table queue Record format Indexer Tablets Materialized views Filtering Language Implementation Reports Flow extraction Flow summary Flow filter Time-based reports Customer reports Evaluation Environment Collector Preparation Results Indexer Flow storage Flow extraction Preparation Results Materialized views Conclusions Summary Future work Bibliography 63 A. Flow protobuf 65 B. Collector configuration protobuf 69 C. Grammar for the filter language 71 viii

9 List of Figures 2.1. Flow probe internals Relative sampling error NetFlow v5 export packet Structure of NetFlow v9 export packet NetFlow v9 template flowset NetFlow v9 data flowset NetFlow edge deployment Example router with both ingress and egress monitoring enabled Provider network with both ingress and egress monitoring enabled An overview of the qflow system Collector overview Directory layout of the flow database Flow capture pipeline Structure of a block Directory layout for flow tablets Internal layout of a flow tablet View file format ix

10 LIST OF FIGURES 3.9. Parse tree for example filter expression File size of materialized view Update time for materialized view Query time for materialized view Export time for materialized view x

11 List of Tables 2.1. Format of NetFlow v5 header Format of NetFlow v5 record Format of NetFlow v9 header Collector performance results for NetFlow v Collector performance results for NetFlow v Indexer performance for 10M records Indexer performance for 20M records Indexer performance for 30M records Indexer performance for 40M records Storage efficiency of qflow vs. flow-tools Flow extraction performance for a single IP Flow extraction performance for a network xi

12

13 1. Introduction 1.1. Motivation Internet service providers (ISPs) in Iceland must store and query large volumes of network traffic monitoring data in order to charge customers and comply with data retention laws. A typical ISP might need to store and process up to 50K flow records per second, which translates into more than 8 GB per hour of monitoring data [9]. Data retention laws require the data to be stored for six months, and for 50K records per second it would take 34 TB to store the resulting 770 billion records. A number of commercial and open source solutions exist for storing and managing network flow data, e.g. Cisco NetFlow collector, pmacct, flow-tools and nfdump. They usually store flow records in either a relational database or raw binary files. In general, relational databases offer flexibility and a powerful query language, but they are known to be slower, especially in terms of insertion rate, and consume more disk space when compared to specialized tools that use raw binary files [6]. Furthermore, scaling relational databases to billions or trillions of records can be a real challenge. For example, the iibench 1B row insert benchmark for MySQL shows that the insert rate is highly dependent on the table size. As the table grows larger and the data no longer fits in memory, the performance degrades dramatically. In the beginning of the benchmark, MySQL can sustain around 40,000 inserts per second, but after inserting 200M rows, it has fallen down to 5,000 inserts per second. Close to 1B rows, the rate is down to 876 inserts per second. With 50K records per second, the table would reach 200M rows in one hour, and 1B rows in less than six hours. In contrast, tools based on raw binary files can handle a high insertion rate, e.g. nfdump can store over 250K records per second on a dual-core machine [6]. The binary files are flat, without any sort of index, and the insertion rate does not degrade over time. The flow records are written in the order they are received and files are usually rotated regularly, e.g. every 5, 10 or 15 minutes. Although they support fast insertion, such tools present two problems when they 1

14 1. Introduction are used for data retention and billing. First, extracting flows for a given customer requires sequentially scanning all the flow records within the given time period. For a typical ISP, this could mean scanning hundreds of gigabytes or terabytes to locate a relatively few number of records. Secondly, fetching hourly/daily/monthly traffic summaries per customer also requires sequentially scanning all the flow records. Typically, ISPs write scripts to precompute summaries and store them in a relational database. After each rotation of a raw binary file, a script will compute the volume per IP and update summaries in a relational database. This can also be problematic to scale when the number of files and IPs grows large. In this thesis we present a new system that stores flow records in custom sorted flow tables that allow for fast extraction of customer flows. In addition, the system maintains an index for each flow table that can be used to answer queries about traffic volume per customer, and it also maintains higher-level aggregate summaries (hourly, daily, monthly) in an efficient manner Requirements This section describes the key requirements of the system. The requirements are based on our experience working with Icelandic ISPs on flow accounting and data retention. 1. Data retention fulfillment The EU data retention directive, adopted into article 42 of the Icelandic Electronic Communications Act no. 81/2003 [1], requires that every Internet service provider retain a log of Internet traffic metadata for the purpose of law enforcement. The log must contain customer IP addresses, all connections that were established, time of connection, and the IP address of both connection endpoints. The log must be retained for six months, after which any older records are to be removed. The ISP must be able to deliver customer metadata to law enforcement upon request. 2. Compressed storage Given the large volume of data and the requirement to store it for six months, the system should store the data in a compressed format. Typical zlib compression ratios are on the order of 2:1 to 5:1 [13]. Given 50K 2

15 1.2. Requirements records per second and 34 TB of data over six months, zlib compression could be expected to reduce the storage requirements to something between 6 to 17 TB. 3. Traffic summaries The system needs to maintain time-based aggregates of traffic volume per customer IP, e.g. hourly, daily and monthly aggregates. The time periods should be configurable. Updates and queries must be fast. This requirement is motivated by the needs of both ISPs and their customers. First, for usage-based charging of network traffic, an ISP needs to determine the traffic volume per customer during a given billing period (usually 1 month). This data is then imported into a billing system at the end of the billing period. Secondly, according to the Icelandic regulation no. 526/2011 on electronic telecommunications billing [18], customers are entitled to get detailed itemization of bills for telecommunications charges. In particular, they are entitled to data usage details broken down by time period with hourly precision. Internet service providers usually provide a web interface for customers where they can view their traffic volumes broken down by different time periods (monthly, daily, hourly). 4. Flow extraction When necessary, the system must be able to drill down and extract raw records for a given customer. This is necessary both for data retention fulfillment, and to settle any billing disputes when customers complain about usage-based charges. The system should respond to extraction queries in a reasonable amount of time. This condition precludes the possibility of sequentially scanning the records to filter out customer records, since it would be prohibitively expensive to scan gigabytes or terabytes for each extraction query. 5. Scalability Ideally, the system should scale to any volume of data given that enough hardware is available to store and process it. To support large installations that exceed the capacity of a single machine, the system should be able to run on multiple machines in a distributed fashion. 3

16 1. Introduction 6. Customer tagging Internet service providers usually use some sort of an IPAM (IP address management) system to manage and allocate IP blocks within their IP space. At a given point in time, every active IP address should be allocated to a specific customer. IP allocations can change over time, and some IP allocations can be dynamic, e.g. with RADIUS, where users get assigned an IP for a limited period of time from a shared pool of IPs. To accurately track and account for traffic volume per customer, every flow record should be associated with the appropriate customer according to the IPAM system. Customer allocations in the IPAM system are usually associated with some type of unique identifier that is understood by the billing system. By tagging flows using this identifier, the correct customer can be charged when the records are summarized and imported into the billing system. The system must support live tagging of flows as they arrive. The tag should follow the flow throughout the system and also be part of any aggregate, such that it is possible to separate the traffic volume of two customers that shared the same IP during a given billing period. It must be possible to push updated tags to the system. 7. Destination-sensitive billing In Iceland, the dominant ISP billing model is to charge customers only for foreign traffic that originates or terminates outside of Iceland. Domestic traffic, exchanged within Iceland, is free. This is because submarine cables are expensive and foreign bandwidth is still scarce relative to domestic bandwidth. The system must provide features to distinguish and separately account foreign and domestic traffic. It must work both for smaller ISPs that do not have their own foreign links and purchase foreign transit from a larger upstream provider, and for larger ISPs that have their own dedicated foreign links Contribution In this thesis, we present a new flexible high-performance NetFlow accounting system. It meets all the key requirements laid out in previous section and provides several key features that are not present in other flow accounting systems. 4

17 1.4. Related work Our first contribution is a flow collector that can distribute flows to multiple backends in a configurable way. Unlike other systems (e.g. flow-tools and flowd), our system separates the collection and capture (storage) of flows. This separation allows us to distribute the processing of flows over multiple machines. Furthermore, the collector is stateless and a large ISP could run multiple redundant collector replicas for increased reliability and scalability. Finally, the collector can pre-process the flows and apply common business logic, such as filtering and tagging. Our second contribution is a compressed customer-oriented database for storing flows. The database is designed to sustain high-insertion rates and provide fast access to customer flows and traffic summaries without a sequential scan of all the flows. Other systems, e.g. flow-tools and flowd, store flows in flat binary files and extracting customer flows requires scanning all the flows Related work nfdump [15], silk [11] and flow-tools [8] are the most popular tools for storing flows. When it comes to data retention and flow accounting, these systems have several shortcomings. First, all of them store flow records in compressed flat binary files. They do not provide any sort of indexing. As a result, the entire set of flows needs to be sequentially scanned (and decompressed) to extract the flows of a given customer. Secondly, they do not provide any facilities for maintaining higher level summaries of customer traffic. Finally, these tools are designed as monolithic collectors that dump flows to disk. They provide limited options for distributing work across multiple machines. Column-oriented databases using bitmap indexing have been proposed for indexing flow data [6, 9, 19]. A column-oriented database stores its data vertically in columns rather than rows. The values in a given column are stored contiguously, which often improves compression ratios due to data similarity. Furthermore, a query only needs to read the columns that appear in the query, leading to less I/O when compared to row-oriented databases, especially when dealing with rows with a large number of attributes, such as flow records. A bitmap index can be constructed for a given column and queries can be answered by performing bitwise logical operators on the bitmaps. Bitmap indexes provide good performance for large read-only datasets [19], which makes them especially convenient for flow data. The proposed column-oriented flow systems offer a more general solution to the flow 5

18 1. Introduction extraction problem than our system, but none of them meet all of our other requirements. The column-oriented systems are primarily designed for fast offline analysis of flow data, and they lack essential flow accounting features such as NetFlow v9 support, an extensible storage format, online customer identification/tagging and destination sensitive billing Structure of thesis The remainder of the thesis is organized as follows. Chapter 2 reviews the necessary background information on flow-based monitoring and the NetFlow technology. Chapter 3 describes the architecture, design and implementation of the qflow system. Chapter 4 presents our evaluation and experimental results. Finally, Chapter 5 concludes and outlines future work. 6

19 2. Flow-based monitoring This chapter describes flow-based monitoring and the NetFlow technology Network monitoring The three most common approaches to network monitoring are: SNMP, packet traces, and network flows. Here we will briefly overview each one and discuss their applicability to data retention and traffic billing. Simple Network Management Protocol (SNMP) is the de facto standard protocol used for managing and monitoring network devices such as routers and switches. Devices run an SNMP agent (server) which exposes various knobs and metrics, such as CPU load, memory usage, and interface counters. Clients collect metrics by polling SNMP servers at a regular frequency. Within ISPs, SNMP is used to monitor link utilization and, in some cases, it is also used for billing, e.g. billing based on 95th percentile bandwidth is common for wholesale and large links. But SNMP is not suitable for data retention or destinationsensitive billing because it only provides a coarse-grained overview on link utilization and traffic volume. It does not provide any information on the traffic itself, such as the source/destination IP. In contrast to SNMP, packet traces provide a complete view of the traffic. Packet headers are collected by passively tapping links and feeding the packets to a nearby server for processing. Packet traces are popular within the network research community because they provide the greatest level of detail for studying traffic behavior. However, using packet traces for network accounting and data retention is impractical because it is expensive, both in terms of the hardware for collecting traces on high-speed links, and the costs associated with storing traces over a long period of time. Moreover, the method is unecessarily expensive because packet traces provide much more detail than is required for accounting and data retention. Network flows provide a good compromise in terms of cost and detail. Most net- 7

20 2. Flow-based monitoring work devices support exporting network flow information to an external collector (server). Flows contain less detail than packet level traces, since they do not provide information on individual packets. Instead, flows provide a connection-level view of the traffic. A network flow record summarizes a set of packets that share common header attributes and keeps track of aggregate statistics, such as the total bytes and packets belonging to each flow. The rest of this chapter will discuss network flows and flow-based monitoring in greater detail Flow probes Overview Flow-based traffic measurement methods are based on monitoring network traffic in terms of flows. Packets are observed at an observation point by a flow probe which aggregates packets into flows and exports flow records to an external flow collector. An observation point is a location in the network where packets can be observed, e.g. an interface on a router. Flow probes are usually embedded into flow-capable network devices, such as Cisco routers, but it is also possible to run a standalone software probe on a Linux server, e.g. nprobe [5], that observes packets using a network tap or port mirroring on a switch. A flow is defined as a set of IP packets passing through an observation point during a certain time interval and sharing a set of common properties (the flow key), e.g. source/destination IP, source/destination port, input/output interface, protocol [4]. The flow key can be configurable or fixed, depending on the underlying flow export protocol. Older flow protocols, such as Cisco s NetFlow versions 1 to 8, use a fixed flow key and some of the NetFlow versions differ only in which flow key they use. Newer protocols, such as Cisco s NetFlow version 9 and IETF s IPFIX, are designed to be extensible. They use a flexible template-based approach that allows the user to specify flow keys ( templates ) and multiple flow keys can exist within the same flow probe. 8

21 2.2. Flow probes Traffic Input interface Router Output interface Inspect packet Source IP address Destination IP address Source port Destination port Protocol... Update flow record Flow cache Flow key # pkts # bytes <flow key> Figure 2.1: Flow probe internals The flow key specifies the granularity of the aggregation that happens in the flow probe. For each observed flow, the probe maintains a flow record that includes counters such as the total number of packets/bytes in the flow and the time of the first and last packet in the flow. Active flow records are maintained in a flow cache and the probe updates the appropriate flow record for each observed packet. A small flow key will result in fewer flows being tracked and less memory usage, but it will also provide less detail on the observed network traffic. Flow records have a limited lifetime at the probe. A flow record is created when the first packet belonging to that flow is observed, and it expires when one of five events occurs: 1. The idle timeout of the flow when there has been no activity for a long time. 2. The natural end of the flow, e.g. TCP connection is closed. This only applies to protocols that have connection-oriented semantics, such as TCP. 3. The active timeout of the flow. This ensures that long-lived active flows are regularly exported. 9

22 2. Flow-based monitoring 4. The flow cache is full. The details depend on the probe implementation, but it might, for example, evict the oldest flow record to make room for new entries. 5. Overflow protection. The flow record counters have a fixed size and a record is expired if an update would cause an overflow to occur. An expired flow record is queued for export to an external collector. For the sake of efficiency, the queued records are usually batched in flow export packets according to the semantics of the underlying export protocol, e.g. each NetFlow version 5 export packet can carry 30 flow records Flow export Each flow probe (also known as flow exporter) is configured to export flows to one or more collectors at a given IP and port. Most flow probes, especially those embedded in routers, use a simple exporting strategy: each packet is sent to every configured collector. However, some devices, such as Cisco routers, only support two collector destinations. If more destinations are required, then it must be implemented on the collector side. Export packets are usually transported from probe to collector using either UDP or SCTP. Originally, NetFlow only supported UDP and therefore many older devices still only support using UDP. However, exporting via UDP has several major disadvantages [2]: 1. UDP is congestion unaware. The exporter sends packets as fast as it can generate them, without regard to available bandwidth or how fast the collector is consuming them. 2. UDP is unreliable. Packets can be lost, duplicated, or delivered out of order. The collector must be robust and tolerate such events. 3. UDP is vulnerable to spoofing and packet insertion. Due to the lack of a handshake, an attacker could blindly spoof packets from the exporter. SCTP is a reliable message-oriented transport layer protocol. It has many advantages over UDP, including: 1. SCTP is congestion aware and provides congestion control similar to that for TCP. Messages are buffered as until they can be sent. 10

23 2.2. Flow probes 2. SCTP is reliable. Messages are buffered until they have been acknowledged by the collector. Lost messages are retransmitted. 3. SCTP uses a 4-way handshake with signed cookies, which prevents spoofing. Furthermore, SCTP opens the door for more advanced features in the flow probe. For example, it is possible to configure a backup collector in Cisco routers. The router sends SCTP heartbeat messages to the primary and backup collectors. If the primary goes down, it will start exporting to the backup collector. This is not possible when exporting via UDP because it is a fire-and-forget protocol. It is connectionless and the probe communication is only in one direction: the flow probe talks to the collector, but the collector doesn t talk back. As a result, SCTP is generally preferred when it is available. This is especially true when flow collection serves a critical business function, such as when customers are charged for usage based on flow data. In that case, lost records equal lost revenue Packet sampling Asides from the tradeoff between flow detail and memory, there is also a tradeoff between flow accuracy and CPU cost. As the packet rate grows, it can be expensive for routers to examine every packet. Most flow probes support packet sampling to reduce measurement overhead. Instead of processing every packet, the probe will sample packets with a probability p, i.e. the flow cache is updated only for a fraction p of all packets. Later, the external flow collector can renormalize (invert) the flow counters by multiplying by 1/p to obtain an unbiased estimator of the original [7]. The 95% confidence interval of the estimate percent error can be approximated by [16]: 1 %error 196 (2.1) c where c the number of bytes that belong to the traffic class, e.g. a single customer. Since the error is a function of the number of samples used to make the estimate, the accuracy can be increased by increasing the number of samples. This can be done in two ways: 1. Increasing the sampling probability 2. Sampling over a longer period of time 11

24 2. Flow-based monitoring 100% 80% % Error 60% 40% 20% 0% Number of samples in class Figure 2.2: Relative sampling error Figure 2.2 shows the number of samples required to obtain a given % error. In a typical billing application, where a service provider charges customers, as identified by IP address, for byte usage, the objective is to determine from all the packets traversing the network during the billing period (usually 1 month) how many bytes belong to a particular customer. In that case, the class in the error equation would correspond to a customer. A common ISP billing strategy is to include an allowance, say 10 GB, and charge users for each additional GB. If the sampling probability is 1/100 and the observed interface has an MTU of 1500 bytes, then at least 70,000 samples will be taken for a customer that consumes 10 GB. Using the equation above, the error would be within 0.6%. The lower bound is based on the average packet size being equal to the MTU, which is the worst case, since it will result in the fewest number of packets. For real traffic, the packet size distribution would most likely follow a bimodal distribution with very small (40 bytes) and very big (1500 bytes) packets being most common [21]. Consequently, the expected error would be even lower than 0.6%. Finally, the customer charge can incorporate the confidence interval to counter the sampling error. Billing by the lower bound of the confidence interval would ensure that no customer is overcharged. 12

25 2.3. Cisco NetFlow 2.3. Cisco NetFlow History NetFlow is a set of protocols developed by Cisco for flow-based monitoring. Although originally developed by Cisco, NetFlow has become the de-facto industry standard and is currently supported by a wide range of network devices from different vendors, e.g. Cisco, Alcatel and Juniper devices. Cisco has released nine versions of the NetFlow protocol, designated as NetFlow v1 to v9. In recent years, IETF, the Internet standards body, has defined a new protocol called IPFIX in an attempt to unify and standardize on a common flow protocol. IPFIX is heavily based on NetFlow v9 but also includes several new extensions. Informally, IPFIX is also known as NetFlow v10 because its export format is compatible with the NetFlow versioning scheme and it uses the NetFlow version identifier 10. Currently, the most widely used and supported versions of NetFlow are v5 and v9. The versions before v5 are deprecated and the versions between v5 and v9 are mostly small variations of v5 that are seldom used or supported. Each variation required a new version because all NetFlow versions before v9 used a fixed export format and flow key. NetFlow v9 is designed to be future-proof. It is flexible and extensible: record formats are defined using templates and new fields can be added without changing the protocol itself. NetFlow v5 is still the most popular format, but it is slowly being replaced by NetFlow v9. The main issue with NetFlow v5 is that it is not extensible and it lacks support for IPv6. As Internet service providers move to adopt IPv6, they need to be able to monitor and charge for IPv6 traffic as well. In the sections below, we will describe the inner workings of NetFlow v5 and v Version 5 NetFlow v5 defines a flow as an unidirectional sequence of packets that share the following predefined key fields [3]: 1. Source IP address. 2. Destination IP address. 3. Source port for UDP/TCP, otherwise 0. 13

26 2. Flow-based monitoring 4. Destination port UDP/TCP, type/code for ICMP, otherwise IP protocol type, e.g. 6 for TCP, 17 for UDP. 6. IP Type of Service (ToS). 7. Ingress interface, using the SNMP index of the interface. The values of the key fields differentiate one flow from another. Flow records also contain non-key fields that provide additional information about the flow, e.g. the nexthop IP address, but a change in the value of a non-key field does not create a new flow. It depends on the implementation, but in most cases, the values of non-key fields are decided by the first packet in the flow. The NetFlow v5 export format is based on a fixed-length binary format with a fixed set of fields. Figure 2.3 shows the structure of a typical NetFlow v5 packet exported over UDP. Each packet contains a common header followed by a number of flow records. All fields are encoded using big-endian byte order. Table 2.1 shows the fields present in a NetFlow v5 header. Table 2.2 shows the fields present in a NetFlow v5 flow record. IP header UDP header version no. records probe uptime probe clock residual nanos sequence nr.... Flow header Flow record Flow record... Flow record source IP destination IP nexthop IP input interface output interface no. packets no. bytes... Figure 2.3: NetFlow v5 export packet 14

27 2.3. Cisco NetFlow Name Type Description version uint8 NetFlow export format version number, always 5 count uint16 Number of flow records in this export packet (1-30) uptime_ms uint32 Time in milliseconds since this device booted unix_secs uint32 Time in seconds since epoch unix_nsecs uint32 Residual nanoseconds since epoch flow_sequence uint32 Sequence counter of total flows exported engine_type uint8 Type of flow-switching engine engine_id uint8 Slot number of the flow-switching engine sampling_interval uint16 Sampling mode (2 bits) and interval (14 bits) Table 2.1: Format of NetFlow v5 header When exporting over UDP, the sequence counter can be used to detect lost flows and duplicates. However, the detection must also take into account that packets can be received out of order and the flow probe might restart, which would reset the sequence counter. Name Type Description srcaddr uint32 Source IP address dstaddr uint32 Destination IP address nexthop uint32 IP address of next hop router input uint16 SNMP index of the input interface output uint16 SNMP index of the output interface dpkts uint32 Total number of packets in the flow doctets uint32 Total number of Layer 3 bytes in the packets of the flow first uint32 Uptime at start of flow last uint32 Uptime at the time the last packet of the flow was received srcport uint16 TCP/UDP source port number or equivalent dstport uint16 TCP/UDP destination port number or equivalent pad1 uint8 Unused (zero) bytes tcp_flags uint8 Cumulative OR of TCP flags prot uint8 IP protocol type (for example, TCP = 6; UDP = 17) tos uint8 IP type of service (ToS) src_as uint16 Autonomous system number of the source dst_as uint16 Autonomous system number of the destination src_mask uint8 Source address prefix mask bits dst_mask uint8 Destination address prefix mask bits pad2 uint16 Unused (zero) bytes Table 2.2: Format of NetFlow v5 record 15

28 2. Flow-based monitoring Version 9 Flow header NetFlow v9 is the successor to NetFlow v5. It abandons the fixed record format in favor of a flexible template-based system which allows new record types to be defined. Each NetFlow v9 message starts with a common flow header. It is similar to the one used for NetFlow v5 but it has been slightly refined. Table 2.3 shows the fields present in the NetFlow v9 header. The source ID field is new, but serves a similar purpose to the engine type/id fields found in the NetFlow v5 header. It identifies the NetFlow process within the exporter device. For instance, when a router has multiple line cards that are running separate NetFlow processes, the collector can use the source ID to separate different export streams coming from the same device/source IP. This is especially important in NetFlow v9 because the sequence number and template state is scoped to the observation domain, i.e. the uniqueness of template IDs is local to the observation domain. Therefore, the collector must maintain separate state for each observation domain (source IP + source ID). Name Type Description version uint16 NetFlow export format version number, always 9 count uint16 Total number of records in the export packet uptime_ms uint32 Time in milliseconds since this device booted unix_secs uint32 Time in seconds since epoch sequence_number uint32 Incremental sequence counter of all export packets source_id uint32 Identifies the Observation Domain Table 2.3: Format of NetFlow v9 header 16

29 2.3. Cisco NetFlow Flowsets The flow header is followed by one or more flowsets, as shown in Figure 2.4. Flow header Flowset Flowset Flowset... Flowset Flowset Figure 2.4: Structure of NetFlow v9 export packet There are four types of flowsets: 1. Template flowset: Contains one or more template records. Each template record describes the type and length of individual fields within subsequent flow records that match the template. 2. Data flowset: Contains a template ID and one or more flow records. The records cannot be decoded without the right template. 3. Options template: Special type of template flowset that describes the format of options data records. 4. Options data: Special type of data flowset that contains options data records. Rather than supplying information about flows, these records describe metadata about the NetFlow process itself, e.g. the sampling interval. Different types of flowsets can be interleaved in the same packet in any given order. Every flowset starts with a common flowset header. It contains two fields: the flowset ID, which identifies the flowset type, and the flowset length, which contains the total size of the flowset including the flowset header itself. The remainder of the flowset depends on its type. The following types are supported: 0: Reserved for template flowset. 1: Reserved for options template : Reserved for future use : Data flowsets. 17

30 2. Flow-based monitoring Template flowset The template flowset contains a set of template records. Each template record describes the format (template) of subsequent flow records in data flowsets with the given template ID. Figure 2.5 shows the structure of a typical template flowset. The NetFlow v9 standard (RFC3954) defines 79 field types. The standard types include all the fields of NetFlow v5 and new fields such as MAC addresses, VLAN IDs, MPLS labels, and IPv6 addresses. The field type attribute in a template record is a 16-bit short, so there is plenty of room for future types. When the collector receives a template record, it needs to store the template to be able to decode future data flowsets that match the given template ID. Flow probes will generally send templates periodically to refresh the collector. Templates are not persistent across flow probe restarts. Consequently, if the collector receives a new template definition for an already existing template ID, it must override the previous definition. Flowset header Flowset ID: 0 Length Template record Template record Template record Template record Template ID Field count Field 1 Field 2... Template ID Field count Field 1 Field Template ID Field count Field 1 Field 2... Field N Field N Field N Figure 2.5: NetFlow v9 template flowset Data flowset After sending templates to the collector, the flow probe can transmit flow records using the template. The flowset ID in a data flowset header identifies the template ID required to decode the flow records contained within the flowset. Figure 2.6 shows the structure of a typical data flowset. 18

31 2.4. Observation points Flowset header Template ID Length Flow record Flow record Flow record Flow record Field 1 value Field 2 value... Field 1 value Field 2 value Field 1 value Field 2 value... Field N value Field N value Field N value Figure 2.6: NetFlow v9 data flowset Storage requirements According to Cisco, the volume of NetFlow export data is estimated at roughly 1.5% of the actual traffic observed. The average customer in Iceland uses 60 GB per month, which amounts to roughly 1 GB of NetFlow data per month. Another way to estimate the NetFlow volume is to use the flow export rate. NetFlow v5 uses a fixed-length export format with a 23 byte flow header, followed by up to 30 flow records of 47 bytes each. Assuming that a flow probe waits for 30 records before sending an export packet, we can calculate the NetFlow v5 volume for a flow rate R as follows: 23R + 47R (2.2) 30 For example, given a flow rate R of 50K per second, the resulting NetFlow v5 volume would be 2.27 MB per second or roughly 8 GB per hour Observation points Edge deployment Internet service providers that charge customers for network usage usually deploy NetFlow on the edges of the network, where traffic originates or terminates, as opposed to deploying NetFlow on backbone/core routers. The deployment plan needs to be comprehensive enough to capture all traffic required for billing and data retention, but it must also ensure that the same traffic isn t counted twice, which 19

32 2. Flow-based monitoring can happen for example when a packet traverses a path through the network that contains two active NetFlow probes, resulting in two separate flows being exported for the same traffic. The next section will describe such scenarios in more detail. Figure 2.7 shows an example service provider network with NetFlow deployment on the edges. The circles marked R represent routers and the smaller filled circles represent active NetFlow monitoring on the given network links. ISP network R R R R R R R Edge Core R Edge R Figure 2.7: NetFlow edge deployment Ingress/egress monitoring In general, NetFlow supports both ingress (inbound) and egress (outbound) monitoring. On routers, NetFlow is configured per interface in ingress and/or egress mode. Ingress monitoring accounts for all packets entering an interface, usually before any packet operations are performed by the device, such as ACLs or NAT. Egress monitoring accounts for all packets leaving an interface. Internet service providers that charge for network usage usually need to measure both the download and upload traffic of customers, but enabling both ingress and egress monitoring at the same device can result in duplication. Figure 2.8 depicts a router with six interfaces. Ingress monitoring is enabled on interface 4, and egress monitoring on interface 3. A packet that enters on interface 4 and is switched out on interface 3 will be examined twice. Depending on the flow probe and environment, this might result in two flows with the same information or a single flow with double traffic. NetFlow v9 contains a direction field that allows the collector to distinguish the two flows (one will be marked as ingress, the other as egress), but NetFlow v5 has 20

33 2.4. Observation points Interfaces Egress flow Router Ingress flow Figure 2.8: Example router with both ingress and egress monitoring enabled no such field. As a result, it is recommended not to mix ingress/egress monitoring on the same device when using NetFlow v5. Egress D C A B Ingress Figure 2.9: Provider network with both ingress and egress monitoring enabled The same general principle also applies to the network as a whole. In Figure 2.9, enabling ingress monitoring for packets that enter device B and egress monitoring for packets that exit device C would result in duplicate flows for packets traversing that path. 21

34 2. Flow-based monitoring Deployment strategies For usage-based charging, the objective is that every customer packet is accounted for once and only once. This effectively means that for any path through the network, there must only be one interface that counts the packet. Of course, it is also possible to filter duplicate flows at the collector, but it is easier and less error prone to deploy NetFlow in a way that avoids duplicate flows. A simple deployment strategy is to enable ingress monitoring on all outwards facing customer interfaces of edge routers. A customer packet will be examined at the customer interface where it enters the service provider network, but not at edge interface where it leaves the network, e.g. heading for the Internet. Likewise, a return packet would be examined when it enters the service provider network, but not at the customer edge interface. The two unidirectional flows resulting from a TCP connection could be exported from different devices, but together, they would provide a full view of any traffic going to/from customers Customer traffic With ingress monitoring on all customer/outwards facing interfaces, all flows have the same direction, regardless of whether they are in fact inbound or outbound customer traffic. It is up to the collector to determine the true direction with respect to the customer. This is often accomplished by maintaining a list of customer IPs (local prefixes) at the collector. When a flow is received, the collector will look up the source and destination IPs and determine which endpoint belongs to the customer. If the destination IP address belongs to the customer, then it is inbound (download) traffic with respect to that customer. Otherwise, it is outbound. In the case of local customer-to-customer traffic, both endpoints will be customer IPs. For one customer the flow will be classified as inbound traffic, but as outbound traffic for the other customer. Flow accounting systems would typically produce two accounting records for such a flow, one for each customer. 22

35 3. Design and implementation This chapter presents the architecture, design and implementation of the qflow system Architecture The system is organized as a pipeline that receives and processes flows. The flows are tagged, sorted, indexed, aggregated and stored in a customer-oriented flow database. Conceptually, the database contains a set of flow tables. Each flow table contains a set of flows, organized by customer, along with an index that summarizes the traffic volume per customer and points to the flows in the flow table. Network planning Traffic analysis Lawful intercept IPAM Tags Billing system Reporting Reports to customers NetFlow Collector Capture Indexing Aggregation Figure 3.1: An overview of the qflow system 23

36 3. Design and implementation The system has five parts: 1. Collector: receives flows from exporters, translates them into an internal format, applies tags and filters, and then sends the flows to a number of backends for further processing. 2. Capture: receives flows from collectors and dumps them to disk in a temporary spool directory. Rotates files periodically. 3. Indexing: picks up finalized flow files from the spool directory, transforms them into flow tablets and appends them to a given flow table. 4. Aggregation: maintains higher level aggregates (e.g. daily/monthly) of traffic volume per customer for each flow table. 5. Reporting: provides flow data to external systems, such as billing, reports for customers, etc. Each part will be covered in more detail in the sections that follow Collector Design The collector acts as a broker between flow exporters, such as Cisco routers, and flow backends that consume flows for use in a wide variety of applications. The collector receives NetFlow packets from exporters, extracts the flows, applies common logic such as filters, tags, and other business rules, and then exports the flows to backends for further processing. An overview of the collector is shown in Figure 3.2. The collector supports both NetFlow v5 and v9. It translates received flow records into a common protobuf-based [12] intermediate format that it uses internally during processing and externally when talking to backends. The backends are organized into groups and the collector distributes flows across the backends in a group. The collector configuration decides which flows to send to a given backend group. This design has multiple benefits: 1. It provides loose coupling. New applications can be developed, tested and 24

37 3.2. Collector deployed without re-configuring or changing the NetFlow devices that export flows. 2. It solves the problem with NetFlow devices that only support two export destinations. 3. It abstracts away the complexity of dealing with multiple NetFlow versions and transport protocols. 4. It uses TCP for publishing flows to backends instead of unreliable protocols such as UDP. 5. It allows common functionality such as flow filtering and tagging can be encapsulated in one place. 6. It enables flows to be distributed across multiple backends that don t need to run on the same machine as the collector. Backend group A Backend 1 flow protobuf Backend 2 Exporter NetFlow v5 / v9 Backend 3 Collector Backend group B Config Backend 4 Backend 5 Figure 3.2: Collector overview Flow format The collector transmits flows in a common intermediate format using Google Protocol Buffers [12], also known simply as protobuf. The protobuf library provides a 25