A Scalable High Performance Network Monitoring Agent for CERNET

Transcription

1 A Scalable High Performance Network Monitoring Agent for CERNET ZHANG, Hui CERNET Network Research Center Tsinghua University, Beijing, , China Abstract _ In a cost-effective way, collecting and analyzing data from such a nationwide operational network as China Education and Research Network (CERNET ) is an increasingly challenging task. This paper presents our experience designing and implementing a passive monitoring agent applicable to CERNET, based on which we are not only supporting our network intrusion detection system (IDS), network management system (NMS) for detecting and identifying signs of malicious activities, non-malicious failures, and other exceptional events in real-time, but providing anomaly information to accounting and billing system (ABS) so as to make it healthy. This agent is characterized by a high performance data collecting facility and a methodology of real -time data correlation and analysis. We can deploy a customized agent on a particular link of CERNET for monitoring network dynamically. We will discuss how to conflate, correlate, associate and refine measurement data to dis criminate anomalies such as DoS from normal traffic, and how to respond to the anomalies for the purpose of operational network's health. We conclude with experiences learned from the development and deployment of the agent and o ngoing research work. Keywords: Passive Monitoring, Traffic Collection, Packet Classification, Data Mining, Intrusion Detection I. Introduction This paper presents our experiences designing and developing an IP monitoring agent for the China Education and Research Network (CERNET). CERNET is the first and the largest nationwide education and research computer network in China. It has 12 global and regional channels connected with the United States, Canada, the U.K., Germany, Japan and Hong Kong SAR, and the international gateway bandwidth is over 220Mbps. Meanwhile, its domestic channels connected with other commercial carriers in China are usually up to 1Gbps in bandwidth. More than 1000 education and research institutions, 1.2 million PC's and 8 million users have connected to CERNET, constructing a 4-level hierarchy: campus network, province network, regional network and national backbone. CERNET backbone consists of over 60 OC-48 and OC-3 links, interconnecting 10 region-level nodes and 38 province-level nodes [6].The traffic volume ranges from tens _ This work was supported by the National Science Foundation of China (NSFC) under the grant No Send correspondence to Mr. ZHANG, Hui. Tel: +86 (0) Addr: Rm 210 Main Bldg, Tsinghua University, Beijing China. of Mb/sec on OC-3 province access links to more than 1Gb/sec on OC-48 national backbone links. The hierarchical structure of CERNET as shown in Figure 1 presents the location of our monitoring agents deployed across it. Fig. 1 CERNET hierarchical structure The CERNET backbone IP network provides connectivity over a geographically wide area. The back bone consists of a set of regional nodes connected by high bandwidth links, which are typically 2.5Gb/sec OC-48 links or 155Mb/sec OC-3 links. Each regional node in turn contains links, typically Gigabit Ethernet (GigE ), connecting to region access aggrega tion routers which provide access service for downstream networks. On the other hand, the BGP border routers which connect CERNET to Internet outside China and other major ISPs in China are generally connect ed to backbone via GigE links too. In case of CER NET topology, we can therefore easily deploy multiple monitoring agents across those GigE links to measure network traffic as well as monitor network anomaly and misuse. The monitoring agent is designed to focus on the following issues: 1) Support data rates up to 1Gb/sec. 2) Collect real-time IP packets from multiple carrier peering GigE links and regional access GigE links. 3) Classify ten thousands of IP packets into flows with timestamp with accurate enough fidelity. 4) Provide real-time measurements which characterize the status of link being monitored. 5) Filter out the anomaly signs according to a set of pre-defined signature in terms of multi-dimensions of

2 network flow traffic 6) Transfer the sampling IP packet data and flow data into data repository wherein previously unseen signatures are found off-line via data mining. This data correlation and analysis involves processing gigabytes or terabytes of data, and must have some facilities to handle unusual phenomena such as misuses and/or malicious network behaviors. 7) Provide identified records of traffic anomaly, network attacks, malicious mobile network worms, etc. for CERNET intrusion detection system (IDS), network management system (NMS), accounting and billing system (ABS) with daily network operation s health in mind. The remainder of the paper describes the details of how we address these design issues in our IP monitoring agent and presents some sample results that demonstrate the agent s capabilities. Section II presents architecture of this agent which consists of several major functional components and how to implement a scalable high performance agent via commodity hardware and software in a cost-effective way. Section III illustrates workflow of agent components and how they can work together in a tightly-coupled environment. Section IV presents traffic measurements, detected anomalies and attacks during last 3 months which demonstrate the capabilities of our agent. Section V discusses the cooperative relationship between our agents and IDS, NMS and ABS installed in CERNET. Section VI and VII concludes and discusses areas of ongoing and future research. II. System Architecture To satisfy the above-mentioned requirements, the design and implementation of our monitoring agent collects and aggregates packets to produce flow records with timestamp as shown in Table 1 and Table 2, respectively. This provides the fundamental traffic data concerning links being monitored. A set of components base their function on these meta-data to perform further processing and analysis. saddr daddr sport dport protocol service len_sd len_ds pkt_num_sd pkt_num_ds timestamp Table 1: Flow Record Definition source ip address(es) destination ip address(es) source port(s) destination port(s) ip protocol number(s) Description pre-defined application service(s) traffic from source to destination (in kilobytes) traffic from destination to source (in kilobytes) number of packets from source to destination number of packets from destination to source network time of first and last packet in the flow saddr daddr sport dport protocol Table 2: Captured Packet Record Definition source ip address(es) destination ip address(es) source port(s) destination port(s) ip protocol(s) size packet size (in bytes ) timestamp interface customized network time of the packet Description from which network interface the packet was captured one or more fields captured as needed In our opinion, these fields are the minimal but most useful subset of data available. By this definition of flow, we can describe many kinds of traffic stream pattern normal or anom alous behavior with great flexibility and convenience. With the worst-case, if we capture packet instead of flow, this resolves to about 20 bytes of data per packet, which means we are collecting about 5% of the total traffic given a 400-byte average packet size. At this point, we have reduced our input traffic by a factor of 20, on average. The remaining data is sent to user-space program for saving or doing in-depth traffic analysis and data mining. If we capture flow, we can reduce our input traffic by a factor larger than 20, on average, i.e., produce greater efficiency. A. Data Collection Passive measurement systems include Simple Network Management Protocol (SNMP)-based network traffic measurement tools [1], tcpdump [2] /libpcap [3], NetFlow [4], and CoralReef [5], etc. The typical passive measurement projects include [5] for CAIDA, [7] for NLANR and [8] for Sprint. Our agent makes use of a novel underlying collection facility called Linuxflow [9]. As with [9], this data collection component used by our monitoring agent suffices for gigabit Ethernet too. We implement this on Intel-based x86 hardware, i.e., a dual-processor 933MHz Pentium III machine with both a 64bit/133MHz PCI bus and a 64bit/66-33MHz PCI bus, and 2GB of PC-133 memory, two AceNIC Gigabit Ethernet cards with the Tigon II chipset, and two EtherExpress Pro 100Mb Ethernet cards for management. Also, we make heavy use of the source code for the Linux kernel 2.4.x [16]. At this moment, we modify the source code for further tuning of parameters of AceNIC Gigabit Ethernet s driver so as to improve its performance (rx_coal_tick) [16]. M eanwhile, TCP window size in the kernel is set to 256 Kbytes. Monitoring agent s architecture is shown in more detail in Figure 2. This figure shows a logical view of the hardware and software, and how processing is performed among multiple modules. Agent first collects traffic from the tapped network link via an optical splitter. The two

3 network interface cards one for TX and the other for RX traffic traveling across GigE link, both of which are working in promiscuous mode then collect traffic packets from the tapped network link, pass the data through a special-purpose packet capture protocol stack [9] to the kernel. A globally synchronized clock is provided by Fig. 2 Collect traffic through optical splitter network time protocol (NTP) from [17] upon which the fidelity of captured packet timestamp is up to O(millisecond ). B. Data Analysis All real -time data analysis is performed by user-level multithreading applications which depend heavily on SMP and caching facilities in Linux kernel for high performance. As we can see from Figure 2, packet meter /filter module interfaces the kernel via a newly-defined socket, namely AF_CAPPKT [9]. Some of the basic analysis, such as measuring packet size distribution, packet source/destination port distribution, packet protocol distribution, packets per second (PPS), and average packet size could be performed within it. And it can also filter packets in real time according to dynamically configured rules by using routing lookup algorithm. The output of packet meter, ten thousands of packet records as described in Table 2, are sent to a packet classifier which, among other things, is of the most importance functionality module in this monitoring agent. This module classifies incoming packets into one of several flows based on multiple fields of packets all packets of a flow are treated by a pre-defined rule and are processed in a similar manner in the next step. Routing lookup and packet classification are both important problems in the design and implementation of generic flow-aware routers [10] [11]. At this moment, we make use of a routing lookup algorithm Lulea [12] for longest prefix matching in software to support gigabit speed for packet filtering in monitoring agent instead of packet forwarding in router. This algorithm s objective is to minimize the storage requirements of their data structure, so that it can fit in the L1 cache of a conventional general purpose processor such as Pentium; our agent is based on dual Pentium III processors. On the other hand, we take advantage of a packet classification algorithm Recursive Flow Classification (RFC) [10][11] for packet classification on multiple fields for classifying packets into flows. There are a number of properties that we desire for the purpose of getting high performance and scalability from the algorithms we cho ose. 1) High speed. 2) Low processing time. 3) Flexibility in implementation. 4) Scalability in the number of fields. After collecting a large number of flows from packet classifier, the first work is to perform flow-based classification and filtering. We examine each individual flow and extract the following information: 1) TopN statistics concerning traffic volume, address uniformity, port settings, number of packets and flows, etc. 2) Traffic matrices concerning peering links, AS, and DNS domains etc. 3) Link utilization. 4) Any self-defined type of active flow as needed. 5) Traffic breakdown by protocols, applications (network services). 6) Routing information such as routing loop s and errors. Now we are capable of focusing on the outcomes of our interests which can be acquired from AF_CAPPKT, packet meter/filter, packet classifier and flow-based filters, and do further analysis by the methods we have adopted previously [13][14] and others [15] to conflate, correlate, associate and refine all above-mentioned measurement data to discriminate anomalies such as DoS from normal traffic. C. Data Repository and Mining The data repository is a large RAID-5 array for storage of all kinds of selective data from AF_CAPPKT, packet meter/filter, packet classifier and flow-based filters. It contains 1.8TB of disk space that consists of 10 Ultra SCSI 3 hard disks (180GB each). Data mining (DM) is a promising technique for IDS, especially for detecting novel attacks. By DM, our agent is not only able to give a profile of normal traffic, i.e. attack-free traffic, in the network, and then provide statistical anomaly -based detection results by comparing current data with previously -derived profile (profile-based), but also able to do further in-depth post-mortem analysis off-line on the bad or strange traffic flow and/or packet pattern for identifying novel rules, refining or deleting out-of-date ones to improve the capability and efficiency of current classifier and/or packet filter. Currently, we mainly look at the following statistical calculations for obtaining novel signs of anomalies: intensity measure, distribution measure, categorical and counting measure. Since the values obtained are from disparate distributions, we also need to normalize them. III. Workflow of Operations Figure 3 shows the operational workflow of all components of t his monitoring agent.

4 dport protocol 17(udp) 17(udp) 17(udp) size timestamp interface nic0 nic0 nic payload Signature (multiply fields on which classification is based) derived from anomaly packet record: {saddr: any/32 daddr: any netid/24 sport: any dport: 1434 ms-sql-m protocol: udp pkt size: 404bytes/pkt }. Figure 5 and Figure 6 illustrate the consecutive traffic anomalies on a link in CERNET when the latest large-scale network worm broke out on March 8th, 2003 [19]. The following packet pattern is one of the anomalous signatures detected by this agent. It accounts for 74.3% of the total inbound packets at about 3:00 P.M., and consumes network resources maliciously. Fig. 3 workflow of the monitoring agent IV. Monitoring Results In this section we present a sample of monitoring results to demonstrate its performance and capability of traffic anomaly detection. Figure 5 illustrate the sharp increase in link utilization when MS-SQL Slammer worm broke out globally at almost exact ly 13:30 P.M. (CST) on Saturday January 25th, 2003 [18]. Fig. 4 link utilization of a GigE (Slammer worm outbreak) Table 3: Packet Records of SQL Slammer Worm value saddr daddr / / /24 sport Fig 5 traffic mix of a GigE link Table 4: Packet Records of Worm value saddr

5 daddr / / /24 sport dport protocol 6(tcp) 6(t cp) 6(tcp) size timestamp :09: :09:01 interface nic1 nic1 nic :09:01 payload Packet record signature: {s addr: any/32 daddr: any netid/24 sport: any dport: 445 microsoft-ds protocol: tcp pkt size: 48bytes/pkt }. V. Cooperation with Other Systems Figure 6 shows how the monitoring agent cooperates harmoniously with NMS, IDS and ABS in CERNET. Fig. 6 agent cooperates with IDS, NMS and ABS VI. Conclusions We describe the CERNET passive monitoring agent that is capable of supporting Gigabit Ethernet data rate and presents its capabilities with several sample measurements in this paper. The advantages of this agent are as follows: 1) The design and implementation of it is suitable for deploying either on a SMP host or a tightly-coupled environment (cluster or server farm). 2) It equips a powerful collection engine for real-time packet capture on high -speed links. 3) It provides a daily traffic measurement report. 4) It performs packet pattern and flow-based traffic analysis on-line and off-line for network anomaly detection. 5) It adjusts dynamically the packet filter and classification rules, and flow filter rules for accurately identify network anomalies and attacks with great flexibility. 6) Its monitoring results are useful in many diverse systems such as IDS, NMS and ABS. VII. Ongoing and Future Work We are still devoting to enhance this agent s capability of finding novel filter rules and anomaly signatures with a formalized description scripts. However, the major part of our future work is to deploy much more monitoring agents throughout CERNET to detect, identify network anomalies and attacks, to correlate these which occur at different positions to study the possible context amongst them, and to depict dynamically the operational status of CERNET. References [1] W. Stallings. SNMP, SNMPv2, and SNMPv3, and RMON 1 and 2, Addison Wesley, 3 rd edition, [2] Tcpdump web page, [3] S. McCanne, V. Jacobson, The BSD Packet Filter: A New Architecture for User-level Packet Capture, In Proceedings of the Winter 1993 USENIX Conference, pp USENIX Association, January, [4] NetFlow services and applications. h/napps_wp.htm, 2002, Cisco white paper. [5] CoralReef web page. [6] CERNET. [7] T. McGregor, H. W. Braun, J. Brown, The NLANR Network Analysis Infrastructure, IEEE Communications, Vol. 38, No. 5, May, [8] C. Fraleigh, C. Diot, B. Lyles, S. Moon, P. Owezarski, D. Papagiannaki, F. Tobagi, Design and Deployment of a Passive Monitoring Infrastructure, Passive and Active Measurement Workshop (PAM) 2001, Amsterdam, The Netherlands, April, [9] Z. C. Li, H. Zhang, et al. Linuxflow: A High Speed Backbone Measurement Facility, accepted for Passive and Active Measurement Workshop (PAM) 2003, La Jolla, California, USA, April, [10] P. Gupta, N. Mckeown, Packet Classification on Multiple s, In Proceedings of ACM SIGCOMM 99, ACM, August, [11] P. Gupta, Algorithm for Routing Lookups and Packet Classification, Ph.D. Dissertation, Computer Science Department, Stanford University, December, [12] M. Degermark, A. Brodnik, S. Carlsson and S. Pink, Small Forwardin g Tables for Fast Routing Lookups, Proceedings of ACM SIGCOMM, pp3-14, October, [13] H. Zhang, G. Xu, Advanced Method for Detecting Unusual Behaviors on Networks in Real-Time, In Proceedings of ICCT-2000, Beijing, China, August, [14] H. Zhang, Z. M. Li, A Stream Pattern Based Live Traffic Analysis Model System in CERNET, In Proceedings of ISFST 2002, Wuhan, China, October, [15] G. M. Voelker, S. Savage, Inferring Internet Denial-of-Service Activity, USENIX Security Symposium, [16] L. Torvalds and Free Software Community. The Linux Kernel, September, [17] CERNET Network Time Service. December, [18] CERT Advisory CA MS-SQL Server Worm. January, 25, [19] CERT Advisory CA Increased Activity Targeting Windows Shares, March 11,