SMARTxAC: A Passive Monitoring and Analysis System for High-Speed Networks

SMARTxAC: A Passive Monitoring and Analysis System for High-Speed Networks Pere Barlet-Ros, Josep Solé-Pareta, Javier Barrantes, Eva Codina, Jordi Domingo-Pascual Advanced Broadband Communications Center (CCABA), Computer Architecture Department Universitat Politècnica de Catalunya (UPC) Jordi Girona 1-3, 08034 Barcelona, Catalunya, Spain {pbarlet, pareta, jbarranp, ecodina, jordi.domingo}@ac.upc.edu Abstract This paper presents SMARTxAC, a tailor-made passive monitoring and analysis system for high-speed IP networks. This system has been designed to operate at gigabit speeds without packet loss, while providing very detailed information about the network usage. The main difference between SMARTxAC and other monitoring systems and infrastructures for high-speed networks is that SMARTxAC has been explicitly designed to analyze the network traffic online, whereas in other approaches traffic analysis tasks are usually postponed to an offline stage. Since 2003, SMARTxAC is being used for continuously monitoring the Catalan National Research and Education Network (Anella Científica), and it has become a very powerful tool for its network operators to know in detail the usage of their network and to detect network faults, anomalies and attacks when they occur. We describe the main features of SMARTxAC and the architecture resulting from the experience we acquired after more than two years of continuous operation of the system and the feedback we have received from the network operators of the Anella Científica. We also discuss the design challenges we had to face to cope with the requirements of such a system. Keywords Traffic monitoring, Traffic analysis, Network anomaly detection 1. Introduction From the beginnings of the Internet, traffic monitoring and analysis systems have been consolidated as very helpful tools for classical network administration and management tasks, and currently are widely used by network operators and administrators. Network monitoring systems provide very valuable information for a wide range of tasks such as traffic engineering, network dimensioning, capacity planning, traffic control, fault diagnosis and troubleshooting, and network performance evaluation (e.g. delays and packet looses). At the same time, network researchers have relied on network monitoring as a methodology to collect and analyze network data from real network viewpoints to study and evaluate network protocols and the behaviour of the Internet traffic. Nowadays, novel and diverse usages for network monitoring systems are also emerging. Given the current proliferation of network attacks, viruses and worms, network monitoring systems are also being used for network security tasks. Examples of security tasks that can be carried out by network monitoring systems include intrusion and anomaly detection, worm and virus detection, and protection against other network attacks, such as Denial-of-Service, flooding attacks, exploit attempts, port scanning, etc. During the last years, the Advanced Broadband Communications Center (CCABA) of the UPC has been involved in several projects related to Internet traffic monitoring and analysis in the Spanish National Research and Education Network (RedIRIS), namely CASTBA, MEHARI [1] and MIRA [2, 3]. As a result of such an experience, we have developed a traffic monitoring and analysis system, called 1

SMARTxAC, in joint collaboration with the Supercomputing Center of Catalonia (CESCA). SMARTxAC is a tailor-made always-on passive monitoring and analysis system that operates at gigabit speeds without packet loss. It also includes a web-based graphical interface that offers numerous traffic statistics online. Since July 2003, SMARTxAC is being used for monitoring the Catalan National Research and Education Network (Anella Científica), which is managed by CESCA, and connects about fifty universities and research centres in Catalonia. SMARTxAC is continuously monitoring the link that connects the Anella Científica to RedIRIS, which constitutes its main connection to the global Internet. This link is built from a pair of full-duplex Gigabit Ethernet links by using load balancing techniques. The average load of this link during working hours is about 1.5 Gbits/sec (270 Kpackets/sec). A GPS-synchronized and anonymized IP header trace of this link was collected for the NLANR-PMA project and can be downloaded at [4]. A first version of SMARTxAC was also successfully tested on one of the NLANR OC192MON s located on the SDSC s TeraGrid cluster. The data collection was facilitated by a pair of Endace DAG 6.1 OC192c measurement cards [5] operating in 10 Gigabit-LAN mode. A set of results of this experiment and details about the measurement point can be found at [6]. The objective of this paper is to describe the architecture of SMARTxAC and present its main features, together with the design challenges we had to address to cope with the requirements of such a system. The rest of this paper is organized as follows: Section 2 briefly reviews the related work. Section 3 presents the architecture of SMARTxAC and describes its main components in detail. Section 4 describes its anomaly detection module and some possible improvements that are currently under development. Finally, Section 5 concludes the paper and discusses the future work. 2. Related work Network monitoring techniques traditionally have been classified into two main categories according to the methodology used to measure the network traffic and characteristics, namely active and passive monitoring. Active monitoring techniques consist of injecting measurement packets onto the network to infer its characteristics and performance parameters. The main advantage of active measurement techniques is their flexibility, because the measurement devices can be installed at the edge of the network (i.e. end hosts) without requiring instrumentation inside the network. There exists a large variety of active monitoring systems and infrastructures. The most representative ones include the National Internet Measurement Infrastructure (NIMI) [7], NLANR s Active Measurement Project (AMP) [8] and several tools developed by the Cooperative Association for Internet Data Analysis (CAIDA) [9]. Passive monitoring techniques consist of collecting and analyzing the traffic from operational networks, without injecting any additional traffic. However, passive monitoring presents several limitations. On the one hand, passive monitors must be deployed inside of the network. On the other hand, the difficulty of measuring highspeed links often makes traffic measurement and online analysis unviable in practice without using specialized hardware. Examples of passive monitoring systems and infrastructures include Sprint s IPMON [10], AT&T s Gigascope [11], NLANR s Passive Measurement Project (PMA) [12], CAIDA s CoralReef [13] and Intel s CoMo [14]. 2

The main difference between SMARTxAC and other passive measurement infrastructures and systems for high-speed networks, such as those deployed by NLANR and Sprint, lies in that SMARTxAC has been explicitly designed for online traffic analysis, whereas in other approaches traffic analysis tasks are usually postponed to an offline stage. Passive measurement systems also include simple tools based on the Simple Network Management Protocol (SNMP) and network monitoring software integrated into some routers and switches, such as Cisco s Netflow [15]. The main drawback of this approach is that often requires administrative access to network devices and traffic statistics are highly aggregated, so that it is not possible to obtain packet-level information and short-term congestion cannot be detected. 3. Architecture The design of SMARTxAC has been mainly driven by two main requirements. On the one hand, given the nature of our network scenario, the system has to be lightweight enough for collecting and analyzing online the network traffic from highspeed links without packet loss. On the other hand, it has to provide very detailed information about the network operation and usage. This information is very useful for helping network operators in their daily network management and administration tasks and, in particular, for detecting irregular network usage, network anomalies and attacks. Monitoring high-speed links by using either commodity or even specialized hardware is a challenging task. It involves the collection of gigabytes of traffic per second and the analysis and storage of terabytes of data per day. The problems of monitoring very high-speed links are well known by network researchers and developers of network monitoring applications. These problems are largely discussed in [16] and include limitations of current hardware, such as processing power, memory access speeds, disk bandwidth, and obviously storage capacity. In SMARTxAC we avoided these limitations by carefully designing the system to operate at gigabit speeds. We partitioned the main monitoring and traffic analysis tasks into several components according to their real-time constraints. For example, traffic collection tasks, which have strong real-time constraints, can be executed on a different computer than traffic analysis tasks, which usually are very expensive but have less real-time requirements. Finally, traffic analysis results are only presented under user request. Fig. 1 illustrates the three main components of the SMARTxAC platform that are described later in this section: the capture system, the traffic analysis system and the result visualization system. Due to performance reasons, each of them is executed on a different computer, although in other scenarios under lower traffic loads, they can run on a single computer. In order to provide useful information for network administration and management tasks, we have relied on the large experience of CESCA as the network operator of the Anella Científica. They provided us with very valuable knowledge about the statistics the system should offer in order to provide useful information, and which statistics were not needed or were rarely used in practice. It allowed us to apply very aggressive data reduction policies, early discarding unnecessary information, and consequently, significantly improving the performance and storage needs of our system. 3

Fig. 1. SMARTxAC platform overview A. Capture system We refer to capture system as the hardware and software components of the SMARTxAC platform that are used for traffic collection. Both Gigabit Ethernet links are tapped by using four optical splitters (one per link and direction) that passively send an entire copy of the traffic to an off-the-shelf computer equipped with a pair of Endace DAG 4.3GE cards [5]. The DAG 4.3GE cards are dual port, so that only two cards are needed for monitoring both links. The internal clocks of both DAG cards are synchronized via an external GPS source. The capture system basically collects packet headers at full-line rate, without packet loss, and performs a first aggregation of the traffic into traditional 5-tuple flows. This first aggregation stage allows us to perform an important reduction of the data to be sent and analysed by the traffic analysis system. However, as the traffic capture system has strong real-time constraints and has to avoid packet losses, more complex data reduction techniques, together with traffic analysis tasks, are delegated to the traffic analysis system. The capture software is based on a multithreaded implementation to exploit the presence of multiple processors on the capture system. In particular, it has been carefully designed to avoid packet losses, especially when dumping captured flows to disk or sending them to the traffic analysis system. Moreover, the software has been written to take the maximum advantage of the DAG cards by using directly the API provided by their manufacturers. However, our software also supports the libpcap packet capture library, so that it can be used with standard NIC cards as well, when monitoring links under lower traffic loads. B. Traffic analysis system The main task of the traffic analysis system is to aggregate the collected flows into a special kind of flows we have called classified flows, while computing several traffic statistics online. This aggregation is performed by translating the identifying values of the 5-tuple flows collected by the traffic capture system (source/destination IP addresses, ports and protocol) into more general and meaningful values (origins, destinations and applications, respectively). We refer to origins as the institutions connected to the monitored network, whereas destinations are considered as the external networks that Anella Científica is connected to (see Fig. 1). However, the system can be configured to support any other network scenario, since origins and destinations can be easily configured by giving their network prefixes or AS numbers. 4

This classification not only allows us to compute detailed statistics per institution and destination network, but also provides us valuable information about the nature of the traffic. For instance, P2P traffic that is routed to a commercial network (e.g. Espanix) is probably less academic than mail traffic routed to an academic or research network (e.g. RedIRIS or GÉANT). This information can be useful for charging and billing purposes in commercial networks. In particular, in National Research and Education Networks (NREN) it can be used for cost-sharing purposes or simple for knowing if users are using network resources and services in accordance with a given policy of use, considering that NRENs are usually financed with public funds. In SMARTxAC, a pricing matrix can be defined by setting a price or weight per byte according to its origin, destination, application and traffic direction as described in [17], but this feature is not currently used in our scenario. Although this classification process needs to perform some expensive operations, such as computing twice per flow the longest prefix match algorithm, it can be done online. The main advantage of this approach lies in drastically reducing the volume of data to be processed and stored onto disk. It makes the permanent storage of historical data feasible without losing valuable information about the network usage. For example, a header-only trace of the link we are currently monitoring would require more than 500 GB/day of disk space, while the total amount of data stored in our scenario by SMARTxAC usually does not exceed 30 MBytes/day. Thus, historical data about the Anella Científica s usage is being permanently stored since July 2003. However, if more detailed information is needed, for instance by network researchers or for intrusion detection purposes, the system can be configured to collect header or full packet traces as well. Moreover, some statistics (e.g. Top-N origin and destination IP addresses) are computed before this second aggregation stage, mainly because are bounded by N and do not significantly increase the volume of data to be stored. In addition, when a network anomaly is detected, as it will be discussed later, finer-grained information related to the anomaly is kept, since can be useful to analyze the causes of the detected anomaly. C. Available statistics and visualisation system During the classification process performed by the traffic analysis system, several statistics per institution are computed and stored onto disk. SMARTxAC includes a web-based graphical interface that allows network operators, as well as network administrators of the institutions connected to the network, to directly access to the traffic analysis statistics online. All figures and graphs are generated on demand when a user is interested in them, because SMARTxAC can offer more than 300 different combinations of graphs per institution and day. However, in order to preserve institution privacy, the access to these statistics is protected by password and data transmission is encrypted, so that institutions only can access to their own traffic statistics. A complete list of the figures and graphs available from the web-based interface can be found at [18]. As an example, here we mention only a few: the time-series plots of the link utilization per applications (Fig. 2) and destinations per applications (Fig. 3), Top-N IP addresses, protocols and ports (Fig. 4), time-series plot of the average packet size per application (Fig. 5) and anomaly detection graphs (Fig. 6). Users not only can access to the statistics that are being computed online by the traffic analysis system, but also to historical data, including the complete statistics of any particular day, and aggregated statistics and reports per week, month and year. 5

Fig. 2. Traffic per application time series Fig. 3. Traffic per application and destination Fig. 4. Top-25 ports Fig. 5. Average packet size time series (HTTP) 4. Anomaly detection The statistics provided by SMARTxAC are very useful to manually detect irregular usage or anomalies. Nevertheless, in order to facilitate to network operators the tough task of reviewing the network statistics and graphs looking for anomalies, our aim is to add some intelligence to our system, so that it could come to some conclusions by itself after having extracted and analyzed the collected traffic. Traditional intrusion detection systems (IDS) are usually designed to protect enduser systems or small networks. Most of their detection techniques are based on payload or system log analysis, since they assume that log information can be easily accessed or the traffic volume to be analyzed will be low enough to perform online payload analysis (e.g. signature-based IDSs). On the contrary, SMARTxAC is focused on the analysis of large networks and the measurement of high-speed links. Therefore, the anomaly detection method used by SMARTxAC must be lightweight enough to operate with strong real-time constraints. Under this assumption, SMARTxAC integrates a simple anomaly detection method based on traffic thresholds. It consists of establishing an upper and lower traffic limit per institution, so that the system will consider as anomalies those situations in which the traffic of an institution exceeds a predefined threshold. These thresholds can be configured manually or automatically by training the system with historical data, and can be independently set for both traffic directions in bits/second, packets/second and flow/second units. Moreover, all thresholds can also be associated to a particular interval, so that it is possible to configure different thresholds for different periods of time (e.g. day/night, workday/weekend). Once a threshold is exceeded, an alarm is triggered and notified 6

through the SMARTxAC web-based graphical interface, and a mail is sent to the network operator. When more than one alarm is triggered at the same time, the system is able to group all of them and show them as a unique alarm, to keep their number as low as possible. At this point, the system also stores some additional data concerning the anomaly during a fixed window of time. Thus, it is possible to carry out an offline analysis of stored data to infer the anomaly causes. Fig. 6 shows a daily flows/second time-series plot per application for a particular institution of the Anella Científica. Three alarms were raised at 8:06, at 9:36, and at 11:06, respectively, because the upper flows/second threshold was exceeded. For example, after reviewing the collected information, it was concluded that the last one was due to a TCP port scanning attack. Fig. 6. Flows/second threshold-based anomaly detector This simple technique is effective to detect those anomalies causing an important degradation of the network performance, although its detection power for other anomalies is relatively limited. For this reason, we are currently working on more effective, but still lightweight, techniques for detecting and classifying network anomalies. If the regular traffic of all the institutions that are connected to the network could be inferred, to detect anomalies would be as easy as comparing the regular traffic of an institution to its actual measured traffic. Nevertheless, because of inherent variations of Internet traffic by itself (e.g. burstiness, day/night, workday/weekend, etc.) to infer the regular traffic pattern of an institution is not an easy task. However, some anomalies can also be seen as unexpected changes on the traffic profile of an institution. Thus, unexpected changes may be easily detected by using traffic prediction techniques. With traffic prediction, anomalies can be detected when the actual measured traffic differs significantly from the predicted one for the same interval. Therefore, regular traffic profiles for each institution would not have to be explicitly known. Moreover, its implementation is lightweight enough to be used in a measurement system for high-speed networks, like SMARTxAC. Consequently, we are currently improving the SMARTxAC anomaly detection module to use adaptive traffic prediction to automatically detect network anomalies. Traffic prediction techniques will used in conjunction with other simpler techniques, like our current threshold-based detectors, in order to solve some limitations of adaptive prediction in the presence of anomalies that imply progressive and long-term traffic changes. Preliminary results of this work are described in [19]. 5. Conclusions and Future work In this paper we have described the architecture and main features of SMARTxAC, a passive monitoring and analysis system for high-speed networks. This system is the result of the experience we acquired in the participation in previous 7

projects and the guidelines and continuous feedback we have received from the network operators of the Anella Científica (CESCA) after more than two years of continuous usage of the system. Their experience and expertise in network administration and management fields have constituted an invaluable source of information that allowed us to improve the performance and functionalities of our system. The best proof of the success of SMARTxAC is that is daily used by CESCA to obtain detailed statistics about the Anella Científica s usage, and to detect network anomalies and attacks, network performance problems and network faults when they occur. In the design of this system we addressed the limitations of monitoring highspeed links, while focusing on providing useful information for CESCA and the network administrators of the institutions connected to the Anella Científica network. In order to address these challenges we had to trade generality for performance, resulting in an efficient but very specific tailor-made monitoring system that can be easily configured to be used in other network scenarios, but that is difficult to be used for other applications than the one it was designed for. For this reason, we are currently collaborating with the Intel s CoMo project [20] in developing a general-purpose network monitoring system that will allow the fast prototyping of network monitoring and network data mining applications. Currently, besides of improving the anomaly detection module, we are working on including in our system sampling measurement techniques, IPv6 support, and some recent proposals for detecting network applications based on heuristic techniques (e.g. [21]). Moreover, we plan to deploy more measurement points in the Anella Científica network and modify our traffic analysis system to analyze the traffic collected from several traffic capture systems at the same time. Acknowledgements This work is partially supported by the Supercomputing Center of Catalonia (CESCA), under the SMARTxAC agreement, and by the Spanish Ministry of Education, under contract TSI2005-07520-C03-02. References [1] P. J. Lizcano, A. Azcorra, J. Solé-Pareta, J. Domingo-Pascual and M. Álvarez- Campana. MEHARI: A System for Analyzing the Use of the Internet Services. Computer Networks, 31(21):2293-2307, 1999. [2] C. Veciana-Nogués, J. Domingo-Pascual and J. Solé-Pareta. Servers Location and Verification Tool for Backbone Access Points. In Proceedings of 13th ITC Specialist Seminar: IP Traffic Measurement, Modeling and Management, Sep. 2000. [3] C. Veciana-Nogués, J. Domingo-Pascual and J. Solé-Pareta. Cost-Sharing and Billing in the National Research Networks: the MIRA Approach. Terena Networking Conference, June 2002. [4] NLANR Special Traces Archive: CESCA-I Data Set. http://pma.nlanr.net/special/cesc1.html [5] Endace Measurement Systems. http://www.endace.com [6] NLANR Teragrid-II 10GigE real-time analysis. http://pma.nlanr.net/special/tera2.html [7] V. Paxson, J. Mahdavi, A. Adams and M. Mathis. An architecture for large-scale Internet measurement. IEEE Communications, 36(8):48-54, Aug. 1998. 8

[8] T. McGregor, H.-W. Braun and J. Brown. The NLANR network analysis infrastructure. IEEE Communications, 38(5):122-128, May 2000. [9] The Cooperative Association for Internet Data Analysis (CAIDA). http://www.caida.org [10] C. Fraleigh, S. Moon, B. Lyles, C. Cotton, M. Khan, R. Rockell, D. Moll, T. Seely and C. Diot. Packet-Level Traffic Measurements from the Sprint IP Backbone. IEEE Network, 17(6):6-16, Nov./Dec. 2003. [11] C. Cranor, T. Johnson, O. Spataschek and V. Shkapenyuk. Gigascope: A stream database for network applications. In Proceedings of ACM SIGMOD International Conference on Management of Data, June 2003. [12] NLANR: National Laboratory for Applied Network Research. http://www.nlanr.net [13] D. Moore, K. Keys, R. Koga, E. Lagache, K. C. Claffy. The CoralReef software suite as a tool for system and network administrators. In Proceedings of USENIX LISA, Dec. 2001. [14] G. Iannaccone, C. Diot, D. McAuley, A. Moore, I. Pratt and L. Rizzo. The CoMo White Paper. Technical report IRC-TR-04-017, Intel Research, Sep. 2004. [15] NetFlow Services Solutions Guide. White paper, Cisco Systems, 2001. http://www.cisco.com/univercd/cc/td/doc/cisintwk/intsolns/netflsol/nfwhite.pdf [16] G. Iannaccone, C. Diot, I. Graham and N. McKeown. Monitoring very high speed links. In Proceedings of ACM SIGCOMM Internet Measurement Workshop (IMW), Nov. 2001. [17] B. Stiller, P. Barlet-Ros, J. Cushnie, J. Domingo-Pascual, D. Hutchison, R. Lopes, A. Mauthe, M. Popa, J. Roberts, J. Solé-Pareta, D. Trcek, C. Veciana and L. Wolf. Pricing and QoS. Quality of Future Internet Services: COST Action 263 Final Report, 2856:263-291, Springer-Verlag, Dec. 2003. [18] The SMARTxAC project. http://www.ccaba.upc.edu/smartxac [19] P. Barlet-Ros, H. Pujol, J. Barrantes, J. Solé-Pareta and J. Domingo-Pascual. A System for Detecting Network Anomalies based on Traffic Monitoring and Prediction. Technical report, UPC-DAC-RR-CBA-2005-5, June 2005. [20] The CoMo Project, Intel Research Cambridge. http://como.intel-research.net [21] A. W. Moore and D. Zuev. Internet Traffic Classification Using Bayesian Analysis Techniques. Proc. of the ACM SIGMETRICS, Banff, Canada, June 2005. Vitae Pere Barlet-Ros received the M.Sc. degree in Computer Science from the Universitat Politècnica de Catalunya (UPC) in 2003. He is Assistant Professor and Ph.D. student at the Computer Architecture Department of the UPC. In 2004, he was visiting Ph.D. student at the National Laboratory for Applied Network Research (NLANR) for two months and intern at Intel Research Cambridge for three months. His research interests are in the fields of network measurement, traffic analysis and network performance evaluation. Josep Solé-Pareta was awarded his Master's degree in Telecommunication Engineering in 1984, and his Ph. D. in Computer Science in 1991, both from the Universitat Politècnica de Catalunya (UPC). In 1984 he joined the Computer Architecture Department of UPC, where currently is Full Professor in Computer Science and Communications. He did a Postdoc stage (summers of 1993 and 1994) at the Broadband and Wireless Networking Lab. of the Georgia Institute of Technology. His current research interests are in broadband Internet and high-speed and optical 9

networks, with emphasis on traffic engineering, traffic characterization, traffic management, QoS provisioning and MAC protocols for legacy and optical metro networks. Javier Barrantes received the M.Sc. degree in Computer Science from the Universitat Politècnica de Catalunya (UPC) in 2003. He is currently a Projects Graduate Scholarship Holder and Ph.D. student in the Computer Architecture Department, UPC. His research interests are in the fields of traffic measurements, modeling and prediction. Eva Codina received the M.Sc. degree in Computer Science from the Universitat Politència de Catalunya (UPC) in 2006. She is currently a Projects Graduate Scholarship Holder in the Computer Architecture Department, UPC. Her research interests are in the fields of traffic measurements, modeling and prediction. Jordi Domingo-Pascual is Full Professor of Computer Science and Communications at the Universitat Politècnica de Catalunya (UPC) in Barcelona. There, he received the engineering degree in Telecommunication (1982) and the Ph.D. Degree in Computer Science (1987). In 1983 he joined the Computer Architecture Department. He was visiting researcher at the International Computer Science Institute in Berkeley (California) for six months. His research topics are broadband communications and applications, IP/ATM integration, QoS management and provision, traffic engineering, IP traffic analysis and characterization, group communications and multicast. 10