Live Traffic Monitoring with Tstat: Capabilities and Experiences Maurizio M. Munafò Alessandro Finamore Marco Mellia Michela Meo Dario Rossi WWIC - Luleå, June 3, 2010
Outline Motivations Tstat - TCP STatistic and Analysis Tool Deployment Scenarios Features Layer-3 / Layer-4 Characterization Layer-7 Analysis and DPI LibTstat Outputs Gallery of Tstat Capabilities Conclusions
Traffic Classification & Measurement Why? Identify normal and anomalous behavior Characterize the network and its users Quality of service monitoring Traffic engineering Firewall tuning Pricing
Tstat at a Glance
Worm and Viruses? Did someone open a Christmas card? Happy new year to Windows!!
Anomalies (Good!) Spammer Disappear McColo SpamNet shut off on Tuesday, November 11th, 2008
New Applications P2PTV Fiorentina 4 - Udinese 2 Inter 1 - Juventus 0
TCP STatistic and Analysis Tool Tstat is a long term software project from the TLC Networks Group in Politecnico di Torino Project born to characterize the behavior of TCP connections Evolved to a full-fledged tool to monitor and analyze the traffic in IP networks Runs on most Linux/FreeBSD/NetBSD systems Working both as passive live sniffer and as offline trace analyzer Support for integration in other monitoring tools (libtstat)
Live Monitor Probe Just passively listen (sniff) the traffic passing on an operative link No need for special equipment Good performance with off-the-shelf hardware Manage hundreds of Mpbs with common PC hardware and integrated NIC Support for hi-end NIC cards Using Endace DAG able to manage a couple of Gbps of traffic with no fuss
Monitor Probe Setup LOCAL OUT IN EXTERN
Offline Traffic Analyzer Processing of already captured traffic traces for offline analysis Popular packet trace formats: pcap, erf, etherpeek Common compression formats: gzip, bzip, and 7zip
Tstat Workflow L7 L4 L3 Behavioral FSM DPI Pure DPI TCP/UDP IPv4/IPv6 Skype, Encrypted P2P Web, Mail, IM Peer-to-peer, P2P-TV #bytes, #flows, IP bitrate, packet length,
Tstat Workflow Layer 2 MAC encapsulation Tstat supports several MAC encapsulations (Ethernet, VLAN, MPLS), but no explicit statistic on them L4 L3 TCP/UDP IPv4/IPv6 Layer 3 IP IPv4 and IPv6 datagrams: anything different is ignored Layer 4 TCP and UDP Identification and complete characterization of TCP flows: flow length, lifetime, RTTs, window size, UDP flows: size, length, ports usage
Tstat Features L7 L4 Behavioral FSM DPI Pure DPI TCP/UDP Layer 7 Internet applications protocols L3 Pure Deep Packet Inspection Simple matching of known signatures in the packet payload P2P file sharing (emule/kad, Bittorrent), P2P-TV (Sopcast, PPLive, TVAnts) Finite State Machine DPI Mixes the Pure DPI with a FSM to consider packets in both directions Internet Protocols (HTTP,SMTP,SSL, SSH, ), Instant Messaging (MSN, Yahoo, Jabber), Web 2.0 Applications (Facebook, YouTube, RapidShare, Megaupload, ) Behavioral Classifier Classification of encrypted traffic through statistical properties Skype, Obfuscated emule/kad, Encrypted Bittorrent IPv4/IPv6
Tstat Outputs Connection Logs Text files reporting all of the relevant measures collected for the identified flows Histograms Text files collecting the empirical frequencies distributions for the collected parameters, saved at regular intervals RRD Round Robin Database Popular compact format to collect monitored statistics on several timescales. Used to monitor the probe through a CGI Web page Packed traces Dump of classes of packets into pcap traces for further elaborations
LibTstat Tstat can be compiled as a library to be linked with other measurement tools Simple API to pass packets to the Tstat engine The linking application can control all the aspects of the analyzed traffic Anonymization Packed payload Traffic filtering Successfully used by TIE (Univ. of Naples) and METAWIN (ftw.)
Where Tstat Lives
Gallery of Tstat Capabilities Live probe on the edge of the Politecnico campus network 1 Gbps link connecting to GARR, the Italian Research Network Traffic quite regular, with common workplace patterns (nine-to-five activity, no traffic in the weekends) Hybrid research/education/administration environment, so possibility of peculiar behaviors Traffic from February 2010
One Year of TCP Flows 40 flows [x1000] 20 0-20 -40 http smtp ssl/tsl unknown Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
IP Traffic bitrate [Mbps] 150 100 50 0-50 -100-150 -200-250 tcp udp Bitrate Mon Tue Wed Thu Fri Sat Sun Mon 250 Flows flows [x1000] 200 150 100 50 0-50 -100-150 tcp udp -200 Mon Tue Wed Thu Fri Sat Sun Mon
IP Bitrate bitrate [Mbps] 150 100 50 0-50 -100-150 -200-250 tcp udp Mon Tue Wed Thu Fri Sat Sun Mon
IP Flows flows [x1000] 250 200 150 100 50 0-50 -100-150 -200 Mon Tue Wed Thu Fri Sat Sun Mon tcp udp
Chat Sessions flows 450 400 350 300 250 200 150 100 50 msn act msn pre xmpp act xmpp pre yahoo act yahoo pre 0 Mon Tue Wed Thu Fri Sat Sun Mon
Tracked Flows flows [x1000] 140 120 100 80 60 40 udp tcp 20 0 Mon Tue Wed Thu Fri Sat Sun Mon
CPU Load %cpu load 90 80 70 60 50 40 30 20 10 max system+user system+user avg system avg 0 Mon Tue Wed Thu Fri Sat Sun Mon
TCP Bitrate per Application 150 100 50 bitrate [Mbps] 0-50 -100-150 -200-250 http bit+obf ssl/tls ssh other unknown Mon Tue Wed Thu Fri Sat Sun Mon
HTTP Bitrate per Application 50 0 bitrate [Mbps] -50-100 -150-200 -250 http-get megaupload facebook -300 youtube rapidshare other Mon Tue Wed Thu Fri Sat Sun Mon
Conclusions Mature tool for network monitoring and analysis Always on the cutting edge, adapting to the changes in the Internet and to the research trends in networking outperform [other] signature based tools used in the literature (IMC 2009) Web site http://tstat.polito.it/
Frågor?
Tack!