Who is Generating all This Traffic? Network Monitoring in Practice Luca Deri <deri@ntop.org>
Who s ntop.org? Started in 1998 as open-source monitoring project for developing an easy to use passive monitoring application. Several project spin-off Accelerated packet capture 1 and 10 Gbit packet capture NetFlow/sFlow probes Peer-to-Peer VPN 2
ntop.org at a Glance 3
Who is Using ntop Products? International Domestic 4
Some ntop Partners 5
Some Common Monitoring Questions [1/2] Top N talkers (those who transmit most traffic). Top N conversations (the host pairs that transmit most traffic between each other). Top N Applications (e.g. SAP is using 70% of the available bandwidth). Data volume per entity basis (link, location, region/subnet, class of users/cluster). 6
Some Common Monitoring Questions [2/2] Data volume and rates per AS (e.g. do we need to sign a new peering contract?). QoS marking per application or entity basis (e.g. does BGP reports us that we re sending the traffic on the optimal path?). Reports about traffic we don t expect to see on the network (e.g. why host X is sending IPX packets although we speak pure IP?). 7
Some Challenges SNMP is good for element management (e.g. router and server monitoring) but poor for traffic measurement. Not all routers/switches speak NetFlow/ sflow: we need to deploy soft probes. 1 and 10 Gbit networks can produce a lot of monitoring data: our monitoring apps must be able to handle all this traffic. 8
Networks are Changing [1/2] Wireless Edge Intranet Internet Central Mgmt Core Distribution Wired Edge 9
Networks are Changing [2/2] Without edge control there s no real network control. Central traffic monitoring isn t enough anymore: not all traffic flows through the center. Edge equipment is often very basic and it means that there s no visibility at the edge: this about this before purchasing your network equipement. 10
Typical Monitoring Deployment: LAN 11
Typical Monitoring Deployment: Internet Traffic 12
Typical Monitoring Deployment: Cloud and Intra-VM Monitoring 13
Some Lessons Learnt In order to monitoring the traffic we need to deploy a probe where the traffic is flowing. We need to make sure we can handle both NetFlow and sflow if we want to have complete network visibility. Cloud computing and server virtualization push us to monitor in-vm virtual networks. 14
What if we Upgrade to 10 Gbit? Be prepared to: handle 10x as much traffic as with 1 Gbit. be able to handle encapsulations (GRE, GTP) and tagging (MPLS, VLANs) in your monitoring software. buy 10 Gbit probes (costly and rare). Good news: 10 Gbit adapters are now commodity (< 1 000 Euro/port). nprobe support 10 Gbit already. 15
How can ntop help me? Central network monitoring console already integrated into Würth-Phoenix NetEye. Software NetFlow/sFlow probes that can be deployed across the network. 10 Gbit packet capture acceleration and filtering using nprobe. Ability to handle billion of flows with subsecond response time. 16
What is ntop? 17
Network Inventory 18
Traffic Trends 19
Host Health 20
VoIP Support 21
ntop Scripting using Python http://ntop.local:3000/python/hello.py HTTP(S) <html> </body>... </body> </html> handlepythonhttprequest(...) 22
Where is my Traffic Going To? 23
nprobe: IPFIX/NetFlow Soft Probe sflow NetFlow Packet Capture nprobe Flow Export Data Dump Würth-Phoenix NetEye Raw Files / MySQL / SQLite / FastBit 24
ntop on-the-go [1/2] Apple iphone is commonly used as mobile web pad. Accessing ntop information in mobility is often required by network administrators. The ntop web GUI can be accessed via Apple Safari, however a tighten and more comprehensive interface was necessary. Ability to control several ntop instances via a single device. Access traffic information as well as configuration information. Available (soon) on the AppleStore. HTTP(S) JSON ntop 25
ntop on-the-go [2/2] 26
nprobe: Main Features Ability to keep up with Gbit speeds on Ethernet networks handling thousand of packets per second without packet sampling on commodity hardware. Support for major OS including Unix, Windows and MacOS X. Full NetFlow v9/ipfix support V9 extensions: payload, network/application latency, VoIP, RTP. Ability to extend the probe with user-written plugins. BGP Peering with the router for full AS monitoring. Würth-Phoenix NetEye can be used as collector for nprobe. 27
nprobe: Network Performance and Response Time 28
nprobe: Network Awareness 29
Handling Billion Flows nprobe+fastbit vs MySQL Query MySQL nprobe + FastBit Q1 22.6 5.6 Q2 69 0.5 Q3 971 12.5 Q4 1341 48.2 Q5 2257 30.7 nprobe+fastbit vs nfdump nprobe+fastbit nfdump 45 1500 All measurements are in seconds 30
How can I Improve my Internet Presence? 31
Interactive Data Search 32
10 Gbit Wire-rate Traffic Monitoring with Commodity Hardware 33