Autonomous Ladislav Lhotka lhotka@cesnet.cz Martin Žádník xzadni00@stud.fit.vutbr.cz TF-CSIRT meeting, September 15, 2005
Outline 1 2 Specification Hardware Firmware Software 3 4 Short-term fixes Test Plan Phase 2 5
Outline 1 2 Specification Hardware Firmware Software 3 4 Short-term fixes Test Plan Phase 2 5
NetFlow version 9 defined in RFC 3954 (informational) flexible and extensible templates for data and options supports IPv6 independent of transport protocol (UDP, TCP, SCTP) basis for future IETF protocol (IPFIX)
Traditional Setup exporter collector BUT: your router may not support NetFlow v9 (or you:-) router is an L3 system vulnerable fixed set of templates rigid sampling schemes
Autonomous Probe probe exporter collector
Advantages stealth device no access to NetFlow v9 aware router needed additional flexibility: finer control of export data user-defined templates adaptive sampling pre-processing data in hardware correlation with other monitoring sources
Disadvantages routers see more traffic some data fields don t make sense (next hops) AS information can be supplied by running BGP
Outline Specification Hardware Firmware Software 1 2 Specification Hardware Firmware Software 3 4 Short-term fixes Test Plan Phase 2 5
Phase 1 Specification Specification Hardware Firmware Software Linux PC with hardware accelerator T-splitter with Gigabit Ethernet interfaces operate at 1 Gbps line rate (one-way) simultaneous monitoring of IPv4 and IPv6 NetFlow version 9 output cache for 64K flows user-configurable active and inactive timeout
The Hardware Specification Hardware Firmware Software COMBO6 motherboard connected to PCI COMBO-4MTX interface card with 4 metallic Gigabit Ethernet ports Both cards were developed for Liberouter/6NET project and used in SCAMPI Other interface card can also be used, e.g., COMBO6-4SFP with SFP transceivers
Specification Hardware Firmware Software Lhotka, Z a dnı k COMBO6
Specification Hardware Firmware Software COMBO6+COMBO-4MTX Lhotka, Z a dnı k
Firmware features Specification Hardware Firmware Software cache for 64 Kflows hash function: CRC-64 with random initialization probability of flow collision 10 4 configurable timeouts: active 0 1200s inactive 0 60s maskable key fields 50 MHz clock rate theoretical throughput limit 800 Mb/s (will be increased to 100 MHz soon)
Block Diagram Specification Hardware Firmware Software 1 Gb/s IBUF hash mem. hash FIFO active records empty items export HFE HASH HSRCH MAN SCTRL statistical FIFO
Device driver Specification Hardware Firmware Software module for Linux kernel flow records transferred in chunks via PCI DMA applications share a single physical memory area with flow records common ring buffer applications may lock multiple records zero copy (records mapped directly into the app. memory area) applications access the driver through a library (libcsflow) contains common functions aids debugging of applications
Exporter Specification Hardware Firmware Software generates valid NetFlow v9 data (RFC 3954) supports 8 templates: TCP, UDP, ICMP, OTHER for both IPv4 and IPv6 configurable period of re-sending templates Current limitations (to be removed): single collector bugs (bogus IPv6 byte counts,...)
Outline 1 2 Specification Hardware Firmware Software 3 4 Short-term fixes Test Plan Phase 2 5
Tests with FTAS collector live IPv4/IPv6 traffic from a large campus peak rate 14 Kflows/s collector software: FTAS by Tom Košňar results look reasonable
Performance Limiting factors small packets: HFE (415 Kpackets) big packets: system throughput due to 50 MHz clock (735 Mb/s)
Outline Short-term fixes Test Plan Phase 2 1 2 Specification Hardware Firmware Software 3 4 Short-term fixes Test Plan Phase 2 5
Pending improvements Short-term fixes Test Plan Phase 2 1 1 Gb/s line rate really soon 100 MHz clock Optimized HFE design is ready for testing 2 Finish the exporter NetFlow v9 options & option templates arbitrary number of collectors probe control IPv6 transport debugging
Planned tests for GN2/JRA2 Short-term fixes Test Plan Phase 2 CESNET ordered and will cover manufacturing expenses for 5 pcs of COMBO6+COMBO-4MTX (available in October)... and will lend them to interested JRA2 partners and provide support for installation and operation Recommended setup: connect to a dedicated switch port mirror traffic from other port(s) The device should be tested with various collectors heavy traffic loads real-life IPv4 and IPv6 traffic mix
Phase 2 Plans Short-term fixes Test Plan Phase 2 new motherboards COMBO6X for 64/66 PCI and PCI-X COMBO6E for Express PCI new interface cards special NetFlow version with more static RAM support for STM-16 10GE issue with 10 Gbps Phyters 1,6 Gb/s throughput 1 million flows more flexible records (MAC addr., VLAN tags) standard sampling sample and hold
Further Options Short-term fixes Test Plan Phase 2 adaptive sampling flow counting extension (Estan et al., SIGCOMM04) Combination with other functions: packet filtering payload scanning IPFIX implementation????
Outline 1 2 Specification Hardware Firmware Software 3 4 Short-term fixes Test Plan Phase 2 5
The results look promising so far. In combination with existing collectors it can be useful now and for various purposes. CESNET is looking for a commercial partner to finalize it as a product. A lot of room for improvements and further research. Source code for software and firmware available from our CVS: http://www.liberouter.org/cgi-bin2/cvsweb.cgi/ Guinea pigs needed!
The results look promising so far. In combination with existing collectors it can be useful now and for various purposes. CESNET is looking for a commercial partner to finalize it as a product. A lot of room for improvements and further research. Source code for software and firmware available from our CVS: http://www.liberouter.org/cgi-bin2/cvsweb.cgi/ Guinea pigs needed!
The results look promising so far. In combination with existing collectors it can be useful now and for various purposes. CESNET is looking for a commercial partner to finalize it as a product. A lot of room for improvements and further research. Source code for software and firmware available from our CVS: http://www.liberouter.org/cgi-bin2/cvsweb.cgi/ Guinea pigs needed!
The results look promising so far. In combination with existing collectors it can be useful now and for various purposes. CESNET is looking for a commercial partner to finalize it as a product. A lot of room for improvements and further research. Source code for software and firmware available from our CVS: http://www.liberouter.org/cgi-bin2/cvsweb.cgi/ Guinea pigs needed!
The results look promising so far. In combination with existing collectors it can be useful now and for various purposes. CESNET is looking for a commercial partner to finalize it as a product. A lot of room for improvements and further research. Source code for software and firmware available from our CVS: http://www.liberouter.org/cgi-bin2/cvsweb.cgi/ Guinea pigs needed!
The results look promising so far. In combination with existing collectors it can be useful now and for various purposes. CESNET is looking for a commercial partner to finalize it as a product. A lot of room for improvements and further research. Source code for software and firmware available from our CVS: http://www.liberouter.org/cgi-bin2/cvsweb.cgi/ Guinea pigs needed!