High-Density Network Flow Monitoring

Transcription

1 Petr Velan High-Density Network Flow Monitoring IM May 2015, Ottawa

2 Motivation What is high-density flow monitoring? Monitor high traffic in as little rack units as possible Why do we want high-density flow monitoring? Flow monitoring is deployed on many lines Number of flow probes is growing Management and operational costs are growing One probe per link does not scale Petr Velan High-Density Network Flow Monitoring / 18

3 Our Approach Use our custom made network interface cards to monitor multiple 10G links See what throughput can be achieved in one machine Test how advanced features of our NICs help the monitoring Identify performance bottlenecks Petr Velan High-Density Network Flow Monitoring / 18

4 Monitoring Setup The Testbed Petr Velan High-Density Network Flow Monitoring / 18

5 The Server Dell PowerEdge R720 server (2 rack units) 2 E v1 CPUs (8 cores, 3 GHz) 64GB DDR3 RAM (1600 MHz) Scientific Linux 6.5 with kernel 2 COMBO-80G cards Petr Velan High-Density Network Flow Monitoring / 18

6 COMBO-80G NIC FPGA based programmable hardware Two QSPF+ interfaces in 40 G or 4 10 G Ethernet mode 80 G per card PCI-Express gen3 x8 bus (64 Gb/s) Additional features: Accurate timestamps Hash based packet distribution Packet trimming Packet feature extraction into Unified Header Our setup allows to monitor G links Petr Velan High-Density Network Flow Monitoring / 18

7 COMBO-80G NIC Core 0... Core 7 CPU DMA PCI-Express RAM FPGA Channel Hash Data (UH or Packets) Select UH UH Packets Parsing Trimming Timestamp Assign QSFP+ QSFP+ COMBO-80G 8 x 10 GE Petr Velan High-Density Network Flow Monitoring / 18

8 Flow Exporter Architecture Thread 2N+1 Export Multi-threaded design Utilizes 2N + 1 CPU cores where N is number of ring buffers Thread N+1... Thread 2N Flow cache Thread 1... Input Thread N 0... N-1 Ring Bu ers Petr Velan High-Density Network Flow Monitoring / 18

9 Data Generation Spirent TestCenter hardware 1 10 G repeated to all 16 interfaces IPv4 UDP packets Packet sizes 64 B, 128 B, 256 B, 512 B Flow counts 2 11, 2 18, 2 21 Petr Velan High-Density Network Flow Monitoring / 18

10 Results The Measurement Petr Velan High-Density Network Flow Monitoring / 18

11 Basic Performance on Full Packets 120 x x 10 6 Maximum Ethernet Maximum PCIe 128 flows [Card 0] flows [Card 0] flows [Card 0] 128 flows [Card 1] flows [Card 1] flows [Card 1] Packets/s 80 x x x x x Packet size Full packet processing performance in packets/s. Petr Velan High-Density Network Flow Monitoring / 18

12 Basic Performance on Full Packets PCI-Express limit is reached only for the longest packets NUMA architecture affects the performance Number of flows has large impact mainly on short packets Petr Velan High-Density Network Flow Monitoring / 18

13 Hardware Accelerated Performance 250 x 10 6 Maximum Ethernet Maximum PCIe Full Packets Trimmed Packets Unified Headers 200 x 10 6 Packets/s 150 x x x x Packet size Packet processing performance in packets/s for 2 18 flows. Petr Velan High-Density Network Flow Monitoring / 18

14 Hardware Accelerated Performance Packet trimming ad Unified Headers help significantly Full throughput monitoring for 256 B and longer packets Packet trimming and Unified Headers solve the problem of PCI-Express throughput Petr Velan High-Density Network Flow Monitoring / 18

15 Impact of CPU Choice Comparison of the E with E Only 6 cores, 2 Ghz frequency 128 flows [E5-2670] flows [E5-2670] flows [E5-2670] 128 flows [E5-2620] flows [E5-2620] flows [E5-2620] 140 x x x 10 6 Packets/s 80 x x x x x Packet size Comparison for two different CPUs on trimmed packets Petr Velan High-Density Network Flow Monitoring / 18

16 Impact of CPU Choice Faster CPU helps greatly for less flows Large number of flows has greater impact on memory bus utilization Both CPU are doing well for longer packets Petr Velan High-Density Network Flow Monitoring / 18

17 Conclusions & Future Work Petr Velan High-Density Network Flow Monitoring / 18

18 Conclusions It is possible to monitor G links in one 2U box Hardware acceleration can significantly help to improve the performance PCI-Express can be limiting for commodity cards NUMA architecture must be taken into consideration Number of flows has significant impact on performance Petr Velan High-Density Network Flow Monitoring / 18

19 Future Work Test the performance with COMBO-100G cards (PCIe gen3 x16) High-speed expriment with application flow measurements Build better framework for measurements Different flow lengths More complex packets and flows Different packets in one flow Petr Velan High-Density Network Flow Monitoring / 18

20 Thanks for your attention High-Density Network Flow Monitoring IM May 2015, Ottawa Petr Velan