Flowgrind A TCP Traffic Generator for Developers Arnd Hannemann <arnd.hannemann@credativ.de> 05.10.2016 Arnd Hannemann credativ GmbH 1 / 29
Overview Introduction Flowgrind Architecture Example measurements Summary Arnd Hannemann credativ GmbH 2 / 29
Overview Introduction Motivation Flowgrind Architecture Example measurements Summary Arnd Hannemann credativ GmbH 3 / 29
Measuring network performance Tool requirements Wired Internet Backbone Background: Wireless Mesh Networks Creating load anywhere in the network Backbone Mesh Gateways Measuring TCP performance between any two nodes Backbone Mesh Routers Testing TCP variants Extensive list of TCP metrics Routing Mesh Clients Wired Connection Wirless Mesh Connection Non-routing Mesh Clients Wireless Access-Point Connection Separation of control and test traffic Arnd Hannemann credativ GmbH 4 / 29
Related works Feature Iperf Iperf3 Netperf Thrulay TTCP NUTTCP TCP UDP SCTP Other protocols Kernel statistics Interval reports Concurrent tests against same hosts Concurrent tests against different hosts Distributed tests Bidirectional test connections Test scheduling Traffic generation Control/test data separation Arnd Hannemann credativ GmbH 5 / 29
Motivation for a new tool Shortcomings of existing tools Client-server architecture hard to generate cross-traffic Separation of data/control traffic Arnd Hannemann credativ GmbH 6 / 29
Overview Introduction Flowgrind Architecture Architecture Client-server architecture RPC Example measurements Summary Arnd Hannemann credativ GmbH 7 / 29
Flowgrind Flowgrind Is a distributed network performance measurement tool Focuses on TCP testing/debugging Knobs to test TCP variants against each other Dump packet headers with libpcap Gathers TCP statistics from kernel (Linux/FreeBSD) Arnd Hannemann credativ GmbH 8 / 29
Terminology in Flowgrind Flows One data connection for each flow Flows have a source and a destination endpoint Test data can be sent in either direction Scheduling, flows can run sequentially, in parallel or can overlap Individual parameters for each flow Backbone Mesh Gateways Backbone Mesh Routers Routing Mesh Clients Wired Connection RPC Connection Flowgrind Controller Wired Internet Backbone Wirless Mesh Connection Flowgrind Daemon Non-routing Mesh Clients Test Connection Wireless Access-Point Connection Arnd Hannemann credativ GmbH 9 / 29
Problems with client-server architecture Wireless multi-hop network Arnd Hannemann credativ GmbH 10 / 29
Problems with client-server architecture Wireless multi-hop network Client Server Arnd Hannemann credativ GmbH 10 / 29
Problems with client-server architecture Wireless multi-hop network Client Server Arnd Hannemann credativ GmbH 10 / 29
Client-server architecture Overview Tools like iperf: split into client and server Flows can only be established between a client and a server, not between servers Architecture implemented in older versions of Flowgrind Problems with client-server architecture For multiple clients: external synchronization of test start is needed Potential different data handling in client and server (e.g. Thrulay) Arnd Hannemann credativ GmbH 11 / 29
Distributed architecture Controller (flowgrind) Parses the test parameters Configures all involved daemons Presents the results Backbone Mesh Gateways Wired Internet Backbone Daemon (flowgrindd) Backbone Mesh Routers Started on every test node Performs actual tests Measures performance metrics Routing Mesh Clients Wired Connection RPC Connection Flowgrind Controller Non-routing Mesh Clients Wirless Mesh Wireless Access-Point Connection Connection Test Connection Flowgrind Daemon Arnd Hannemann credativ GmbH 12 / 29
Remote Procedure Calls (RPC) RPC in Flowgrind Uses XML-RPC All calls initiated by controller, no RPC between daemons Can employ different IP address / interface to separate control and test traffic During a test Controller periodically queries all daemons for interval results Formats and prints results upon receiving them Arnd Hannemann credativ GmbH 13 / 29
Overview Introduction Flowgrind Architecture Example measurements Wireless Multi-Hop Network with Cross Traffic AWS example: congestion control algorithms Summary Arnd Hannemann credativ GmbH 14 / 29
Cross-Traffic in a Wireless Multihop Network Test scenario Measurement performed on testbed Two flows between two unique pairs of nodes Routes overlap, one bottleneck link Second flow started after a delay, stopping earlier Arnd Hannemann credativ GmbH 15 / 29
Topology A E C D Bottleneck link B F Flow 1 between nodes A and E Flow 2 between nodes B and F Arnd Hannemann credativ GmbH 16 / 29
Topology Arnd Hannemann credativ GmbH 17 / 29
WMN example: Flowgrind arguments flowgrind -n 2 -i 5 -O b=tcp_cong_module=reno -F 0 -H s=wlan0.mrouter16/mrouter16,d=wlan0.mrouter8/mrouter8 -T b=900 -F 1 -H s=wlan0.mrouter17/mrouter17,d=wlan0.mrouter9/mrouter9 -T b=300 -Y b=300 Arnd Hannemann credativ GmbH 18 / 29
WMN example: Output # ID begin end through RTT RTT RTT IAT IAT IAT # ID [s] [s] [Mbit] min avg max min avg max S 0 375.011 380.004 0.288782 12916.913 14135.647 15035.946 30.069 183.367 969.321 R 0 375.008 380.001 0.446299 5378.736 7304.811 8322.028 12.080 138.115 1206.780 S 1 375.008 380.009 0.157245 1284.537 2348.903 3978.513 70.058 418.893 2341.099 R 1 375.009 380.010 0.026211 11766.836 11766.836 11766.836 2919.213 2919.213 2919.213 S 0 380.004 385.000 0.288551 13335.203 14015.217 15029.046 63.087 269.419 1427.218 R 0 380.001 385.003 0.406170 7380.097 8201.946 9628.294 16.043 191.917 987.361 cwnd ssth uack sack lost retr fack reor rtt rttvar rto castate mss mtu status # # # # # # 83.000 59 83 0 0 0 0 3 3276.500 50.000 4940.000 open 1448 1500 (n/n) 128.000 107 128 0 0 0 0 3 2879.000 6.000 4252.000 open 1448 1500 (n/n) 44.000 7 44 0 0 0 0 3 2880.500 256.000 4208.000 open 1448 1500 (n/n) 8.000 5 8 0 0 0 0 3 2832.500 149.000 3848.000 open 1448 1500 (n/n) 86.000 59 86 0 0 0 0 3 3654.500 190.000 5072.000 open 1448 1500 (n/n) 142.000 107 142 0 0 0 0 3 3388.500 65.000 4520.000 open 1448 1500 (n/n) Arnd Hannemann credativ GmbH 19 / 29
WMN example: Goodput 1.6 1.4 Flow 0 from node 16 to 8 Flow 1 from node 17 to 9 1.2 Goodput [ Mb /s] 1 0.8 0.6 0.4 0.2 0 0 100 200 300 400 500 600 700 800 900 Time [s] Arnd Hannemann credativ GmbH 20 / 29
WMN example: Congestion Window 400 350 Congestion Window Slowstart threshold 300 Window size [segments] 250 200 150 100 50 0 0 100 200 300 400 500 600 700 800 900 Time [s] Arnd Hannemann credativ GmbH 21 / 29
Test of congestion control algorithms in AWS Test scenario Measurement performed in VPC Four flows between a pair of nodes Four different congestion control algorithms Arnd Hannemann credativ GmbH 22 / 29
AWS: Flowgrind arguments flowgrind -n 4 -H s=172.30.0.122,d=172.30.0.123 -T s=900 -F 0 -O s=tcp_congestion=yeah -F 1 -O s=tcp_congestion=cubic -F 2 -O s=tcp_congestion=highspeed -F 3 -O s=tcp_congestion=htcp Arnd Hannemann credativ GmbH 23 / 29
AWS example: Output # ID 0 S: 172.30.0.138 (Linux 4.6.0-1-amd64), random seed: 1611955119, sbuf = 12582912/0 [B] (real/req), rbuf = 12582912/0 [B] (real/req), SMSS = 8949 [B], PMTU = 9001 [B], Interface MTU = 9001 (unknown) [B], CC = yeah, duration = 900.003/900.000 [s] (real/req), through = 5.758049/0.000000 [Mbit/s] (out/in), request blocks = 79075/0 [#] (out/in) # ID 0 D: 172.30.0.139 (Linux 4.6.0-1-amd64), random seed: 1611955119, sbuf = 12582912/0 [B] (real/req), rbuf = 12582912/0 [B] (real/req), SMSS = 1448 [B], PMTU = 9001 [B], Interface MTU = 9001 (unknown) [B], through = 0.000000/5.684553 [Mbit/s] (out/in), request blocks = 0/78065 [#] (out/in), IAT = 0.004/11.529/281.197 [ms] (min/avg/max), delay = 18.708/11481.539/27539.894 [ms] (min/avg/max)... Arnd Hannemann credativ GmbH 24 / 29
AWS example: Goodput 500 400 YeAH-TCP CUBIC TCP Highspeed TCP H-TCP 500 400 Goodput in Mb s 1 300 200 100 300 200 100 0 0 100 200 300 400 500 600 700 800 900 0 Time in s Arnd Hannemann credativ GmbH 25 / 29
Overview Introduction Flowgrind Architecture Example measurements Summary Arnd Hannemann credativ GmbH 26 / 29
Summary Feature Iperf Iperf3 Netperf Thrulay TTCP NUTTCP Flowgrind TCP UDP SCTP Other protocols Kernel statistics Interval reports Conc. tests w. same hosts Conc. tests w. different hosts Distributed tests Bidirectional traffic Test scheduling Traffic generation Control/test data Arnd Hannemann credativ GmbH 27 / 29
Summary Flowgrind Distributed architecture well suited for complex test scenarios Extensive TCP metrics Advanced traffic generation features https://github.com/flowgrind/flowgrind Possible future improvements Option for easier multi-core support, performance support for TCP Fast Open Add support for other procotols: UDP/DCCP/SCTP Arnd Hannemann credativ GmbH 28 / 29
Thanks for listening. Questions? Arnd Hannemann credativ GmbH 29 / 29