TCP Labs. WACREN Network Monitoring and Measurement Workshop Antoine Delvaux perfsonar developer

Transcription

1 TCP Labs WACREN Network Monitoring and Measurement Workshop Antoine Delvaux perfsonar developer

2 Hands-on session We ll explore practical aspects of TCP Checking the effect of loss and latency on TCP performance Looking the TCP window size Checking and setting buffers values Looking at the effects of tuning on TCP performance 2

3 VM setup On your local setup: 3 VM Debian + perfsonar-testpoint CentOS + perfsonar-testpoint perfsonar toolkit full install (CentOS) Need to check/change network card before boot Should be on Bridged adapter If not possible, then in Host-only adapter or NAT Local network on your machine (IP if host-only adapter config): : your host : Debian host wacren : CentOS host wacrencos : perfsonar wacrenps 3

4 VM reconfig Login: pst/abuja Beware of keyboard: AZERTY Change after login with: On Debian: sudo dpkg-reconfigure keyboard-configuration On CentOS (as root) vi /etc/sysconfig/keyboard Timezone On Debian: sudo dpkg-reconfigure tzdata 4

5 Mininet What is mininet? Mininet creates a realistic virtual network, running real kernel, switch and application code, on a single machine (VM, cloud or native), in seconds, with a single command Used by many networking courses: 5

6 Mininet Simulates network using OpenFlow (Open vswitch), all on your local machine Simulate network topology and link characteristics bandwidth losses delays queuing etc You define the network topology in a Python script 6

7 Launch and use mininet Default topology: h1 s1 h2 $ sudo mn *** Creating network ( ) mininet> help Ping hosts in mininet shell: pingall h1 ping h2 Show interfaces h1 ifconfig h2 ifconfig 7

8 What is iperf? Throughput: Rate of successful message delivery over a communication channel How much data can I shovel into the network for some given amount of time? What does this tell us? Opposite of utilization (e.g. its how much we can get at a given point in time, minus what is utilised) Utilisation and throughput added together are capacity Tools that measure throughput are a simulation of a real work use case (e.g. how well could bulk data movement perform) Ways to game the system Parallel streams Manual window size adjustments memory to memory testing no spinning disk 8

9 How does iperf work? Data Transfer between memory on local and remote host through end systems TCP/IP- Stack and network Two modes: TCP, UDP Best practise: Interactive session on both sides Take result from Server (Receiver) 9

10 iperf server One program, two roles : Server (Receiver) Client (Sender) Run the server as daemon or interactively On wacrenps (CentOS host) iperf3 -s 10

11 iperf client output iperf3 -c Connecting to host , port 5201 [ 4] local port connected to port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] sec 78.1 MBytes 655 Mbits/sec KBytes [ 4] sec 72.0 MBytes 604 Mbits/sec KBytes [ 4] sec 72.8 MBytes 611 Mbits/sec KBytes [ 4] sec 75.7 MBytes 635 Mbits/sec KBytes [ 4] sec 69.8 MBytes 585 Mbits/sec KBytes [ 4] sec 50.7 MBytes 425 Mbits/sec KBytes [ 4] sec 64.8 MBytes 544 Mbits/sec KBytes [ 4] sec 75.6 MBytes 634 Mbits/sec KBytes [ 4] sec 73.8 MBytes 619 Mbits/sec KBytes [ 4] sec 75.1 MBytes 630 Mbits/sec KBytes [ ID] Interval Transfer Bandwidth Retr [ 4] sec 709 MBytes 594 Mbits/sec 856 sender [ 4] sec 708 MBytes 594 Mbits/sec receiver iperf Done. 11

12 iperf server output ~]$ iperf3 -s Server listening on Accepted connection from , port [ 5] local port 5201 connected to port [ ID] Interval Transfer Bandwidth [ 5] sec 75.6 MBytes 634 Mbits/sec [ 5] sec 72.0 MBytes 604 Mbits/sec [ 5] sec 72.9 MBytes 612 Mbits/sec [ 5] sec 75.8 MBytes 636 Mbits/sec [ 5] sec 69.8 MBytes 586 Mbits/sec [ 5] sec 51.9 MBytes 430 Mbits/sec [ 5] sec 62.6 MBytes 531 Mbits/sec [ 5] sec 75.8 MBytes 636 Mbits/sec [ 5] sec 73.2 MBytes 614 Mbits/sec [ 5] sec 75.6 MBytes 634 Mbits/sec [ 5] sec 2.67 MBytes 661 Mbits/sec [ ID] Interval Transfer Bandwidth [ 5] sec 0.00 Bytes 0.00 bits/sec sender [ 5] sec 708 MBytes 592 Mbits/sec receiver 12

13 iperf udp measurement iperf3 -c u -b 50M Connecting to host , port 5201 [ 4] local port connected to port 5201 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] sec 5.40 MBytes 45.2 Mbits/sec 691 [ 4] sec 5.96 MBytes 50.0 Mbits/sec 763 [ 4] sec 5.95 MBytes 49.8 Mbits/sec 761 [ 4] sec 5.97 MBytes 50.1 Mbits/sec 764 [ 4] sec 5.95 MBytes 49.9 Mbits/sec 762 [ 4] sec 5.96 MBytes 50.0 Mbits/sec 763 [ 4] sec 5.96 MBytes 50.0 Mbits/sec 763 [ 4] sec 5.95 MBytes 49.9 Mbits/sec 762 [ 4] sec 5.98 MBytes 50.2 Mbits/sec 765 [ 4] sec 5.95 MBytes 49.9 Mbits/sec [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] sec 59.0 MBytes 49.5 Mbits/sec ms 0/0 (-nan%) [ 4] Sent 0 datagrams iperf Done. 13

14 iperf in mininet Default topology: h1 s1 h2 (2 hosts, 1 switch) $ sudo mn *** Creating network ( ) mininet> help Ping hosts in mininet shell: pingall h1 ping h2 iperf h2 iperf3 -s -D h1 iperf3 -c h2 We need to have iperf3 running in server mode on h2 (cannot run interactively in mininet) Notes: the mininet `iperf` command uses iperf version 2 14

15 Simulating latency Mininet topology hsssh h1 s1 s2 s3 h2 sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh mininet> h1 ping h2 h2 iperf3 -s -D h1 iperf3 -c h2 h1 iperf3 -c h2 -u -b 100M What s different? latency throughput UDP tests: no retransmits, no cwnd 15

16 Simulating losses Mininet topology hsssh-loss h1 s1 s2 s3 h2 and losses between s1 and s2 sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh-loss mininet> h1 ping h2 h2 iperf3 -s -D h1 iperf3 -c h2 What s different? do ping see losses? latency retransmits congestion window (Cwnd) 16

17 Increased latency Mininet topology hsssh-loss-lat h1 s1 s2 s3 h2 with losses between s1 and s2 and increased delay between s2 and s3 sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh-loss-lat mininet> h1 ping h2 h2 iperf3 -s -D h1 iperf3 -c h2 What s different? throughput retransmits congestion window (Cwnd) 17

18 iperf 18

19 Throuput Expectations Q: What iperf through should you expect to see on a uncongested 10Gbps network? A: Gbps, depending on RTT TCP tuning CPU core speed, and ratio of sender speed to receiver speed It is good to have tools to measure! 19

20 OWAMP OWAMP = One Way Active Measurement Protocol (RFC 4656) E.g. one way ping Some differences from traditional ping: Measure each direction independently (recall that we often see things like congestion occur in one direction and not the other) Uses small evenly spaced groupings of UDP (not ICMP) packets Ability to ramp up the interval of the stream, size of the packets, number of packets OWAMP is most useful for detecting packet train abnormalities on an end to end basis Loss Duplication Out of order packets Latency on the forward vs. reverse path Number of Layer 3 hops Does require some accurate time via NTP the perfsonar toolkit does take care of this for you. 20

21 About time One Way Measurement means time measure happens on different host, need for accurate time synchronisation sync to good, close (network wise) time servers avoid pool.ntp.org servers that are too overloaded or not close enough to you perfsonar configuration provides a different set even better if you have a Stratum 1 time server on your network (and good service to your users!) OWAMP refuses to start if NTP sync is not good enough 21

22 OWAMP usefulness OWAMP is very useful in regular testing Congestion or queuing often occurs in a single direction Packet loss information (and how often/how much occurs over time) is more valuable than throughput This gives you a why to go with an observation. If your router is going to drop a 50B UDP packet, it is most certainly going to drop a 1500B/9000B TCP packet Overlaying data Compare your throughput results against your OWAMP do you see patterns? Alarm on each, if you are alarming (and we hope you are alarming ) 22

23 OWAMP lab Mininet topology hsssh sudo service owampd stop sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh mininet> h2 service owampd start h1 owping h2 h1 owping -a 1 h2 Output in 2 directions 23

24 OWAMP with loss Mininet topology hsssh-loss sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh-loss mininet> h2 service owampd restart h1 owping -a 1 h2 What s different? same losses on each direction? 24

25 Some Quick Words on BWCTL BWCTL is a wrapper around a few tools: iperf, iperf3, nuttcp, ping, owping, traceroute and tracepath Some general notes: Runs as a daemon and as a CLI client Use -c to specify a catcher (receiver) Use -s to specify a sender The bwctl client can be a 3rd machine! Will default to IPv6 if available (use -4 to force IPv4 as needed, or specify things in terms of an address if your host names are dual homed) Policy specification can do things like prevent tests to subnets, or allow longer tests to others. See the man pages for more details (bwctl.limits) 25

26 BWCTL tools Couple of varieties of testers that BWCTL knows how to talk with (use -T switch): Iperf2 Default for the command line (e.g. bwctl c HOST will invoke this) Some known problems (CPU hog, hard to get UDP testing to be correct) Iperf3 Default for the perfsonar regular testing framework, can invoke via command line switch (bwctl T iperf3 c HOST) New brew, has features iperf2 is missing (retransmissions, JSON output, daemon mode, etc.) Nuttcp Different code base, more control over how the tool behaves on the host (bind to CPU/core, etc.) Similar feature set to iperf3 Traceroute: useful to do reverse traceroutes Tracepath: very useful to see MTU and asymmetric routing issues 26

27 BWCTL lab: preparation Don t need mininet for these, can run directly on the VM (we ll use only the client) We need to install it on VM cd /etc/apt/sources.list.d sudo wget wget -qo - sudo apt-key add - sudo apt-get update sudo apt-get install bwctl-client BWCTL doesn t work behind a firewall, it needs a public IP, but we ll trigger BWCTL tests remotely 27

28 BWCTL lab On the VM: bwtraceroute -T tracepath -c newy-pt1.es.net -s nmm0.wacren.net bwtraceroute -T traceroute -c newy-pt1.es.net -s nmm0.wacren.net bwtraceroute -T tracepath -c amst-pt1.es.net -s nmm1.wacren.net bwtraceroute -T traceroute -c amst-pt1.es.net -s nmm1.wacren.net bwtraceroute -T tracepath -c nmm1.wacren.net \ -s jnbwits-1g.perfsonar.ac.za bwtraceroute -T traceroute -c nmm1.wacren.net \ -s jnbwits-1g.perfsonar.ac.za bwtraceroute -T tracepath -c nmm0.wacren.net -s pfsnr-pe.kenet.or.ke bwtraceroute -T traceroute -c nmm0.wacren.net -s pfsnr-pe.kenet.or.ke 28

29 BWCTL lab Test reverse direction, swap -s and -c Try other tools, try other hosts psmp-gn-bw-01-ath-gr.geant.net perfsonar2.mi.infn.it perfsonar02.hephy.oeaw.ac.at perfmon.dur.scotgrid.ac.uk psmp-gn-bw-01-tal-ee.geant.net ps0001.m45.ihep.su repos.indiacms.res.in dur-dut-1g.perfsonar.ac.za ps00-bsas.innova-red.net 29