Linux TCP Implementation Issues in High-Speed Networks



Similar documents
TCP over Multi-hop Wireless Networks * Overview of Transmission Control Protocol / Internet Protocol (TCP/IP) Internet Protocol (IP)

High-Speed TCP Performance Characterization under Various Operating Systems

Lecture 15: Congestion Control. CSE 123: Computer Networks Stefan Savage

Application Level Congestion Control Enhancements in High BDP Networks. Anupama Sundaresan

CSE 473 Introduction to Computer Networks. Exam 2 Solutions. Your name: 10/31/2013

TCP in Wireless Mobile Networks

Simulation-Based Comparisons of Solutions for TCP Packet Reordering in Wireless Network

The Effect of Packet Reordering in a Backbone Link on Application Throughput Michael Laor and Lior Gendel, Cisco Systems, Inc.

Transport Layer Protocols

Improving the Performance of TCP Using Window Adjustment Procedure and Bandwidth Estimation

TCP Tuning Techniques for High-Speed Wide-Area Networks. Wizard Gap

Low-rate TCP-targeted Denial of Service Attack Defense

Data Networks Summer 2007 Homework #3

CSE331: Introduction to Networks and Security. Lecture 9 Fall 2006

Prefix AggregaNon. Company X and Company Y connect to the same ISP, and they are assigned the prefixes:

First Midterm for ECE374 03/24/11 Solution!!

A Survey on Congestion Control Mechanisms for Performance Improvement of TCP

Lecture Objectives. Lecture 07 Mobile Networks: TCP in Wireless Networks. Agenda. TCP Flow Control. Flow Control Can Limit Throughput (1)

A packet-reordering solution to wireless losses in transmission control protocol

End-to-End Network/Application Performance Troubleshooting Methodology

A Study of Internet Packet Reordering

TCP/IP Optimization for Wide Area Storage Networks. Dr. Joseph L White Juniper Networks

17: Queue Management. Queuing. Mark Handley

Challenges of Sending Large Files Over Public Internet

CS268 Exam Solutions. 1) End-to-End (20 pts)

Key Components of WAN Optimization Controller Functionality

2 TCP-like Design. Answer

STUDY OF TCP VARIANTS OVER WIRELESS NETWORK

Outline. TCP connection setup/data transfer Computer Networking. TCP Reliability. Congestion sources and collapse. Congestion control basics

TCP over Wireless Networks

4 High-speed Transmission and Interoperability

Allocating Network Bandwidth to Match Business Priorities

The Problem with TCP. Overcoming TCP s Drawbacks

TCP loss sensitivity analysis ADAM KRAJEWSKI, IT-CS-CE

A Survey: High Speed TCP Variants in Wireless Networks

Visualizations and Correlations in Troubleshooting

The Data Replication Bottleneck: Overcoming Out of Order and Lost Packets across the WAN

ICTCP: Incast Congestion Control for TCP in Data Center Networks

TCP Adaptation for MPI on Long-and-Fat Networks

TCP/IP Inside the Data Center and Beyond. Dr. Joseph L White, Juniper Networks

Final for ECE374 05/06/13 Solution!!

COMP 361 Computer Communications Networks. Fall Semester Midterm Examination

Names & Addresses. Names & Addresses. Hop-by-Hop Packet Forwarding. Longest-Prefix-Match Forwarding. Longest-Prefix-Match Forwarding

Improving our Evaluation of Transport Protocols. Sally Floyd Hamilton Institute July 29, 2005

A Passive Method for Estimating End-to-End TCP Packet Loss

Improving Effective WAN Throughput for Large Data Flows By Peter Sevcik and Rebecca Wetzel November 2008

Mobile Communications Chapter 9: Mobile Transport Layer

TCP/IP Over Lossy Links - TCP SACK without Congestion Control

Congestion Control Review Computer Networking. Resource Management Approaches. Traffic and Resource Management. What is congestion control?

CS551 End-to-End Internet Packet Dynamics [Paxson99b]

TCP performance optimization for handover Management for LTE satellite/terrestrial hybrid. network.

Optimization of Communication Systems Lecture 6: Internet TCP Congestion Control

MMPTCP: A Novel Transport Protocol for Data Centre Networks

TCP Westwood for Wireless

SJBIT, Bangalore, KARNATAKA

AN IMPROVED SNOOP FOR TCP RENO AND TCP SACK IN WIRED-CUM- WIRELESS NETWORKS

RTT 60.5 msec receiver window size: 32 KB

Optimizing TCP Forwarding

APPENDIX 1 USER LEVEL IMPLEMENTATION OF PPATPAN IN LINUX SYSTEM

Congestion Control Overview

EFFECT OF TRANSFER FILE SIZE ON TCP-ADaLR PERFORMANCE: A SIMULATION STUDY

Chapter 6 Congestion Control and Resource Allocation

Effect of Packet-Size over Network Performance

High Speed Internet Access Using Satellite-Based DVB Networks

Delay-Based Early Congestion Detection and Adaptation in TCP: Impact on web performance

Master s Thesis. Design, Implementation and Evaluation of

TCP for Wireless Networks

SELECTIVE-TCP FOR WIRED/WIRELESS NETWORKS

Mobile Computing/ Mobile Networks

Congestions and Control Mechanisms n Wired and Wireless Networks

Linux 2.4 Implementation of Westwood+ TCP with rate-halving: A Performance Evaluation over the Internet

Sockets vs. RDMA Interface over 10-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck

TCP Fast Recovery Strategies: Analysis and Improvements

ICOM : Computer Networks Chapter 6: The Transport Layer. By Dr Yi Qian Department of Electronic and Computer Engineering Fall 2006 UPRM

TCP and Wireless Networks Classical Approaches Optimizations TCP for 2.5G/3G Systems. Lehrstuhl für Informatik 4 Kommunikation und verteilte Systeme

TCP in Wireless Networks

Computer Networks. Chapter 5 Transport Protocols

Active Queue Management (AQM) based Internet Congestion Control

International Journal of Scientific & Engineering Research, Volume 6, Issue 7, July ISSN

TCP tuning guide for distributed application on wide area networks 1.0 Introduction

Homework 2 assignment for ECE374 Posted: 02/21/14 Due: 02/28/14

Sample Network Analysis Report

TCP PACKET CONTROL FOR WIRELESS NETWORKS

TCP based Denial-of-Service Attacks to Edge Network: Analysis and Detection

La couche transport dans l'internet (la suite TCP/IP)

Network Probe. Figure 1.1 Cacti Utilization Graph

A Compound TCP Approach for High-speed and Long Distance Networks

First Midterm for ECE374 03/09/12 Solution!!

TFTP TRIVIAL FILE TRANSFER PROTOCOL OVERVIEW OF TFTP, A VERY SIMPLE FILE TRANSFER PROTOCOL FOR SIMPLE AND CONSTRAINED DEVICES

Transcription:

Linux TCP Implementation Issues in High-Speed Networks D.J.Leith Hamilton Institute, Ireland www.hamilton.ie 1. Implementation Issues 1.1. SACK algorithm inefficient Packets in flight and not yet acknowledged are held on the writequeue linked list at the transmitter. On receipt of SACK information, the writequeue is walked and the relevant packets tagged as sacked or lost. In the original implementation writequeue is walked for each sack block (i.e. up to three times) and again to mark packets as lost. Lost packets are retransmitted via another walk of writequeue. For large delay-bandwidth products, the writequeue is very large (many thousands of packets) and these walks become too time consuming to complete between the arrival of ACKs. 45 25 Mbs, 15 ms, q max =1Kbytes HS-TCP Linux 2.4.23 web1 implementation 45 25 Mbs, 15 ms, q max =1Kbytes HS-TCP Linux 2.4.23 web1 with updated sacktag implementation 35 35 3 3 25 15 25 15 1 1 5 5 2 4 6 8 1 12 14 16 18 2 2 4 6 8 1 12 14 16 18 2 (a) (b) Figure 1 Example illustrating impact of SACK processing inefficiency. Transfer between servers (Dell 165, 2.8GHz Xeon, PCI-X 133Mhz/64bit & Intel Pro/1 NICs) running Linux 2.4.23 with web1 (sbufmode==rbufmode; WAD_IFQ=1, WAD_FloydAIMD=1) via dummynet router of same specification running FreeBSD 4.8. Repeated timeouts during loss recovery are evident in Figure 1(a) (timeouts occur due to throttle action of softirq queue when rate at which incoming packets are serviced is persistently too low see below). The corresponding performance with a more efficient SACK processing algorithm is shown in Figure 1(b). More test results are presented in a later section. This algorithm incorporates the following changes. - Tagging algorithm modified to walk writequeue holes (i.e. packets not yet sacked), rather than walking entire writequeue. Walk therefore now scales with number of lost packets rather than delay-bandwidth product. - Tagging algorithm modified to make a single walk rather than a walk per sack block (possibl;e by presorting sack blocks by sequence number). - We now cache pointers for the retransmission (using idea of Kelly) walk and also for walk on receipt of a dsack (for which a walk of the writequeue holes is insufficient). - when number of holes in writequeue becomes large, sack blocks of incoming ACK s are cached and the walk of the writequeue performed less frequently typically there is redundancy in the SACK information of ACK s whereby later sack information subsumes that in earlier ACK s and so caching leads to significant efficiency savings. Note that Kelly proposes an alternative approach based on specific redundancy in the sack blocks between ACK s this fails when ACK s are being frequently lost (which is commonly the case on high-speed links due to pressure on the computational resources and associated dropping from softirq queue).

1.2. Throttle action excessively severe On overflow, the netdev softirq queue enters a throttle state whereby all incoming packets are dropped until the queue empties. This invariably leads to a TCP timeout. The timeout is often particularly damaging as the throttle action breaks ACK clock (many incoming ACK s are dropped) so that recovery from the timeout can be very slow for large delay-bandwidth products. 16 1 1 w5gva->slac H-TCP with original Linux SACK algorithm 1 8 6 1 2 3 4 5 6 7 8 9 Figure 2 TCP transfer between Geneva, Switzerland and Stanford, California using Linux 2.4.23 with web1 and standard SACK processing algorithm. The slow recovery from the throttle-induced TCP timeouts is evident. It is proposed that the throttle action be disabled as being unnecessarily severe. Instead, a receiver just drops incoming packets when the softirq queue overflows. Note that simply increasing the length of the netdev queue is often not a good alternative solution as the additional queueing delay can lead to frequent TCP timeouts. Disabling of the throttle action also does not remove the need for a more efficient SACK processing algorithm (see previous section). Disabling the throttle action while retaining an inefficient SACK processing algorithm leads to a heavy processor burden with large numbers of ACK s being dropped and an associated loss of SACK information and damage to the TCP ACK clock. 1.3. Prevalence of SACK reneging Inspection of data flows on high speed links reveals that sack reneging by receivers (i.e. ACKing of a packet that has already been acknowledged via a SACK) is surprisingly common. Presumably this arises due to pressure on the receiver at high data rates. The standard Linux TCP implementation treats sack reneging in a similar way to a timeout i.e. it resets cwnd to one and enters loss state. This is unnecessarily severe. It is proposed that the implementation be modified to reset the SACK tagging of the packets in flight (i.e. the packets on writequeue) but otherwise leave cwnd unchanged (i.e. still enter loss state to take care of packet accounting etc, but do not reset cwnd and snd_ssthresh). 1.4. Prevalence of packet re-ordering Re-ordering of packets is common on high-speed networks (e.g. due to router implementations) and can lead to spurious entry of the TCP recovery state. The undo mechanism in Linux will then back out of this state once reordering becomes evident. - in original implementation, undo mechanism is switched off (prior_ssthresh reset to zero) after first entry into recovery state. Reordering during recovery or repeated reordering therefore cannot be undone. Implementation modified to retain undo mechanism after first entry to recovery state. - original undo mechanism usually stores.75cwnd in prior_ssthresh and resets snd_ssthresh to prior_ssthresh on an undo; if cwnd is below snd_ssthresh then it slow start s up to snd_ssthresh before re-entering congestion avoidance. This is inaccurate, and performs poorly when reordering etc causes repeated undo actions, see for example Figure 3. Implementation is modified to save exact undo value of snd_ssthresh. Use of accurate snd_ssthresh undo is activated when sysctl net.ipv4.tcp_htcp_undo>, otherwise original algorithm is used.

1 1 ordinary backoffs 1 8 6 TCP undo actions RTT (x1ms) throughput (Mbps) 7 75 8 85 9 95 1 Figure 3 TCP transfer between Dublin and Stanford using modified Linux implementation with net.ipv4.tcp_htcp_undo= (original undo mechanism). The impact of inaccuracy in the undo mechanism is evident. Compare with the time history shown in Figure 8 which uses the same Linux implementation but with net.ipv4.tcp_htcp_undo=1 (accurate undo mechanism). 1.5.Burst moderation inefficient Moderation to reduce packet bursts due to dropped ACK s etc is achieved in original linux implementation by capping cwnd based on the estimated number of packets in flight. In high-speed networks the accounting for packets in flight is often inaccurate during loss recovery. - Packets in flight cap on cwnd in routine cwnd_down() removed. Change activated when sysctl net.ipv4.tcp_htcp_cap>. The negative impact of the cap in cwnd_down() can be seen in Figures 4 and 5. Compare with the time history shown in Figure 8 which uses the same Linux implementation but with net.ipv4.tcp_htcp_cap=1 (cap removed). - burst moderation is re-implemented using direct accounting of packets transmitted per ACK during loss recovery. Changes activated when sysctl net.ipv4.tcp_htcp_moderation>, otherwise original algorithm is used. The negative impact of the original burst moderation mechanism can be seen in Figures 6 and 7. 1 9 8 7 6 5 3 1 5 1 15 - Figure 4 TCP transfer between Dublin and Stanford using modified Linux implementation with net.ipv4.tcp_htcp_cap= (cwnd restricted to packets_in_flight+1 in cwnd_down()).

1 9 8 packet loss 9 8 packet loss 7 7 6 5 3 packet re-ordering 6 5 3 1 1 92 94 96 98 1 12 14 16 18 11 112 115 12 125 13 135 14 (a) (b) Figure 5 Detail from Figure 4 showing packet reordering episodes with associated invocation of the undo mechanism. During packet re-ordering the impact of capping cwnd is evident (compare with Figure 3). Also shown are a number of backoff events due to packet loss. Notice that capping cwnd leads to a large reduction in cwnd during loss recovery, followed by the slow starting of cwnd back up to snd_ssthresh. It can be seen from Figure 5(b) that this reduction in cwnd (these events are regular loss recovery, not timeouts) can be sufficiently large that the ACK clock is damaged, leading to very slow loss recovery. 1 9 8 7 6 5 3 1 5 1 15 Figure 6 TCP transfer between Dublin and Stanford using modified Linux implementation with net.ipv4.tcp_htcp_moderation= (original burst moderation mechanism). 9 9 8 8 7 7 6 5 6 5 3 3 1 TCP undo actions (packet reordering) 1 TCP undo actions (packet reordering) 1 15 2 25 9 95 1 15 11 115 12 125 (a) (b) Figure 7 Detail from Figure 6 showing episodes with associated invocation of the undo mechanism.

1 9 8 7 6 5 3 1 5 1 15 Figure 8 TCP transfer between Dublin and Stanford using full functionality of modified Linux implementation. The data shown in Figures 3-8 was collected on the evening of Wednesday March 24 th 24; all transfers were carried out within a single one hour period. Unless otherwise stated, all tests are made using the H-TCP variant of TCP.

2. Brief Test Results Dublin machine is a Dell PowerEdge 175, 2.4GHz Xeon, PCI-X 133MHz/64bit & Intel Pro/1 NIC. Dublin, Ireland SLAC, California 1 1 8 6 goodput (x1mbs) RTT (x1ms) 5 1 15 2 25 traceroute to iepm-resp.slac.stanford.edu (134.79.24.36), 3 hops max, 38 byte packets 1 calpurnia-vlan7.bh.access.hea.net (193.1.31.97).818 ms.811 ms.867 ms 2 mantova-vlan3.bh.access.hea.net (193.1.198.245).354 ms.222 ms.246 ms 3 hyperion-gige3-3-.bh.core.hea.net (193.1.196.121).479 ms.35 ms.372 ms 4 deimos-gige5-1.cwt.core.hea.net (193.1.195.129).65 ms.599 ms.621 ms 5 heanet.ie1.ie.geant.net (62.4.13.229).73 ms.723 ms.746 ms 6 ie.uk1.uk.geant.net (62.4.96.138) 13.221 ms 13.9 ms 13.112 ms 7 uk.ny1.ny.geant.net (62.4.96.169) 81.836 ms 81.738 ms 81.765 ms 8 esnet-gw.ny1.ny.geant.net (62.4.13.214) 81.748 ms 81.741 ms 89.963 ms 9 chicr1-oc192-aoacr1.es.net (134.55.29.57) 12.141 ms 12.16 ms 12.132 ms 1 snvcr1-oc192-chicr1.es.net (134.55.29.53) 15.311 ms 15.283 ms 15.11 ms 11 slac-pos-snv.es.net (134.55.29.2) 15.498 ms 15.484 ms 15.594 ms 12 * * * 13 * * * 14 iepm-resp.slac.stanford.edu (134.79.24.36) 151.768 ms 151.83 ms 151.775 ms

Dublin, Ireland LBL, California 1 1 1 8 6 goodput (x1mbs) RTT (ms) 5 1 15 2 25 3 35 traceroute to net1.lbl.gov (131.243.2.93), 3 hops max, 38 byte packets 1 calpurnia-vlan7.bh.access.hea.net (193.1.31.97).81 ms 1.434 ms.868 ms 2 mantova-vlan3.bh.access.hea.net (193.1.198.245).23 ms.223 ms.247 ms 3 hyperion-gige3--.bh.core.hea.net (193.1.196.173).355 ms.349 ms.373 ms 4 deimos-gige5-1.cwt.core.hea.net (193.1.195.129).65 ms.6 ms.621 ms 5 heanet.ie1.ie.geant.net (62.4.13.229).73 ms.725 ms.747 ms 6 ie.uk1.uk.geant.net (62.4.96.138) 13.98 ms 13.9 ms 13.237 ms 7 uk.ny1.ny.geant.net (62.4.96.169) 81.83 ms 81.8 ms 81.82 ms 8 esnet-gw.ny1.ny.geant.net (62.4.13.214) 81.83 ms 81.798 ms 81.82 ms 9 chicr1-oc192-aoacr1.es.net (134.55.29.57) 12.41 ms 12.5 ms 12.154 ms 1 snvcr1-oc192-chicr1.es.net (134.55.29.53) 15.128 ms 15.95 ms 15.126 ms 11 lbl-snv-oc48.es.net (134.55.29.6) 151.485 ms 151.47 ms 151.58 ms 12 lbnl-ge-lbl2.es.net (198.129.224.1) 153.971 ms 153.27 ms 153.247 ms 13 ir1gw.lbl.gov (131.243.128.21) 151.612 ms 151.588 ms 151.621 ms 14 net1.lbl.gov (131.243.2.93) 151.484 ms 151.597 ms 151.5 ms

Dublin, Ireland CERN, Switzerland 7 6 5 3 1 throughput (Mbps) RTT (x1ms) 2 4 6 8 1 12 14 traceroute to 192.91.239.4 (192.91.239.4), 3 hops max, 38 byte packets 1 calpurnia-vlan7.bh.access.hea.net (193.1.31.97).885 ms.831 ms 1.23 ms 2 mantova-vlan3.bh.access.hea.net (193.1.198.245).37 ms.244 ms.228 ms 3 hyperion-gige3-3-.bh.core.hea.net (193.1.196.121).55 ms.375 ms.362 ms 4 deimos-gige5-1.cwt.core.hea.net (193.1.195.129).639 ms.639 ms.633 ms 5 heanet.ie1.ie.geant.net (62.4.13.229).911 ms.917 ms.757 ms 6 ie.uk1.uk.geant.net (62.4.96.138) 13.342 ms 13.25 ms 13.341 ms 7 uk.fr1.fr.geant.net (62.4.96.89) 2.543 ms 28.688 ms 2.48 ms 8 fr.ch1.ch.geant.net (62.4.96.29) 28.713 ms 28.563 ms 28.58 ms 9 switch-bckp-gw.ch1.ch.geant.net (62.4.13.22) 28.813 ms 28.831 ms 28.792 ms 1 r4gva-v-7.cern.datatag.org (192.91.239.234) 28.847 ms 28.792 ms 28.767 ms 11 w4gva-ge-.cern.datatag.org (192.91.239.4) 28.941 ms 28.787 ms 28.664 ms Delay-bandwidth product for this path is thought to be around packets, with a 496 packet queue located at the link in CERN.

CERN, Switzerland SLAC, California 1 1 1 8 6 throughput (x1mbs) 5 1 15 2 25 traceroute to iepm-resp.slac.stanford.edu (134.79.24.36), 3 hops max, 38 byte packets 1 r4chi-v-57.caltech.datatag.org (192.91.239.56) 116.555 ms 116.392 ms 116.368 ms 2 abilene-tge.cern.ch (192.91.236.197) 116.563 ms 116.539 ms 116.527 ms 3 iplsng-chinng.abilene.ucaid.edu (198.32.8.77) 124.311 ms 126.566 ms 12.273 ms 4 kscyng-iplsng.abilene.ucaid.edu (198.32.8.81) 129.591 ms 129.539 ms 129.494 ms 5 dnvrng-kscyng.abilene.ucaid.edu (198.32.8.13) 148.182 ms 14.79 ms 14.11 ms 6 snvang-dnvrng.abilene.ucaid.edu (198.32.8.1) 164.923 ms 164.923 ms 164.857 ms 7 losang-snvang.abilene.ucaid.edu (198.32.8.94) 172.375 ms 172.364 ms 172.363 ms 8 hpr-lax-gsr1--abilene-la-1ge.cenic.net (137.164.25.2) 172.931 ms 173.353 ms 172.542 ms 9 dc-lax-dc1--lax-hpr1-ge.cenic.net (137.164.22.12) 172.726 ms 172.737 ms 172.65 ms 1 dc-sac-dc1--lax-dc1-pos.cenic.net (137.164.22.127) 181.87 ms 181.945 ms 181.971 ms 11 dc-oak-dc2--csac-dc1-ge.cenic.net (137.164.22.11) 184.68 ms 183.861 ms 183.716 ms 12 dc-oak-dc1--oak-dc2-ge.cenic.net (137.164.22.36) 184.9 ms 184.273 ms 183.627 ms 13 dc-stan--oak-dc1-ge.cenic.net (137.164.23.42) 184.681 ms 184.716 ms 184.623 ms 14 rtr-dmz1-vlan41.slac.stanford.edu (192.68.191.85) 175.147 ms 175.81 ms 175.92 ms 15 * * * 16 iepm-resp.slac.stanford.edu (134.79.24.36) 175.39 ms 175.49 ms 175.62 ms

2026 © DocPlayer.net Privacy Policy | Terms of Service | Feedback  | Do Not Sell My Personal Information