Unintentional bandwidth throttling — where else to look?

Question

This is embarrassing, but I seem to have inadvertently set a persistent bandwidth throttle on SSH connections from my Ubuntu machine.

When transferring files from a remote server via scp, sftp, or rsync, the download rate is throttled to 2MB/s. When transferring files to the remote server, the upload rate is unthrottled. From other computers on the same network, transferring the same files from the same remote server, I can fully saturate my download bandwidth.

This didn't begin until roughly three weeks ago, and prior to that, I have frequently transferred files from this remote server for several years. Sometimes I would throttle the download rate with rsync's --bwlimit flag.

It's possible that I limited the bandwidth via some other mechanism while intoxicated, but I have no recollection of doing so, and I cannot find anything in my command history that indicates this.

I've checked my router's QOS settings multiple times, it is completely disabled. I've combed through the rest of the router's settings interface (it's running Tomato FWIW) to no avail.

What could possibly be throttling this specific machine's download rate?

Things I have checked:

• usage of ionice or nice
• usage of wondershaper
• usage of trickle
• router settings

Does anyone have any other thoughts on what to check or how to debug this?

Answer 1

I recently had the task of determining why traffic between our application server and a specific, single ip was experiencing transfer rates of 160 kbits/sec while all other traffic on both systems was 10-100 Mbits/sec.

The throttling was unique to these two endpoints communicating with each other.

I checked all of the obvious traffic shaping and traffic control solutions with no success.

Finally, after analyzing some packet captures I noticed the ECN flag being set only on transfers between these two endpoints.

Turns out the client was requesting Explicit Congestion Notification (ECN) and setting Congestion Experienced (CE) in all SYN packets causing the server to respond by reducing the transmission rate to relieve the congestion. The CE flag is set in the significant digits of the Differentiated Services Field.

You can check your linux server settings (/proc/sys/net/ipv4/tcp_ecn) using the following command:

sysctl -a | grep _ecn

Set tcp_ecn using:

sysctl net.ipv4.tcp_ecn=2

Possible values are:

0 - Disable ECN. Neither initiate nor accept ECN.
1 - Enable ECN when requested by incoming connections and also request ECN on outgoing connection attempts.
2 - Enable ECN when requested by incoming connections but do not request ECN on outgoing connections.

Default: 2

I used tcpdump and WireShark to verify that the CE code was in fact being transmitted.

Start a capture with tcpdump, saving it to a pcap file, start the request you want to capture, and then use Ctl+c to stop the capture:

tcpdump -I any -w my_capture.pcap

Then you can open this pcap in WireShark and use filters to see if this is happening.

All packets using ECN:

tcp.flag.ecn==1

All packets sending Congestion Experienced:

ip.dsfield.ecn==3

I am not sure this solves this specific issue, but the OP asked for any possibilities. Hopefully someone coming across this post who may be at wits end as to what else to check finds this information useful and does not spend days checking config files.

I ended up setting the server tcp_ecn to 0 and the bandwidth immediately jumped up to normal ranges. This is a temporary fix while we determine why the client is sending a CE flag in the SYN packets. I've read that this might be how some systems negotiate that they are using ECN, but that's a whole other topic.

Understanding CoS Explicit Congestion Notification
tcp_ecn