Problem: my music server code, using a simple TCP connection on a blocking socket, needs to stream bytes out to a client (happens to be a Logitech squeezebox). It's not complicated - read 64k from a file, write it to the squeezebox, repeat. It's all running on a not-at-all-busy local LAN, and the server and squeezebox client are plugged into the same switch. The squeezebox doesn't consume the stream very quickly so the server, on pretty much any hardware, should have no trouble keeping the client fed.

And when the server runs on a Raspberry pi 3B+, it in fact has no problem at all. A pi zero could probably keep up. When it runs on my Linux laptop, ditto, everything is fine. I can ask the squeezebox periodically how full it's internal buffer is and it quickly gets up to about 99+%, and stays there. The server write()s (after the first few), spend most of their time blocked, as you'd expect.

But I move the server to a Azulle Inspire running Linux, plugged into the same switch, and something goes horribly wrong. Music starts to play but rapidly stutters and dies. The squeezebox reports the buffer is starting to fill, but then something stalls out and the buffer quickly empties (sometimes ticking up slightly, so I think some traffic gets through, but not near enough), halting the music. The server claims it's continuing writing, though writes take longer than I'd expect.

Note that the Azulle has other occasional networking duties and they all work fine, though I probably wouldn't notice short networking delays for most of those other applications. But when the music server is running, the NUC (and network) are otherwise idle - this isn't a CPU or bandwidth problem.

I've tried changing cables, changing switches and using different ports on the switches. I've tried sending different buffer sizes. To no effect. All I can come up with is there is something very wonky about the TCP stack or ethernet hardware.

How do I debug this? The linux laptop, which streams out just fine, is running Linux 4.15.0-55-generic (and apt upgrade doesn't change that). The Azuelle is running Linux 4.15.0-64-generic, Mint. I can't believe there's a radical change in TCP handling within 4.15.0. I'm not very familiar with tools like tcpdump, let along kernel config or debugging, so I'm looking for some hand-holding...

ping times between the linux laptop and Azuelle are consistently around 0.2ms and 0.35ms, with 0.33ms typical.

I'm lost. TIA.


Use tcpdump to capture your stream:

tcpdump -i iface -s 1500 -w out.cap 'tcp and port xxx'

where iface is the network interface and xxx is one of the two port numbers.

Then open out.cap with wireshark and see what you can make from the trace. It should be obvious what is going on there. If not, post again.

FWIW, from what you are saying, it sounds like an MTU issue.

| improve this answer | |

Well, that was the clue I needed.

I found this when I got curious about MTU sizes:

/sys/class/net/enp1s0/mtu:1500 /sys/class/net/lo/mtu:65536 /sys/class/net/wlp2s0/mtu:1500

All good, but wlp looked like a wireless connection. Wireless? Was that even on? So I did the tcpdump on the wlp... interface, and I saw one message I recognized from the protocol and then an long series of ACKs, and nothing else, and the streaming played a few seconds of music and failed.

Then I turned off the wireless and tried again. No stuttering. Everything smooth.

The weird thing is that the server is a few feet from the wireless access point. Even if it was using it there should be no way there wasn't enough bandwidth. I wonder if, for some reason, having both on at once caused a problem, but I thought that was impossible...

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.