In a setup with three docker instances, one running haproxy and two others running a flask-based python application through uWsgi, we run into a situation after about a day where no new connections is accepted on one or both instances.
uWsgi is set up to accept up to 100 backlogged connections. This is less than the default 128 configured in
/proc/sys/net/core/somaxconn. uWsgi gives up on the 101st connection.
ss confirms that there is a backlog of 101.
root@ad9380a94c50:/# ss -nlpt State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 101 100 *:8080 *:* users:(("uwsgi",pid=25,fd=3),("uwsgi",pid=19,fd=3)) LISTEN 0 128 127.0.0.11:38230 *:*
There is no corresponding connection when running, for example,
The source code for uwsgi shows that the backlog queue length is obtained by calling
getsockopt abd retrieving the
tcpi_unacked field. In other words, this does not appear to be a bug in uwsgi, it seems the linux kernel and/or docker literally thinks there are connections there that aren't really there. I suspect they were there once upon a time, in the shape of a health check made by haproxy.
There is no slow-building of back log. While the instance is happy, the backlog remains at zero. It seems as if something goes wrong, and after that the backlog runs up to 100 very quickly and things hang.
We're running docker on an Amazon virtual machine.