CLOSE_WAIT not visible on the kubernetes node

Question

Debugging a tcp: out of memory error, I found that a process (from a container) has a lot of connection on CLOSE_WAIT status aka 08 when I cat /proc/XXX/net/tcp

But neither netstat or ss were showing those leaked connections.

133: 0E03540A:9D9C 804CC2AD:01BB 08 00000000:00059D7A 00:00000000 00000000 0 0 316215 1 ffff8f677201df00 20 4 0 10 -1
134: 0E03540A:8316 80A7E940:01BB 08 00000000:00000000 00:00000000 00000000 0 0 255647 1 ffff8f67c9592600 20 4 1 10 -1
135: 0E03540A:8874 808C7D4A:01BB 08 00000000:00037EED 00:00000000 00000000 0 0 331603 1 ffff8f68e37a7200 20 4 1 10 -1
136: 0E03540A:E226 804CC2AD:01BB 08 00000000:0005E30B 00:00000000 00000000 0 0 215782 1 ffff8f67bd1edf00 20 4 0 10 -1
137: 0E03540A:DAEC 804CC2AD:01BB 08 00000000:0005B41A 00:00000000 00000000 0 0 216048 1 ffff8f67daf9af80 20 4 0 10 -1
138: 0E03540A:9AEA 8005FB8E:01BB 08 00000000:000D6360 00:00000000 00000000 0 0 243082 1 ffff8f67db637200 20 4 30 10 -1
140: 0E03540A:BAE4 800FB16C:01BB 08 00000000:000D8432 00:00000000 00000000 0 0 245062 1 ffff8f67640f8980 20 4 1 10 -1
141: 0E03540A:9754 804CC2AD:01BB 08 00000000:00003186 00:00000000 00000000 0 0 298890 1 ffff8f676e1a5f00 20 4 1 10 -1
142: 0E03540A:C6FC 800FB16C:01BB 08 00000000:000658C9 00:00000000 00000000 0 0 299343 1 ffff8f68dcef5580 20 4 0 10 -1
143: 0E03540A:CB24 804CC2AD:01BB 08 00000000:0005BBB4 00:00000000 00000000 0 0 316285 1 ffff8f6772019300 20 4 1 10 -1
144: 0E03540A:8204 80A7E940:01BB 08 00000000:0005DD3A 00:00000000 00000000 0 0 217390 1 ffff8f67dbc20000 20 4 0 10 -1
145: 0E03540A:8BC8 80016642:01BB 08 00000000:00059847 00:00000000 00000000 0 0 275095 1 ffff8f67b6d7a600 20 4 1 10 -1
146: 0E03540A:C612 8005FB8E:01BB 08 00000000:0003EC48 00:00000000 00000000 0 0 252281 1 ffff8f67cf014280 20 4 1 10 -1

Why netstat is not showing those connection and how to get them without digging into each process details ?

Answer 1

If you are looking at this from the host, you're in the initial network namespace rather than in the container's network namespace: these connections or states are not seen because they are not handled by the initial network namespace's network stack. When following an entry in the process directory in /proc, this entry is seen from the process point of view ... sometimes, so will sometimes display the relevant namespace information, but tools are not meant to use this.

So you have to switch to the studied process' network namespace first.

It's as simple as (with the root user):

nsenter -t XXX --net -- ss -tn

Or to find the process(es) (as seen in the initial pid namespace, not the container's):

nsenter -t XXX --net -- ss -tnp state CLOSE-WAIT

Normally one would search per pod (or container in other techonogies) rather than per process. Various container technologies allow to retrieve a PID process from the container name (eg: LXC's lxc-info -Hp -n containername or Docker's docker inspect --format '{{.State.Pid}}' containername ), but I don't know if and how this information is available with Kubernetes if the backend is not Docker.

Also for some tools it's a bit more difficult than this, because for example /sys should be remounted for /sys/class/net to reflect the new network namespace's interfaces view: now there would be two namespaces to change: target process' namespace and temporary mount namespace (to not damage the initial, nor use the target that might not have the required commands). Anyway, the ss command is operating purely on sockets and wouldn't need this.

For example the obsolete brctl show command would require this to work properly:

nsenter -t XXX --net -- unshare --mount -- sh -c 'mount -t sysfs sysfs /sys; brctl show'