I've got a front end machine with about 1k persistent, very low-bandwidth TCP connections. It's a bit memory constrained so I'm trying to figure out where a few hundred MBs are going. TCP buffers are one possible culprit, but I can't make a dent in these questions:
- Where is the memory reported? Is it part of the
top, or is it part of the process's
- If I want to reduce it on a per-process level, how do I ensure that my reductions are having the desired effect?
- Do the buffers continue to take up some memory even when there's minimal traffic flowing, or do they grow dynamically, with the buffer sizes merely being the maximum allowable size?
I realize one possible answer is "trust the kernel to do this for you," but I want to rule out TCP buffers as a source of memory pressure.
Investigation: Question 1
This page writes, "the 'buffers' memory is memory used by Linux to buffer network and disk connections." This implies that they're not part of the
RES metric in
To find the actual memory usage,
/proc/net/sockstat is the most promising:
sockets: used 3640 TCP: inuse 48 orphan 49 tw 63 alloc 2620 mem 248 UDP: inuse 6 mem 10 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0
This is the best explanation I could find, but
mem isn't addressed there. It is addressed here, but 248*4k ~= 1MB, or about 1/1000 the system-wide max, which seems like an absurdly low number for a server with hundreds of persistent connections and sustained .2-.3Mbit/sec network traffic.
Of course, the system memory limits themselves are:
$ grep . /proc/sys/net/ipv4/tcp*mem /proc/sys/net/ipv4/tcp_mem:140631 187510 281262 /proc/sys/net/ipv4/tcp_rmem:4096 87380 6291456 /proc/sys/net/ipv4/tcp_wmem:4096 16384 4194304
tcp_mem's third parameter is the system-wide maximum number of 4k pages dedicated to TCP buffers; if the total of buffer size ever surpasses this value, the kernel will start dropping packets. For non-exotic workloads there's no need to tune this value.
Next up is
/proc/meminfo, and its mysterious
Cached items. I looked at several sources but couldn't find any that claimed it accounted for TCP buffers.
... MemAvailable: 8298852 kB Buffers: 192440 kB Cached: 2094680 kB SwapCached: 34560 kB ...
Investigation: Questions 2-3
To inspect TCP buffer sizes at the process level, we've got quite a few options, but none of them seem to provide the actual allocated memory instead of the current queue size or maximum.
ss -m --info:
State Recv-Q Send-Q ESTAB 0 0 ... <snip> .... skmem:(r0,rb1062000,t0,tb2626560,f0,w0,o0,bl0) ...<snip> rcv_space:43690
So we have
Send-Q, the current buffer usage
t, which are explained in this excellent post, but it's unclear how they're different from
- Something called
rb, which looks suspiciously like some sort of max buffer size, but for which I couldn't find any documentation
rcv_space, which this page claims isn't the actual buffer size; for that you need to call
This answer suggests
lsof, but the size/off seems to be reporting the same buffer usage as
COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME sslocal 4032 michael 82u IPv4 1733921 0t0 TCP localhost:socks->localhost:59594 (ESTABLISHED)
And then these answers suggest that lsof can't return the actual buffer size. It does provide a kernel module that should do the trick, but it only seems to work on sockets whose buffer sizes have been fixed with
setsockopt; if not, SO_SNDBUF and SO_RCVBUF aren't included.