18

I'm working on a software which connects to a Real Time data server (using TCP) and I have some connections dropping. My guess is that the clients do not read the data coming from the server fast enough. Therefore I would like to monitor my TCP sockets. For this I found the "ss" tool.

This tool allows to see the state of every socket - here's an example line of the output of the command ss -inm 'src *:50000'

ESTAB      0      0             184.7.60.2:50000       184.92.35.104:1105
  mem:(r0,w0,f0,t0) sack rto:204 rtt:1.875/0.75 ato:40

My question is: what does the memory part mean? Looking at the source code of the tool I found that the data is coming from a kernel structure (sock in sock.h). More precisely, it comes from the fields :

r = sk->sk_rmem_alloc
w = sk->sk_wmem_queued;
f = sk->sk_forward_alloc;
t = sk->sk_wmem_alloc;

Does somebody know what they mean? My guesses are:

  • rmem_alloc : size of the inbound buffer
  • wmem_alloc : size of the outbound buffer
  • sk_forward_alloc : ???
  • sk->sk_wmem_queued : ???

Here are my buffers sizes :

net.ipv4.tcp_rmem = 4096        87380   174760
net.ipv4.tcp_wmem = 4096        16384   131072
net.ipv4.tcp_mem = 786432       1048576 1572864
net.core.rmem_default = 110592
net.core.wmem_default = 110592
net.core.rmem_max = 1048576
net.core.wmem_max = 131071
  • What's your buffer size configuration? Do you see receive buffers getting saturated on the socket connections? Does your party drop connection on EWOULDBLOCK? – Karlson Mar 10 '12 at 6:28
  • My sockets sizes are quite small I think, i updated the post with them. For the EWOULDBLOCK I can't tell. My client is in JAVA and just say it has been disconnected by the server. The server is in C++ and it just say that he dropped the connection without any information. I do not have the source code of the server so I cannot change its behavior. It seems that clients get disconnected when they are a bit overloaded, even if it only last a few seconds. – Twister Mar 12 '12 at 10:56
  • Is the configuration of buffer sizes adjustable on the server? Can you watch buffer sizes on the client? Do you have access to the client's source? Have you run netstat -apnc to watch the buffer sizes? Did you try to increase the buffer sizes in the kernel to see what happens? – Karlson Mar 12 '12 at 12:42
  • Yes they are, and are already set to the max value of the server (i believe they can't be bigger than the net.ipv4.tcp_* properties, right ?) For netstat -apnc it does not give me the buffers sizes, that why I looked at ss. For the kernel I am not root on the server, and the IT teams here are pretty stubborn. I need to be sure of what happens before I ask them to change the values... And yes I have access to the client source, and my investigation on the client confirm that the disconnection comes from the server. – Twister Mar 12 '12 at 13:05
  • netstat -apnc gives you total size of send and receive queues on Linux. If the server sets the buffer to maximum available and you are still saturating maybe you need higher buffer settings at the OS level – Karlson Mar 12 '12 at 13:45
7

sk_forward_alloc is the forward allocated memory which is the total memory currently available in the socket's quota.

sk_wmem_queued is the amount of memory used by the socket send buffer queued in the transmit queue and are either not yet sent out or not yet acknowledged.

You can learn more about TCP Memory Management in chapter 9 of TCP/IP Architecture, Design and Implementation in Linux By Sameer Seth, M. Ajaykumar Venkatesulu

| improve this answer | |
  • I don't understand how this definition of sk_wmem_queued differs from sk_wmem_alloc, could you expand a little on this? (If you know the answer, feel free to add an answer to this question: unix.stackexchange.com/questions/551444/…) – little-dude Nov 11 '19 at 9:07
1

See the man page of ss.

<fwd_alloc>
   The  memory allocated by the socket as cache, but not used for receiving/sending packet yet. If need memory to send/receive packet, the memory in this cache will be used before allocate additional memory.

<wmem_queued>
   The memory allocated for sending packet (which has not been sent to layer 3)
| improve this answer | |
0

Regarding sk_wmem_queued and sk_wmem_alloc, I asked the same question so I'll copy the answer here:

I emailed Eric Dumazet, who contributes to the Linux network stack, and here is the answer:

sk_wmem_alloc tracks the number of bytes for skb queued after transport stack : qdisc layer and NIC TX ring buffers.

If you have 1 MB of data sitting in TCP write queue, not yet sent (cwnd limit), sk_wmem_queue will be about 1MB, but sk_wmem_alloc will be about 0

A very good document for understanding what these three types of queues (socket buffer, qdisc queue and device queue) are is this article (rather long) article. In a nutshell, the socket starts by pushing the packets directly onto the qdisc queue, which forwards them to the device queue. When the qdisc queue is full, the socket starts buffering the data in its own write queue.

the network stack places packets directly into the queueing discipline or else pushes back on the upper layers (eg socket buffer) if the queue is full

So basically: sk_wmem_queues is the memory used by the socket buffer (sock.sk_write_queue) while sk_wmem_alloc is the memory used by the packets in the qdisc and device queues.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.