4

The sock struct defined in sock.h, has two attributes that seem very similar:

  • sk_wmem_alloc, which is defined as "transmit queue bytes committed"
  • sk_wmem_queued, defined as "persistent queue size"

To me, the sk_wmem_alloc is the amount of memory currently allocated for the send queue. But then, what is sk_wmem_queued?

References

  • According to this StackOverflow answer:

    wmem_queued: the amount of memory used by the socket send buffer queued in the transmit queue and are either not yet sent out or not yet acknowledged.

  • The ss man also gives definitions, which don't really enlighten me (I don't understand what the IP layer has to do with this):

    wmem_alloc: the memory used for sending packet (which has been sent to layer 3) wmem_queued: the memory allocated for sending packet (which has not been sent to layer 3)

  • Someone already asked a similar question on the LKML, but got no answer
  • The sock_diag(7) man page also has its own definitions for these attributes:

    SK_MEMINFO_WMEM_ALLOC: The amount of data in send queue. SK_MEMINFO_WMEM_QUEUED: The amount of data queued by TCP, but not yet sent.

All these definitions are different, and none of them clearly explain how the _alloc and _queued variants are different.

2 Answers 2

8

I emailed Eric Dumazet, who contributes to the Linux network stack, and here is the answer:

sk_wmem_alloc tracks the number of bytes for skb queued after transport stack : qdisc layer and NIC TX ring buffers.

If you have 1 MB of data sitting in TCP write queue, not yet sent (cwnd limit), sk_wmem_queue will be about 1MB, but sk_wmem_alloc will be about 0

A very good document for understanding what these three types of queues (socket buffer, qdisc queue and device queue) are is this article (rather long) article. In a nutshell, the socket starts by pushing the packets directly onto the qdisc queue, which forwards them to the device queue. When the qdisc queue is full, the socket starts buffering the data in its own write queue.

the network stack places packets directly into the queueing discipline or else pushes back on the upper layers (eg socket buffer) if the queue is full

So basically: sk_wmem_queues is the memory used by the socket buffer (sock.sk_write_queue) while sk_wmem_alloc is the memory used by the packets in the qdisc and device queues.

1

TL;DR: let's believe the man pages :-). They say there are two different places that data could be queued up. So if you want to know the total memory usage, you need to add up the two values.

Disclaimer: My assertions are conclusions drawn from well-informed sources, but I have not tested this. Also, this is seriously too long and you probably won't want to read it.


A Google search for sk_wmem_alloc returned writings about the introduction of TCP Small Queues (TSQ).

The lower layer queuing is itself composed of two different queues. First there is the qdisc (queueing discipline)[*], and then the queue inside the device.

In the TSQ code, "sk->sk_wmem_alloc [is] not allowed to grow above a given limit, allowing no more than ~128KB [by default] per tcp socket in qdisc/dev layers at a given time."

=> sk_wmem_alloc must include at least the qdisc and dev layers.

The TSQ code made a big difference. "I no longer have 4 MBytes backlogged in qdisc by a single [bulk sender]". And also, "both side socket autotuning no longer use 4 Mbytes".

Where did 4MB come from? It's not the limit of the qdisc used here - that is much larger. The test used the "standard" FIFO qdisc, which defaults to 1000 packets. 4MB / 1000 would be 4K, but that's not a standard packet size. The test used the standard maxed-out TSO packet size: 64K. Answer:

Linux 2.6.17 now has sender and receiver side autotuning and a 4 MB DEFAULT maximum window size.

What is currently implemented in Linux is essentially what is described in Semke '98, but without the max-min fair sharing.

-- https://wiki.geant.org/display/public/EK/TCP+Buffer+Auto+Tuning

Looking this up, the auto-tuning is very roughly "following the 2 * bandwidth * delay rule of thumb". It is implemented using "cwnd [congestion window]: an existing TCP variable (for a single connection) that estimates the available bandwidth-delay product [BDP] to determine the appropriate amount of data to keep in flight."

Finally, the total amount of data "in flight" is limited by the size of the TCP buffer. This is because TCP is a reliable protocol. Even after we physically transmit a packet, we need to keep a copy of the data in the buffer. We need to be able to resend it in case the packet is lost in transit. We must continue to store the data until the receiver tells us it arrived safely.

This explains the different results with TSQ.

To start with, the sender builds up a small queue in the qdisc+device. TCP incrementally increases its window, to probe for spare capacity on the path. It sends more, so it builds up a larger queue. If TCP detected packets being dropped, then it would back off. But the qdisc still has room for more packets, so there is no reason to drop any. The cycle will continue until we hit a limit...

With TSQ, a single TCP sender will not grow the qdisc+device queues beyond 128K. But without TSQ, it can reach the 4MB limit.

=> See how the man pages make sense here.

Without TSQ, sk_wmem_queued would have reached the 4MB maximum. sk_wmem_alloc would have reached 4MB, minus the rest of the BDP.

The results described were from a local test, with a very short physical transmission delay.

If we increased the delay (or bandwidth), the BDP would increase. sk_wmem_queued could reach 4MB, despite TSQ limiting sk_wmem_alloc to 128K.


[*] Advanced qdisc features are used on routers e.g. to prioritize different packets. But all Linux systems will queue some packets in the qdisc layer. Originally, this was a queue of a fixed number of packets ("fifo"). On many modern systems, the default is now "fq_codel". "codel" adaptively limits queuing by transmission time. The slower the device, the smaller the queue is allowed to grow. "fq_codel" additionally tries to give fairer sharing, and also allow non-bulk senders to skip ahead of bulk senders, to improve responsiveness.

Earlier notes

One of the two variables appears to be most relevant to UDP. The other appears to be most relevant to TCP specifically. I looked up uses of sock_writeable() (sk_wmem_alloc) v.s. sk_stream_is_writeable() (sk_wmem_queued).

Of course there are more socket protocols than just UDP and TCP. Looking at the code suggests the difference is related to implementing "reliability".

This would explain why a protocol called DCCP (Datagram Congestion Control Protocol) is using the function called sk_stream_is_writeable() :-P. DCCP is a "reliable" protocol. See dccp_poll().

If it is related to handling dropped packets, that would also explain why I didn't find a call to sk_stream_is_writeable() anywhere in the implementation of AF_LOCAL stream sockets. AF_LOCAL datagrams cannot be lost in transit.

The distinction between TCP and UDP appears especially visible in net/sunrpc/xprtsock.c:

/**
 * xs_udp_write_space - callback invoked when socket buffer space
 *                             becomes available
 * @sk: socket whose state has changed
 *
 * Called when more output buffer space is available for this socket.
 * We try not to wake our writers until they can make "significant"
 * progress, otherwise we'll waste resources thrashing kernel_sendmsg
 * with a bunch of small requests.
 */
static void xs_udp_write_space(struct sock *sk)
{
    read_lock_bh(&sk->sk_callback_lock);

    /* from net/core/sock.c:sock_def_write_space */
    if (sock_writeable(sk))
        xs_write_space(sk);

    read_unlock_bh(&sk->sk_callback_lock);
}

/**
 * xs_tcp_write_space - callback invoked when socket buffer space
 *                             becomes available
 * @sk: socket whose state has changed
 *
 * Called when more output buffer space is available for this socket.
 * We try not to wake our writers until they can make "significant"
 * progress, otherwise we'll waste resources thrashing kernel_sendmsg
 * with a bunch of small requests.
 */
static void xs_tcp_write_space(struct sock *sk)
{
    read_lock_bh(&sk->sk_callback_lock);

    /* from net/core/stream.c:sk_stream_write_space */
    if (sk_stream_is_writeable(sk))
        xs_write_space(sk);

    read_unlock_bh(&sk->sk_callback_lock);
}

Final note

The definitions of sock_writeable() and sk_wmem_alloc suggest that "socket buffer" is a lie.

In the first section above, we have clearly identified a TCP socket buffer, distinct from the qdisc+device queues. That is all true. But TCP is the special case, a reliable ("stream") protocol.

See also man sendmsg -

ENOBUFS

The output queue for a network interface was full. This generally indicates that the interface has stopped sending, but may be caused by transient congestion. (Normally, this does not occur in Linux. Packets are just silently dropped when a device queue overflows.)

The exact wording here does not mention a UDP socket buffer. There is no dedicated send buffer for a UDP socket. We just stuff the packets directly into the qdisc. If sk_wmem_alloc would exceed the "send buffer size", the sendmsg() call will block (wait). But if the qdisc does not have room, packets will be silently discarded.

6
  • My use case is to report memory usage via sock_diag (just like ss does). So to report the memory usage for a socket, I should use the value from sk_wmem_alloc for UDP sockets and sk_wmem_queued for TCP sockets. For other protocols it's more tricky, but according to your findings reliable protocols would likely use sk_wmem_queued. Thank you very much for your research on this! Nov 11, 2019 at 16:16
  • 1
    @little-dude I have a real answer now. Don't do that, just add the two values together. Answer has not been tested (and is too long).
    – sourcejedi
    Nov 11, 2019 at 21:56
  • Thank you so much for digging this out @sourcejedi. I still have to read more carefully all the link you gave, but your explanation makes a lot of sense to me! Nov 12, 2019 at 7:34
  • For reference here is a link that explains how queues and buffering work: coverfire.com/articles/queueing-in-the-linux-network-stack. @sourcejedi if you don't mind I might edit your answer later to add a few things. Otherwise I'll create a new anwer. Nov 12, 2019 at 8:37
  • @little-dude If it makes sense, I'd prefer you repeat the very basic answer (credit it if you like), and then write your own version of the details you think are most useful. I don't mind adding a few links to definitions of specific terms. What I don't want to say is "if my ramblings are not clear enough, go read this long coverfire article first and then maybe you will be able to understand me :-)"
    – sourcejedi
    Nov 12, 2019 at 8:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .