Is it useful for a proxy to buffer less data than Linux RCVBUF can?

Question

I am looking at how HTTP proxies and reverse proxies deal with slow client problems. The idea is that the upstream server only have a limited slots for clients and if the client is slow to receive data, it consumes a slot for a long time. A reverse proxy can be used to buffer the response, free the slot earlier on upstream and then forward slowly the response to the client.

For instance, nginx suggests to enable upstream response buffering by allocating (by default) up to 8 buffers of 8k each. If those buffers are filled, it can start buffering on the disk (but I disabled this feature, my disks are busy enough).

See: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering

However, I did multiple checks and it seems that the kernel allocates a quite large RCVBUF (receive buffer) of around 1-4MB. If upstream sends a response of 2MB while the end client don't read anything, the proxy buffers will be filled soon, and the kernel buffer will be used instead.

Since the proxy will buffer less data than the kernel, I don't see how it helps to deal with slow clients. What can be the advantages of explicitly implementing/enabling a buffering feature in the proxy while the kernel does enough for us?

Edit: after the first response, I would like to give some details about what I tested.

• a client program connects to the reverse proxy, waits during a few seconds and starts reading.
• the reverse proxy only buffers up to 8kB in user space memory, after a read(), it will log the size of the socket's receive buffer, .
• upstream serves a HTTP response of 2MB (plus headers), the log the time it took between accept() and close().

When testing, I can see that the server will never wait on a write(), and even call close() before the slow client performed the first read(). Also, the size of the socket receive buffer will grow and exceed 2MB: the whole response from the server will be buffered.

I ran the tests with the upstream server on the same host than the client and proxy, and with the upstream on a distant host, the observed behavior is the same.

Also, I understand that the kernel may use smaller buffers under memory pressure, but this affects the reverse proxy too (which may thus be unable to buffer the response in user space).

Answer 1

I did multiple checks and it seems that the kernel allocates a quite large RCVBUF (receive buffer) of around 1-4MB.

Not by default it doesn't. The size is per socket; HTTP relationships may involve multiple sockets. There is no system maximum as far as I am aware, except in so far as there's a (pretty high) maximum number of sockets. From man 7 socket:

SO_RCVBUF

Sets or gets the maximum socket receive buffer in bytes. The kernel doubles this value (to allow space for bookkeeping overhead) when it is set using setsockopt(2), and this doubled value is returned by getsockopt(2). The default value is set by the /proc/sys/net/core/rmem_default file, and the maximum allowed value is set by the /proc/sys/net/core/rmem_max file. The minimum (doubled) value for this option is 256.

For me, this is:

> cat /proc/sys/net/core/rmem_default
212992

208 kB. However, it actually varies by protocol. From man 7 tcp:

tcp_rmem (since Linux 2.4)

This is a vector of 3 integers: [min, default, max]. These parameters are used by TCP to regulate receive buffer sizes. TCP dynamically adjusts the size of the receive buffer from the defaults listed below, in the range of these values, depending on memory available in the system.

min: minimum size of the receive buffer used by each TCP socket. The default value is the system page size. (On Linux 2.4, the default value is 4K, lowered to PAGE_SIZE bytes in low-memory systems.) This value is used to ensure that in memory pressure mode, allocations below this size will still succeed. This is not used to bound the size of the receive buffer declared using SO_RCVBUF on a socket.

default: the default size of the receive buffer for a TCP socket. This value overwrites the initial default buffer size from the generic global net.core.rmem_default defined for all protocols. The default value is 87380 bytes. (On Linux 2.4, this will be lowered to 43689 in low-memory systems.) If larger receive buffer sizes are desired, this value should be increased (to affect all sockets). To employ large TCP windows, the net.ipv4.tcp_window_scaling must be enabled (default).

max: the maximum size of the receive buffer used by each TCP socket. This value does not override the global net.core.rmem_max. This is not used to limit the size of the receive buffer declared using SO_RCVBUF on a socket. The default value is calculated using the formula

   max(87380, min(4MB, tcp_mem[1]*PAGE_SIZE/128))

(On Linux 2.4, the default is 87380*2 bytes, lowered to 87380 in low-memory systems).

This value is reported in /proc/sys/net/ipv4/tcp_rmem:

> cat /proc/sys/net/ipv4/tcp_rmem
4096    87380   6291456

Which can be confirmed with a bit of C creating a single TCP socket:

#include <sys/types.h>
#include <sys/socket.h>
#include <stdio.h>
#include <stdio.h>

int main (int argc, const char *argv[]) {
    int rcvbufsz;
    socklen_t buflen = sizeof(rcvbufsz);
    int fd = socket(AF_INET, SOCK_STREAM, 0);

    if (fd == -1) {
        perror("socket() failed");
        return 1;
    }

    if (getsockopt (
        fd,
        SOL_SOCKET,
        SO_RCVBUF,
        &rcvbufsz,
        &buflen
    ) == -1) {
        perror("getsockopt() failed");
        return 1;
    }

    printf("SO_RCVBUF = %d\n", rcvbufsz);

    return 0;
} 

Compiled and run reports SO_RCVBUF = 87380, which matches the figure from /proc. However, nginx is free to tweak this upward, not exceeding /proc/sys/net/core/rmem_max, which is probably 208 kB again.

The bit about how TCP "dynamically adjusts the size of the receive buffer from the defaults [...] depending on memory available in the system" (from man 7 tcp) is also worth restating.

Now to the meat of your question...

Since the proxy will buffer less data than the kernel, I don't see how the it helps to deal with slow clients. What can be the advantages of explicitly implementing/enabling a buffering feature in the proxy while the kernel does enough for us?

Keep in mind the buffer discussed above is not a userspace buffer. Applications generally don't do anything to it directly, although it is where data read comes from. So the data in nginx's own buffer is not simultaneously in the kernel buffer. It's being read out from that. Reading empties the buffer. So this in effect increases the amount of buffered data by 8 * 8 = 64 kB.