7

Is there a way to set the pipe capacity of pipes defined in a Bash (or other shell) script? Take e.g.

cmd1 | cmd2

In recent Linuxes the pipe capacity is set to 64KB by default. I know I can control the amount of data "buffered" between the two processes in two ways:

  1. Using buffer(1): e.g. cmd1 | buffer | cmd2
  2. Using fcntl(2) with the F_SETPIPE_SZ flag from inside cmd1 or cmd2

Each solution has downsides: buffer can only be used to increase the buffer; also writes over the default pipe capacity will still require waking up the downstream command. fcntl, as far as I know, can only be called from inside cmd1 or cmd2.

My question is: is there a way, when the shell creates the pipe, to specify in the shell how much capacity the pipe should have?

  • 1
    Closely related: Increase FIFO size limit – Stéphane Chazelas Dec 5 '16 at 22:09
  • @StéphaneChazelas thanks for the pointer for now I ended up adapting your's and DepressedDaniel's approach. If nobody else comes along with a better answer I will just post my snippet. – CAFxX Dec 5 '16 at 23:46
4

writes over the default pipe capacity will still require waking up the downstream command

If your goal is not to wake up the downstream command too often, did you try using the -p option to buffer? It should cause buffer to hold writes until the buffer is filled to certain percentage. You might need the -s option as well to write large chunks.

Update: D'oh, the pipes between the commands still limit things. Maybe try using the following adapter program:

#define _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
int main(int argc, char** argv)
{
  fcntl(atoi(argv[1]), F_SETPIPE_SZ, atoi(argv[2]));
  execvp(argv[3],argv+3);
  while (1);
}

as in:

adapter 1 (BIGSIZE) cmd1 | cmd2

or even:

adapter 1 (BIGSIZE) cmd1 | adapter 1 (BIGSIZE) buffer [args] | cmd2

if cmd1 still makes small writes.

  • By downstream command I meant buffer. cmd1 can't write more than 64KB (the pipe capacity) before buffer needs to wake up. And that obviously applies between buffer and cmd2 as well, since the pipe capacity is always 64KB. – CAFxX Dec 5 '16 at 21:10
  • @CAFxX Oh I see your point now. Updating. – DepressedDaniel Dec 5 '16 at 21:32
1

Based on DepressedDaniel and Stéphane Chazelas suggestions I settled on the closest thing to a oneliner I could find:

function hugepipe {
  perl -MFcntl -e 'fcntl(STDOUT, 1031, 1048576) or die $!; exec { $ARGV[0] } @ARGV or die $!' "$@"
}

This allows to do:

hugepipe <command> | <command>

and the pipe between the two commands is going to have the capacity specified via the fcntl in the perl script.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.