12

I hear that for named pipes, writes that are smaller than about 512bytes are atomic (the writes won't interleave).

Is there a way to increase that amount for a specific named pipe?

something like:

mkfifo mynamedpipe --buf=2500

Supposedly this is the full documentation: http://www.gnu.org/software/coreutils/manual/html_node/mkfifo-invocation.html#mkfifo-invocation

man mkfifo takes me to that page.

1
  • also wondering how to lookup the buffer size on a given platform, not sure how to do that either. Commented Apr 21, 2018 at 23:08

2 Answers 2

17

A fifo file is just a type of file which when opened for both reading and writing instantiates a pipe like a pipe() system call would.

On Linux at least, the data that transits though that pipe is not stored on the file system at all (only in the kernel as kernel memory). And the size attribute of the fifo file is not relevant and is always 0.

On Linux, you can change the size of a pipe buffer (whether that pipe has been instantiated with pipe() or via opening a fifo file) with the F_SETPIPE_SZ fcntl(), though for unprivileged users, that's bound by /proc/sys/fs/pipe-max-size. Any of the writer or reader to the pipe can issue that fcntl() though it makes more sense for the writer to do it. In the case of a named pipe, you'd need to do that for each pipe instantiated though the fifo file.

$ mkfifo fifo
$ exec 3<> fifo # instantiate the pipe
$ seq 20000 > fifo
^C  # seq hangs because it's trying to write more than 64KiB
$ exec 3<&- 3<> fifo # close the first pipe and instantiate a new one
$ (perl -MFcntl -e 'fcntl(STDOUT, 1031, 1048576)'; seq 20000) > fifo
$ # went through this time

Above, I used perl to issue the fcntl(), harcoding the value of F_SETPIPE_SZ (1031 on my system).

1

Instead of changing F_SETPIPE_SZ, there is a bash shell implementation that have the same effect, replacing the expression fifo by a substitution of processus like this : >(stdbuf -o100M grep -a '^' > fifo) (it will be substituted at runtime by the file descriptor /dev/fd/63)

mkfifo fifo
exec 3<> fifo
seq 12000000 > >(stdbuf -o100M grep -a '^' > fifo)

Not all filters are sensitive to changing buffering mode by stdbuf, you could not simply use cat for example. For the same reason you cannot either say stdbuf -o100M seq 12000000 > fifo. As far as I know, classical sed grep or awk do it.

The drawback of this is to mobilize one more process ; the advantage is that you are free to implement a huge buffer size.

3
  • Note that >(...) is not specific to nor does it originate in bash. It's from the Korn shell in the mid 80 and is also found in zsh. The same or equivalent feature with different syntax is also found in a few other shells (rc and derivatives, fish, yash). Commented Jun 3, 2023 at 7:34
  • Here it relies on the fact that bash is not waiting for the process started for the substitution. Because it is otherwise generally not desirable (as that means that in cmd1 >(cmd2 > file); cmd3 file, cmd3 may be started before cmd2 has finished generating file for instance), it might change in the future. Commented Jun 3, 2023 at 7:36
  • 2
    see pv -qB 100M for a more straightforward and efficient way to buffer those 100M worth of data in memory (+ 2 pipe buffers) Commented Jun 3, 2023 at 7:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .