15

How does a FIFO (named pipe) differs from a regular pipe (|)? As I understand from Wikipedia, unlike a regular pipe, a FIFO pipe "keeps living" after the process has ended and can be deleted sometime afterwards.

But if f the process is based on a shell command containing a pipe (cat x | grep y), we could "keep it alive after the process" if we store it in a variable or a file, isn't it then a FIFO?

Also, a regular pipe also has the first stdout it gets, as stdin for another command, so isn't it also a kind of first in first out pipe as well?

Improve this question
1

2 Answers 2

Reset to default
26

"Named pipe" is actually a very accurate name for what it is — it is just like a regular pipe, except that it has a name (on a filesystem).

A pipe — the regular, un-named ("anonymous") one used in some-command | grep pattern is a special kind of file. And I do mean file, you read and write to it just like you do every other file. Grep doesn't really care¹ that it's reading from a pipe instead of a terminal³ or an ordinary file.

Technically, what goes on behind the scenes is that stdin, stdout, and stderr are three open files (file descriptors) passed to every command run. File descriptors (which are used in every syscall to read/write/etc. files) are just numbers; stdin, stdout, and stderr are file descriptors 0, 1, and 2. So when your shell sets up some-command | grep what it does is something this:

  1. Asks the kernel for an anonymous pipe. There is no name, so this can't be done with open like for a normal file — instead it's done with pipe or pipe2, which returns two file descriptors.⁴

  2. Forks off a child process (fork() creates a copy of the parent process; both sides of the pipe are open here), copies the write-side of the pipe to fd 1 (stdout). The kernel has a syscall to copy around file descriptor numbers; it's dup2() or dup3(). It then closes the read side and other copy of the write side. Finally, it uses execve to execute some-command. Since the pipe is fd 1, stdout of some-command is the pipe.

  3. Forks of another child process. This time, it duplicates the read side of the pipe to fd 0 (stdin), and executes grep. So grep will read from the pipe as stdin.

  4. Then it waits for both of those children to exit.

  5. At this point, the kernel notices that the pipe isn't open any more, and garbage collects it. That's what actually destroys the pipe.

A named pipe just gives that anonymous pipe a name by putting it in the filesystem. So now any process, at any point in the future, can obtain a file descriptor for the pipe by using an ordinary open syscall. Conceptually, the pipe won't be destroyed until both all readers/writers have closed it and it's unlinked from the filesystem.²

This, by the way, is how files in general work on Unix. unlink (the syscall behind rm) just removes one of the names for the file; only when all names are removed and nothing has the file open will it actually be deleted. A couple of answers here explore this:

FOOTNOTES

  1. Technically this probably isn't true — it's probably possible to do some optimizations by knowing, and actual grep implementations have often been heavily optimized. But conceptually it doesn't care (and indeed a straightforward implementation of grep wouldn't).
  2. Of course the kernel doesn't actually keep all the data structures around in memory forever, but rather it recreates them, transparently, whenever the first program opens the named pipe (and then keeps them as long as its open). So it's as if they existed as long as the name does.
  3. Terminal isn't a common place for grep to read from, but it's the default stdin when you don't specify another. So if you type just grep pattern in to your shell, grep will be reading from the terminal. The only use for this that comes to mind is if you're about to paste something to the terminal.
  4. On Linux, anonymous pipes actually are created on a special filesystem, pipefs. See How pipes work in Linux for details. Note that this is an internal implementation detail of Linux.
Improve this answer
12
  • 1
    Some grep implementations have an option to mmap input files if possible, instead of read()ing. (GNU grep no longer has such an option, if it ever did, because it's usually not a performance win these days.) This is of course only possible for regular file and block devices, not anonymous or named pipes. Most grep implementations would want to seek in their input, but if stdin is a regular file, it's seekable. This matters for stuff like a video file: cat foo.mkv | mpv - vs. <foo.mkv mpv -. The latter will notice that stdin is seekable and take advantage.
    – Peter Cordes
    Commented Apr 10, 2018 at 23:53
  • @PeterCordes yep, that'd be one of those optimizations (if it actually were more optimal). There are no doubt others, like reading in large chunks or maybe someone even has a multithreaded grep.
    – derobert
    Commented Apr 11, 2018 at 0:00
  • Grep doesn't really care¹ that it's reading from a pipe instead of a terminal, what is greap reading from terminal? Never came across such need, and thanks for the answer.
    – Arcticooling
    Commented Apr 11, 2018 at 9:22
  • 1
    @user9303970 FIFO is a name from its behavior: the first thing written ("first in") is the first thing read ("first out"). It's a somewhat odd name, though, since e.g., stream sockets work the same way, but aren't called FIFOs.
    – derobert
    Commented Apr 11, 2018 at 16:42
  • 1
    @user9303970 Added two more footnotes for you.
    – derobert
    Commented Apr 11, 2018 at 16:50
6

I think you're getting mixed up between shell syntax for pipelines vs. the underlying Unix systems programming. A pipe / FIFO is a type of file that isn't stored on disk, but instead passed data from a writer to a reader through a buffer in the kernel.

A pipe/FIFO works the same whether the writer and reader got connected by making system calls like open("/path/to/named_pipe", O_WRONLY);, or with a pipe(2) to create a new anonymous pipe and return open file descriptors to both the read and write sides.

fstat(2) on the pipe file descriptor will give you sb.st_mode & S_IFMT == S_IFIFO either way.

When you run foo | bar:

  • The shell forks like normal for any non-builtin command
  • Then makes a pipe(2) system call to get two file descriptors: the input and output of an anonymous pipe.
  • Then it forks again.
  • The child (where fork() returned 0)
    • closes the read side of the pipe (leaving the write fd open)
    • and redirects stdout to the write fd with dup2(pipefd[1], 1)
    • then does execve("/usr/bin/foo", ...)
  • The parent (where fork() returned the non-0 child PID)
    • closes the write side of the pipe (leaving the read fd open)
    • and redirects stdin from the read fd with dup2(pipefd[0], 0)
    • then does execve("/usr/bin/bar", ...)

You get into a very similar situation if you run foo > named_pipe & bar < named_pipe.

A named pipe is a rendezvous for processes to establish pipes between each other.

The situation is similar to anonymous tmp files vs. files with names. You can open("/path/to/dir", O_TMPFILE | O_RDWR, S_IRUSR | S_IWUSR); to create a temporary file with no name (O_TMPFILE), as if you'd opened "/path/to/dir/tmpfile" with O_CREAT and then unlinked it, leaving you with a file descriptor to a deleted file.

Using linkat, you can even link that anonymous file into the filesystem, giving it a name, if it was created with O_TMPFILE. (You can't do this on Linux for files that you created with a name then deleted, though.)

Improve this answer

Not the answer you're looking for? Browse other questions tagged .