Questions tagged [fifo]
Questions about FIFO - first-in first-out special file, also known named pipe
246
questions
55
votes
5
answers
41k
views
Are FIFO, pipe & Unix domain socket the same thing in Linux kernel?
I heard that FIFOs are named pipes. And they have exactly the same semantics. On the other hand, I think Unix domain socket is quite similar to pipe (although I've never made use of it). So I wonder ...
- 713
25
votes
2
answers
5k
views
Are the named pipe created by `mknod` and the FIFO created by `mkfifo` equivalent?
I've used the mkfifo <file> command to create named FIFOs, where one process writes to the file, and another process reads from the file.
Now, I know the mknod command is able to create named ...
- 4,203
24
votes
3
answers
10k
views
Prevent automatic EOFs to a named pipe, and send an EOF when I want it
I have a program that exits automatically upon reading an EOF in a given stream ( in the following case, stdin ).
Now I want to make a shell script, which creates a named pipe and connect the program'...
- 3,468
22
votes
3
answers
77k
views
continuous reading from named pipe (cat or tail -f)
I have configured rsyslog to log certain log events to /dev/xconsole:
*.*;cron.!=info;mail.!=info |/dev/xconsole
/dev/xconsole is a named pipe (fifo). If I want to see what is being logged, I ...
- 483
21
votes
1
answer
9k
views
Why is a named pipe as slow as writing to a file?
I'm trying to understand how named pipes work so that I can streamline my one-way interprocess communication. I expect some overhead due to copying data into a circular buffer, which I would have ...
- 333
16
votes
3
answers
5k
views
A virtual file containing the concatenation of other files
Is there a way of creating a filesystem object akin to this:
mknod files p
cat file1 file2 ... fileN > files
but such that it can be seeked in, as if it were a regular file?
- 781
16
votes
1
answer
1k
views
How to get an average pipe flow speed
If myfile is increasing over time, I can get the number of line per second using
tail -f | pv -lr > /dev/null
It gives instantaneous speed, not average.
How can I get the average speed (i.e the ...
- 4,948
15
votes
2
answers
11k
views
How does a FIFO (named pipe) differs from a regular pipe (unnamed pipe)? [duplicate]
How does a FIFO (named pipe) differs from a regular pipe (|)? As I understand from Wikipedia, unlike a regular pipe, a FIFO pipe "keeps living" after the process has ended and can be deleted sometime ...
15
votes
2
answers
28k
views
Under what conditions exactly does SIGPIPE happen?
Say we have a named pipe called fifo, and we're reading and writing to it from two different shells. Consider these two examples:
shell 1$ echo foo > fifo
<hangs>
shell 2$ cat fifo
foo
shell ...
- 252
15
votes
1
answer
8k
views
How do I use inotify or named pipes over SSHFS?
Thanks sshfs magic, I can mount my home dir from a remote server with
sshfs user@server:/home/user ~/remote
Optimistically, I thought I'd set a local inotify-hook on ~/remote/logFile (in the sshfs ...
- 4,516
14
votes
9
answers
4k
views
Filter or pipe certain sections of a file
I have an input file with some sections the are demarcated with start and end tags, for example:
line A
line B
@@inline-code-start
line X
line Y
line Z
@@inline-code-end
line C
line D
I want to ...
- 434
14
votes
1
answer
1k
views
“Leaky” pipes in linux
Let's assume you have a pipeline like the following:
$ a | b
If b stops processing stdin, after a while the pipe fills up, and writes, from a to its stdout, will block (until either b starts ...
- 231
14
votes
2
answers
12k
views
Can you scp, sftp, or rsync, a pipe?
I want to tar and send a ~700GiB directory to a remote drive I don't control. I don't have the HDD space locally to create the tarball and then copy that over. The remote is also protected by rssh - ...
- 141
12
votes
3
answers
6k
views
Named pipes, file descriptors and EOF
Two windows, same user, with bash prompts. In window-1 type:
$ mkfifo f; exec <f
So bash is now attempting to read from file descriptor 0, which is mapped to named pipe f. In window-2 type:
$ ...
- 1,901
12
votes
2
answers
17k
views
Change buffer size of named pipe
I hear that for named pipes, writes that are smaller than about 512bytes are atomic (the writes won't interleave).
Is there a way to increase that amount for a specific named pipe?
something like:
...
- 9,718
12
votes
2
answers
7k
views
mkfifo - Does disk I/O actually occur?
I have 2 applications:
Producer (N instances)
Consumer (1 instance)
I currently write out intermediate results from the producers, and then the consumer reads these files from disk and produces a ...
- 546
12
votes
3
answers
7k
views
Buffering (named) pipe in GNU OS
In the GNU OS a process can only write data to a pipe if another process reads the same data (from the same pipe) at the same time.
Is there something like a pipe which lets the 1st process write and ...
- 557
11
votes
2
answers
5k
views
Linux named pipes: not as FIFO as thought
In short:
mkfifo fifo; (echo a > fifo) &; (echo b > fifo) &; cat fifo
What I expected:
a
b
since the first echo … > fifo should be the first to have opened the file, so I expect ...
- 25.1k
11
votes
4
answers
1k
views
Program output redirection
When trying to redirect program output with the "some number greater than" syntax (eg foo 2> myfile), what are the possible numbers here and what do they represent?
I believe 1 is /dev/stdout, 2 ...
11
votes
6
answers
4k
views
Semi-asynchronous pipe
Assume I have the following pipe:
a | b | c | d
How can I wait for the completion of c (or b) in sh or bash? This means that script d can start any time (and does not need to be waited for) but ...
- 947
11
votes
3
answers
2k
views
What characterizes a file in Linux/Unix?
What characterizes a file in Linux/Unix?
A file can have many types: regular file, directory, symlink, device file, socket, pipe, fifo, and more that I miss. For example, a symlink:
$ sudo file /...
- 100k
10
votes
3
answers
948
views
How to implement "generators" like $RANDOM?
The special variable $RANDOM has a new value every time it's accessed. In this respect, it is reminiscent of the "generator" objects found in some languages.
Is there a way to implement something ...
- 15k
9
votes
2
answers
13k
views
Send EOF to named pipe - cleaning up / drying up fifo
If I have some random processes reading from a named pipe:
tail -f MYNAMEDPIPED
cat MYNAMEDPIPE | someOtherProc
Elsewhere, I have a handle on MYNAMEDPIPED by name.
is there a safe and clean way to ...
- 9,718
9
votes
2
answers
11k
views
Read named pipe once every time input is written
I need to write a bash program that runs commands echoed to a named pipe it reads, but I cannot get it work only when a command is sent. It keeps repeating the last command until a new one is written.
...
9
votes
2
answers
16k
views
Why does this script with a FIFO pipe not terminate?
This script:
#!/bin/bash
tmppipe=/tmp/temppipe
mkfifo $tmppipe
echo "test" > $tmppipe
cat $tmppipe
exit
does not terminate. I assume that the cat command is waiting for an EOF from the ...
- 5,832
9
votes
1
answer
1k
views
Why doesn't file redirection to and from a named pipe work, but piping to cat does?
This is a simple echo server in Unix, using nc:
mkfifo fifo
cat fifo | nc -k -l 4458 -v | cat >fifo
(based on this)
As I can see it, the data flow works as follows:
fifo (my named pipe)
|
| (...
- 215
8
votes
1
answer
8k
views
Increase FIFO size limit
Thanks to the answers to my other question, I now understand that FIFO on Linux, i.e. /dev/xconsole has a buffer limit of 64 KB.
How can I increase this limit to 128 KB?
Apparently, I will need to ...
- 483
8
votes
2
answers
7k
views
Reading a named pipe: tail or cat?
I made a file descriptor using
mkfifo fifo
As soon as something is written to this pipe, I want to reuse it immediately. Should I use
tail -f fifo
or
while true; do cat fifo; done
?
They seem ...
- 895
8
votes
2
answers
3k
views
Does named pipe modify the filesystem?
If I create a named pipe and then read/write on it, is the filesystem where the named pipe resides affected? I.e. is the data buffered on the filesystem until read, or does it reside in memory only?
- 3,415
8
votes
2
answers
4k
views
How to cat named pipe without waiting
If there is nothing in a named pipe and I do:
cat my_named_pipe
it will wait until data arrives. Is there a flag I can use to exit immediately if there is no data to be read? Or perhaps a command ...
- 9,718
8
votes
1
answer
29k
views
How to save an output of airodump-ng to a file?
I've tried to get a file from airodump-ng via redirecting a output stream via:
airodump-ng mon0 2>&1 | tee file.txt
but this appends to a file rather than rewrites it. So after that I've ...
- 452
8
votes
1
answer
595
views
What are the difference between those four commands (fifo, process substitution, redirection...)
My goal is to create a simple echo server using nc and a single fifo. I'm not looking for the best way to do it, I'm merely trying to understand the semantics of the following commands (when does fork ...
- 285
7
votes
2
answers
1k
views
Why process does not (seem to) exist before fifo read
I have a trivial Python script:
#!/usr/bin/python
import os, sys
sys.stderr.write('I am %s' % os.getpid())
sys.stderr.flush()
print "hello"
sys.stderr.write('I am done')
When I run this script ...
- 4,400
7
votes
1
answer
6k
views
Using exec 3> to keep a named pipe open
The process reading from a named pipe will normally terminate when the process writing to the pipe finishes writing (sends an EOF). In certain situations you may have different processes writing ...
- 732
7
votes
4
answers
4k
views
How do I make systemd sockets close when service is stopped?
I'm currently trying to make a systemd service with two Fifo sockets. These sockets map to stdout and stdin of the application. I'm currently using the following configuration files.
foo.service
[...
- 171
7
votes
1
answer
4k
views
How can I know whether writing to a named pipe would block?
I want to write to a named pipe only if it already has a reader. Currently, I'm using timeout to detect whether the attempt to write to the pipe blocked, like so:
#! /usr/bin/env bash
rm -f pipe
...
7
votes
1
answer
1k
views
bash scripting - read tarball from stdin
I have a task I need to script that I feel should be stupidly simple, but I'm actually having a rather tough time.
I have a short bash script that takes a specific type of application in tarball form ...
- 8,284
6
votes
2
answers
25k
views
Can I increase the system pipe buffer max?
I'd like to make a FIFO pipe with a buffer of ~5MB. I know the default FIFO pipe buffer max in linux is around 1MB. I see that it lives in /proc/sys/fs/pipe-max-size
I tried to set it as follows:
...
- 143
6
votes
3
answers
2k
views
Why does `cat` seem to iterate over JSON arrays?
I'm puzzled by this behaviour of cat when trying to output a heredoc containing JSON in bash 3.2:
input:
$ cat <(cat <<EOF
> {"x":[{"a":1,"b":2}]}
> EOF)
output:
{"x":["a":1]}
{"x":[...
- 363
6
votes
2
answers
626
views
coproc and named pipe behaviour under command substitution
I have a requirement to make a function in a zsh shell script, that is called by command substitution, communicate state with subsequent calls to the same command substitution.
Something like C's ...
- 175
6
votes
2
answers
828
views
grep --exclude option doesn't always skip named pipes
I have a directory that contains, among other files, 3 named pipes: FIFO, FIFO1, and FIFO11. If I try something like
grep mypattern *
in this directory, grep hangs forever on the named pipes, so I ...
- 73
6
votes
1
answer
379
views
Why does my named pipe keep getting modified?
I have a named fifo created from a C program like this:
res = mkfifo("/home/myfolder/myfifo", 0666);
after that there are only reads and writes.
Now, from this answer : Does a named pipe change the ...
- 163
6
votes
1
answer
3k
views
Print current number of bytes in a pipe
I can create a pipe to transfer information between two programs. I can even use a utility like pv to view progress of total data passing through the pipe. In my case I am using a named pipe. When ...
- 508
6
votes
2
answers
209
views
Difference between > and >> when used with a named pipe
Is there any difference between the two redirections in the following code?
mkfifo foo
echo > foo
echo >> foo
user282164
6
votes
2
answers
430
views
Is there a way to implement custom files that work like the 'files' in the /proc file system?
I'm looking for something like a persistent named pipe... something that I can cat or grep multiple times, and always get the current state of whatever process is feeding into the pipe.
For example, ...
6
votes
2
answers
7k
views
Problem with pipes. Pipe terminates when reader done
I am on OSX, using bash, trying to make sense of pipes. I wish to let a program communicate in two directions with a bash shell. I want to set this up in such a way that this is always the same shell, ...
- 83
6
votes
1
answer
2k
views
Communicating from bash with grunt shell
I'm tired of the slow startup time of hadoop fs just to query HDFS. This isn't a problem with HDFS itself though, because using HDFS file system commands within the Pig "grunt shell" is pretty fast. ...
- 459
5
votes
2
answers
13k
views
Why does a named pipe not get deleted after system restart?
As far as I understand, named pipes are not written to disk but are stored in memory. Here's how I created a named pipe -
$ mkfifo pipe21
$ grep "simple" SimpleDoc.txt > pipe21 &
[1] 2775
$ ...
- 1,117
5
votes
1
answer
860
views
Why does "cat /tmp/out1 > /tmp/in2 &" appear as "bash" with the "ps" command?
Consider:
mkfifo /tmp/out1
mkfifo /tmp/in2
cat /tmp/out1 > /tmp/in2 &
When I run
ps
it appears as bash in the CMD section.
Why?
user55443
5
votes
5
answers
5k
views
How to run a pipe safely and sequentially?
In Linux, is it possible to run a pipe:
cmd1 | cmd2
in such a way that:
cmd2 doesn't start running until cmd1 has completely finished, and
If cmd1 has an error, cmd2 doesn't run at all and the exit ...
- 4,194