Hot answers tagged pipe
136
It's not useless - it's a specialised form of the plain > redirect operator (and, perhaps confusingly, nothing to do with pipes). bash and most other modern shells have an option noclobber, which prevents redirection from overwriting or destroying a file that already exists. For example, if noclobber is true, and the file /tmp/output.txt already exists, ...
33
This is an unabashed yes. When one uses ssh to execute a command on a remote server it performs some kind of fancy internal input/output redirection. In fact, I find this to be one of the subtly nicer features of OpenSSH. Specifically, if you use ssh to execute an arbitrary command on a remote system, then ssh will map STDIN and sTDOUT to that of the command ...
26
cd is not an external command - it is a shell builtin function. It runs in the context of the current shell, and not, as external commands do, in a fork/exec'd context as a separate process.
Your third example works, because the shell expands the variable and the command substitution before calling the cd builtin, so that cd receives the value of ${HOME} as ...
25
Sort of. The shell has no idea what the commands you are running will do, it just connects the output of one to the input of the other.
If grep finds more than 10 lines that say "hello world" then head will have all 10 lines it wants, and close the pipe. This will cause grep to be killed with a SIGPIPE, so it does not need to continue scanning a very ...
23
bahamat and Alan Curry have it right: this is due to the way your shell buffers the output of echo. Specifically, your shell is bash, and it issues one write system call per line. Hence the first snippet makes 1000000 writes to a disk file, whereas the second snippet makes 1000000 writes to a pipe and sed (largely in parallel, if you have multiple CPUs) ...
22
The capacity of a pipe buffer varies across systems (and can even vary on the same system). I am not sure there is a quick, easy, and cross platform way to just lookup the capacity of a pipe.
Mac OS X, for example, uses a capacity of 16384 bytes by default, but can switch to 65336 byte capacities if large write are made to the pipe, or will switch to a ...
21
cd does not read standard input. That is why your first example does not work.
xargs needs a command name, that is, a name of an independant executable. cd needs to be a shell built-in command and would have no effect (other than verifying that you can change to that directory and the potential side effects it may have like for automountable directories) if ...
20
The order the commands are run actually doesn't matter and isn't guaranteed. Leaving aside the arcane details of pipe(), fork(), dup() and execve(), the shell first creates the pipe, the conduit for the data that will flow between the processes, and then creates the processes with the ends of the pipe connected to them. The first process that is run may ...
19
The "definitive" answer is of course brought to you by The Useless Use of cat Award.
The purpose of cat is to concatenate (or "catenate") files. If it's only one file, concatenating it with nothing at all is a waste of time, and costs you a process.
Instantiating cat just so your code reads differently makes for just one more process and one more set ...
18
mknod was originally used to create the character and block devices that populate /dev/. Nowadays software like udev automatically creates and removes device nodes on the virtual filesystem when the corresponding hardware is detected by the kernel, but originally /dev was just a directory in / that was populated during install.
So yes, in case of a near ...
17
You'd think there'd be a utility for that, but I couldn't find it. However, this Perl one-liner should do the trick:
perl -pe 's/\e\[?.*?[\@-~]//g'
Example:
$ command-that-produces-colored-output | perl -pe 's/\e\[?.*?[\@-~]//g' > outfile
Or, if you want a script you can save as stripcolorcodes:
#! /usr/bin/perl
use strict;
use warnings;
while ...
17
A good way to grok the difference between them is to do a little experimenting on the command line. In spite of the visual similarity in use of the < character, it does something very different than a redirect or pipe.
Let's use the date command for testing.
$ date | cat
Thu Jul 21 12:39:18 EEST 2011
This is a pointless example but it shows that cat ...
17
Named pipes (fifo) have four three advantages I can think of:
you don't have to start the reading/writing processes at the same time
you can have multiple readers/writers which do not need common ancestry
as a file you can control ownership and permissions
they are bi-directional, unnamed pipes may be unidirectional *
*) Think of a standard shell | ...
16
About your performance question, pipes are more efficient than files because no disk IO is needed. So cmd1 | cmd2 is more efficient than cmd1 > tmpfile; cmd2 < tmpfile (this might not be true if tmpfile is backed on a RAM disk or other memory device as named pipe; but if it is a named pipe, cmd1 should be run in the background as its output can block ...
16
Well, the generic case that works with any command that writes to stdout is to use xargs, which will let you attach any number of command-line arguments to the end of a command:
$ find … | xargs grep 'search'
Or to embed the command in your grep line with backticks or $(), which will run the command and substitute its output:
$ grep 'search' $(find …)
...
16
Just the usual && and || operators:
cmd1 < input.txt |
cmd2 |
( [[ "${DEFINED}" ]] && cmd3 || cat ) |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
(Note that no trailing backslash is needed when the line ends with pipe.)
Update according to Jonas' observation.
If cmd3 may terminate with non-zero exit code and you not want cat to ...
14
There are 3 ways of doing this. However your current setup should work. The reason here being that the grep won't match anything if the command fails, so grep will return with status 1 (unless the program always shows that text no matter what).
Pipefail
The first way is to set the pipefail option. This is the simplest and what it does is basically set the ...
13
Well, its fairly "easy" with named pipes (mkfifo). I put easy in quotes because unless the programs are designed for this, deadlock is likely.
mkfifo fifo0 fifo1
( prog1 > fifo0 < fifo1 ) &
( prog2 > fifo1 < fifo0 ) &
( exec 30<fifo0 31<fifo1 ) # write can't open until there is a reader
# and ...
12
While not exactly what you asked, you could use
#!/bin/bash -o pipefail
so that your pipes return the last non zero return.
might be a bit less coding
Edit: Example
[root@localhost ~]# false | true
[root@localhost ~]# echo $?
0
[root@localhost ~]# set -o pipefail
[root@localhost ~]# false | true
[root@localhost ~]# echo $?
1
11
In bash, ulimit -p tells you.
$ ulimit -p
8
$ ulimit -a | grep pipe
pipe size (512 bytes, -p) 8
so on my system it's 8 * 512 = 4096 bytes.
If you're not using bash, you can use either PIPE_BUF from <limits.h>, e.g.:
/usr/include/linux/limits.h: #define PIPE_BUF 4096
Or pathconf, e.g. using Python:
>>> os.pathconf('.', ...
11
fschmitt's answer is the best when using sed; however, in a more general sense this anti-pattern:
cat infile | filter > infile
is likely to cause you a good number of problems. For instance if I have a file called infile that looks like this:
Hello
World
and run this command:
cat infile | tr "[:upper:]" "[:lower:]"
I get
hello
world
But if ...
11
Many programs that generate colored output detect if they're writing to a TTY, and switch off colors if they aren't. This is because color codes are annoying when you only want to capture the text, so they try to "do the right thing" automatically.
The simplest way to capture color output from a program like that is to tell it to write color even though ...
11
This is a well known bash pitfall, due to this feature:
Each command in a pipeline is executed as a separate process (i.e., in a subshell).
so that modified variables are local to the subshell, and not visible once back in the parent.
To avoid that, rephrase your code to avoid the pipeline, with a process substitution:
for arg in "$@"
do
...
11
I am not sure what shell sh.exe provides (since there are multiple shells that use that name for their Windows executables), but if it is bash or similar, you can use the $PIPESTATUS array. For your example, you would do:
g++ -c source.cpp -o source.o 2>&1 | perl /bin/gSTLFilt.pl
echo "${PIPESTATUS[0]}"
11
When a program tries to write to a pipe and there is no process reading from that pipe, then the writer program receives a SIGPIPE signal. The default action when a program receives SIGPIPE is to terminate the program. A program can choose to ignore the SIGPIPE signal, in which case the write returns an error (EPIPE).
In your example, here's a timeline of ...
11
Standard input and standard output are not commands.
Imagine commands as machines in a factory with an assembly line. Most machines are designed to have one conveyor belt to feed data in and one conveyor belt to feed data out; they are the standard input and the standard output respectively. The standard error is an opening on the side of the machine where ...
11
A convenient way of piping data between hosts when you don't need to worry about security over the wire is using netcat on both ends on the connection.
This also lets you set them up asynchronously:
On the "receiver" (really, you'll have two-way communication, but it's easier to think of it like this), run:
nc -l -p 5000 > /path/to/backupfile.tar
And ...
10
That's the inode number for the pipe or socket in question.
A pipe is a unidirectional channel, with a write end and a read end. In your example, it looks like FD 5 and FD 6 are talking to each other, since the inode numbers are the same. (Maybe not, though. See below.)
More common than seeing a program talking to itself over a pipe is a pair of separate ...
10
If it is a problem with the libc modifying its buffering / flushing when output does not go to a terminal, you should try socat. You can create a bidirectional stream between almost any kind of I/O mechanism. One of those is a forked program speaking to a pseudo tty.
socat EXEC:long_running_command,pty,ctty STDIO
What it does is
create a pseudo tty
...
Only top voted, non community-wiki answers of a minimum length are eligible
