Chains the standard streams of a series of commands
79
votes
2answers
3k views
Bash: What does “>|” do?
I have just seen this written down;
$ some-command >| /tmp/output.txt
Vertical pipes are used in standard redirects "piping" the output of one command to another, is >| in fact completely ...
47
votes
8answers
12k views
Turn off buffering in pipe
I have a script which calls two commands:
long_running_command | print_progress
The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice ...
24
votes
2answers
1k views
In what order do piped commands run?
I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about ...
23
votes
10answers
5k views
Get exit status of process that's piped to another
I have two processes foo and bar, connected with a pipe:
$ foo | bar
bar always exits 0; I'm interested in the exit code of foo. Is there any way to get at it?
23
votes
3answers
3k views
Can I pipe stdout on one server to stdin on another server?
stdout on one CentOS server needs to be piped to stdin on another CentOS server. Is this possible?
Update
ScottPack, MikeyB and jofel all have valid answers. I awarded the answer to Scott because, ...
21
votes
2answers
1k views
What are the advantages of using named pipe over unnamed pipe?
I was reviewing a set of interview questions that are asked from a unix admin; I found a topic called "named pipe".
I googled the topic; to some extent I have been able to understand it :- named ...
20
votes
3answers
13k views
How big is the pipe buffer?
As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote:
Another reason to avoid | true is that if the command ...
20
votes
1answer
656 views
Why does echo >file use more real time than echo | sed >file?
The example, below, surprised me. It seems to be counter intuitive... aside from the fact that there is a whisker more user time for the echo | sed combo.
Why is echo using so much sys time when it ...
19
votes
6answers
1k views
Why can't I redirect a path name output from one command to “cd”?
I am trying to get cd to accept a directory name redirected to it from another command. Neither of these methods work:
$ echo $HOME | cd
$ echo $HOME | xargs cd
This does work:
$ cd $(echo $HOME)
...
18
votes
4answers
599 views
Are Linux utilities smart when running piped commands?
I was just running a few commands in a terminal and I started wondering, does Unix/Linux take shortcuts when running piped commands?
For example, let's say I have a file with one million lines, the ...
17
votes
2answers
508 views
Should I care about unnecessary cats?
A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
16
votes
3answers
3k views
Program that passes STDIN to STDOUT with color codes stripped?
I have a command that produces output in color, and I would like to pipe it into a file with the color codes stripped out. Is there a command that works like cat except that it strips color codes? I ...
16
votes
4answers
6k views
What is the mknod command used for?
I just started using Ubuntu as my main OS and I wanted to learn about things I should not do, and learn by the bad things people have done in the past. I came across these email about horror stories ...
15
votes
3answers
1k views
How to make bidirectional pipe between two programs?
Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second.
But how to make bidirectional pipe, i.e. cross-bind stdin and ...
14
votes
4answers
4k views
Can't pipe into diff?
I wanted to be clever and compare a remote file to a local file without downloading it. I can get the contents of the remote file by
ssh user@remote-host "cat path/file.name"
However, piping that ...
14
votes
4answers
1k views
Process substitution and pipe
I was wondering how to understand the following:
Piping the stdout of a command into the stdin of another is a powerful
technique. But, what if you need to pipe the stdout of multiple
...
14
votes
4answers
1k views
How to understand pipes
When I just used pipe in bash, I didn't think more about this. But when I read some C code example using system call pipe() together with fork(), I wonder how to understand pipes, including both ...
14
votes
1answer
246 views
Make program first read from pipe, then from keyboard
Consider the interactive program interactive. I have to run this program fairly frequently, and each time I run it the first few commands are the same. Obviously, having to type those commands over ...
13
votes
2answers
438 views
Is there a way to pipe the output of one program into two other programs?
Sorry if this is a silly question but I'm trying to accomplish something like this but on one line:
$ prog1 | prog2
$ prog1 | prog3
So, I basically want to execute prog1 and pipe the output to ...
12
votes
6answers
553 views
Conditional pipeline
Say I've got the following pipeline:
cmd1 < input.txt |\
cmd2 |\
cmd4 |\
cmd5 |\
cmd6 |\
(...) |\
cmdN > result.txt
Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. ...
12
votes
2answers
2k views
How do I pass a list of files to grep
I am using find and getting a list of files I want to grep through. How do I pipe that list to grep?
11
votes
3answers
1k views
How can I pipe output to another process, but retain the error state of the first process? [duplicate]
Possible Duplicate:
Get exit code of process that's piped to another
I am using the following command line (in a makefile) to pipe the verbose error messages from my compiler through a ...
10
votes
8answers
3k views
What's a good example of piping commands together?
If you were helping someone to learn the concept of pipes on the command line what example would you use? The example that actually came up was as follows:
cat whatever.txt | less
I feel like ...
9
votes
1answer
187 views
What are guarantees for concurrent writes into a named pipe?
For example, I created a named pipe like the following:
mknod myPipe p
And I read from it from some process (for example, some server). For example purposes, I used tail:
tail -f myPipe
If ...
8
votes
5answers
2k views
How to pipe output from one process to another but only execute if the first has output?
How can I rewrite this command to only email if there is output from the mailq | grep?
mailq | egrep 'rejected|refused' -A 5 -B 5 | mail -s 'dd' email@email
Is this even possible on one line?
8
votes
4answers
3k views
Difference between 2>&1 > output.log and 2>&1 | tee output.log
I wanted to know the difference between the following two commands
2>&1 > output.log
and
2>&1 | tee output.log
I saw one of my colleague use second option to redirect. I know ...
7
votes
2answers
2k views
/proc/PID/fd/X link number
In Linux, in /proc/PID/fd/X, the links for file descriptors that are pipes or sockets have a number, like:
l-wx------ 1 user user 64 Mar 24 00:05 1 -> pipe:[6839]
l-wx------ 1 user user 64 Mar 24 ...
7
votes
3answers
365 views
Piping STDERR vs. STDOUT
According to "Linux: The Complete Reference 6th Edition" (pg. 44), you can pipe only STDERR using the |& redirection symbols.
I've written a pretty simple script to test this:
#!/bin/bash
echo ...
7
votes
4answers
559 views
What is the easiest way to execute text from tail at the command line?
Sometimes I'm working on a new (ubuntu) box and I type git and am alerted:
The program 'git' is currently not installed. You can install it by typing:
apt-get install git-core
If that happens I ...
7
votes
1answer
135 views
“w | tail” chops text at right margin (78 characters)
Summary: w | tail chops rows after 78th column.
When I run the "w" command, it works as expected. However, when I pipe the output to "tail", it chops the output to 78 columns wide (truncates anything ...
7
votes
2answers
161 views
How a piped shell programs balance their output/input rates? [duplicate]
Possible Duplicate:
Bash while loop and reading from pipe
I come from web programming background, and find myself interested in one peculiarity of using a local shell. I understand that ...
7
votes
4answers
318 views
Creating a single output stream out of three other streams produced in parallel
I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format.
This Python script is slow and CPU-bound (to a ...
7
votes
3answers
1k views
Using in/out named pipes for a TCP connection
I've been fiddling with getting this to work for a while now, so I suspect some sort of fundamental misunderstanding about how pipes work is the root cause of my troubles.
My goal is to initiate a ...
6
votes
4answers
220 views
How can I print a conditional header BEFORE stdout, if there is any output on stdout
I have a process that filters a list of files from a directory (having find check to see if there are files older than a certain period to show a queue is stuck). It may or may not return anything, ...
6
votes
3answers
1k views
Find -exec + vs find | xargs. Which one to choose?
I understand that the -exec can take a + option to mimic the behaviour of xargs. Is there any situation where you'd prefer one form over the other?
I personally tend to prefer the first form, if ...
6
votes
3answers
4k views
Pipe find into grep -v
I'm trying to find all files that are of a certain type and do not contain a certain string. I am trying to go about it by piping find to grep -v
example:
find -type f -name '*.java' | xargs grep -v ...
6
votes
2answers
919 views
bash: Piping for loop output prevents local variable modification
First off, sorry for the title. I'm not sure of the correct terminology so if anyone can improve upon it that would be good.
I am trying to write a simple bash function that takes, as it's arguments, ...
6
votes
2answers
1k views
Where do my ANSI escape codes go when I pipe to another process? Can I keep them?
I sometime want to pipe the color-coded output fror a process, eg. grep... but when I pipe it to another process, eg. sed, the color codes are lost...
Is the some way to keep thes codes intact ?
...
6
votes
3answers
2k views
Non-blocking buffered named pipe?
I'm looking for something I suspect doesn't exist: A non-blocking buffered named pipe (fifo) for use from the command line. Is there such a thing?
Here's the use case: Suppose I have a process that's ...
6
votes
3answers
3k views
Redirect stdout over ssh
I would like to run
something > file
on a remote system through ssh, but if I run
ssh host something > file
the redirection is executed locally as ssh etc > file
I've tried it with ' ...
6
votes
1answer
206 views
Why mkfifo behaves like a LIFO?
If i do.
mkfifo /tmp/a
echo 'one'>/tmp/a
in the while from another terminal
echo 'two'>/tmp/a
and from a third terminal
more /tmp/a
Why i obtain as output of the last command this?
...
6
votes
2answers
403 views
Piping output from a segfaulting program
I have a script that calls a program (specifically, ttf2afm, part of tetex 3.0) that sometimes segfaults and sometimes doesn't. The information I need is always printed out before it segfaults, but ...
6
votes
3answers
1k views
Have Bash script wait for status message before continuing
I'm firing up Selenium server with a bash script and as you can see from the timestamps on the log below, it takes about 32 seconds for the thing to fully come online:
Feb 28, 2012 10:19:02 PM ...
6
votes
1answer
440 views
Readable comments on separate lines in a multi-line bash command with pipelines?
When creating shell scripts using pipelines, and using the backslash to continue lines, I want to insert comments on separate lines, in a robust, readable and portable fashion.
For example, given ...
6
votes
1answer
523 views
How can I tell if the pipe buffer is full?
I am piping output from one program into some Perl I wrote. This is a long running process , sometimes days, so I want to find out where my bottlenecks are and try to open them up. I want to know if ...
6
votes
1answer
239 views
echo vs <<<, or Useless Use of echo in Bash Award?
By now the Useless Use of cat Award is very well known, and there's also a mention of a Useless Use of echo (not relevant for this question). I'm wondering if there should be a "Useless Use of echo in ...
5
votes
5answers
353 views
resolve all ip addresses in command output using standard command line tools
I have several log files that contain a bunch of ip addresses. I would love to be able to pipe the data through a program that would match and resolve ip addresses.
I.E.
cat /var/log/somelogfile | ...
5
votes
2answers
2k views
what is meant by connecting STDOUT and STDIN?
This is a newbie question. I'm reading a book, it says:
Every process has at least three communication channels available to it: “standard
input” (STDIN), “standard output” (STDOUT), and ...
5
votes
3answers
216 views
Utility to buffer an unbounded amount of data in a pipeline?
Is there a utility that I can stick in a pipeline to decouple read and write speeds?
$ producer | buf | consumer
Basically, I want a utility buf that reads its input as fast as possible, storing it ...
5
votes
1answer
132 views
Confusion about sed and '>'
The following command works fine:
sed s/input/raw_input/ p.py >p2.py && mv p2.py p.py
However, the following command turns p.py into an empty file:
sed s/input/raw_input/ p.py >p.py
...
