Chains the standard streams of a series of commands

learn more… | top users | synonyms

118
votes
10answers
41k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice ...
99
votes
1answer
6k views

Bash: What does “>|” do?

I have just seen this written down; $ some-command >| /tmp/output.txt Vertical pipes are used in standard redirects "piping" the output of one command to another, is >| in fact completely ...
61
votes
12answers
23k views

Get exit status of process that's piped to another

I have two processes foo and bar, connected with a pipe: $ foo | bar bar always exits 0; I'm interested in the exit code of foo. Is there any way to get at it?
41
votes
4answers
4k views

In what order do piped commands run?

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about ...
38
votes
4answers
37k views

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command ...
30
votes
6answers
8k views

Program that passes STDIN to STDOUT with color codes stripped?

I have a command that produces output in color, and I would like to pipe it into a file with the color codes stripped out. Is there a command that works like cat except that it strips color codes? I ...
29
votes
4answers
9k views

Can I pipe stdout on one server to stdin on another server?

stdout on one CentOS server needs to be piped to stdin on another CentOS server. Is this possible? Update ScottPack, MikeyB and jofel all have valid answers. I awarded the answer to Scott because, ...
28
votes
4answers
11k views

What are the advantages of using named pipe over unnamed pipe?

I was reviewing a set of interview questions that are asked from a unix admin; I found a topic called "named pipe". I googled the topic; to some extent I have been able to understand it :- named ...
26
votes
4answers
2k views

Why does “tail -f … | tail” fail to produce any output?

Why does the following command not produce any output? $ tail -f /etc/passwd | tail After reading about buffering, I tried the following to no avail: $ tail -f /etc/passwd | stdbuf -oL tail Note ...
26
votes
3answers
1k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
25
votes
2answers
13k views

How do I pass a list of files to grep

I am using find and getting a list of files I want to grep through. How do I pipe that list to grep?
25
votes
4answers
19k views

What is the mknod command used for?

I just started using Ubuntu as my main OS and I wanted to learn about things I should not do, and learn by the bad things people have done in the past. I came across these email about horror stories ...
24
votes
6answers
2k views

Why can't I redirect a path name output from one command to “cd”?

I am trying to get cd to accept a directory name redirected to it from another command. Neither of these methods work: $ echo $HOME | cd $ echo $HOME | xargs cd This does work: $ cd $(echo $HOME) ...
24
votes
4answers
9k views

Can't pipe into diff?

I wanted to be clever and compare a remote file to a local file without downloading it. I can get the contents of the remote file by ssh user@remote-host "cat path/file.name" However, piping that ...
24
votes
1answer
1k views

Why does echo >file use more real time than echo | sed >file?

The example, below, surprised me. It seems to be counter intuitive... aside from the fact that there is a whisker more user time for the echo | sed combo. Why is echo using so much sys time when it ...
24
votes
3answers
785 views

Multicolored Grep

I'm trying to get each grep command to highlight it's results in a different color. I can do it manually with a line like this: ls -l GREP_COLORS='mt=01;32' grep c | GREP_COLORS='mt=01;31' grep o | ...
23
votes
1answer
602 views

How do I redirect command completion output?

On the command line, I can redirect or pipe output of a command to a file or another command using the > or | operator after the command. I have come across a less standard situation that I would ...
22
votes
5answers
2k views

Pipe to multiple files in the shell

I have an application which will produce a large amount of data which I do not wish to store onto the disk. The application mostly outputs data which I do not wish to use, but a set of useful ...
21
votes
4answers
877 views

Are Linux utilities smart when running piped commands?

I was just running a few commands in a terminal and I started wondering, does Unix/Linux take shortcuts when running piped commands? For example, let's say I have a file with one million lines, the ...
21
votes
5answers
6k views

How to make bidirectional pipe between two programs?

Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second. But how to make bidirectional pipe, i.e. cross-bind stdin and ...
20
votes
4answers
3k views

Process substitution and pipe

I was wondering how to understand the following: Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple ...
19
votes
4answers
917 views

Is there a way to pipe the output of one program into two other programs?

Sorry if this is a silly question but I'm trying to accomplish something like this but on one line: $ prog1 | prog2 $ prog1 | prog3 So, I basically want to execute prog1 and pipe the output to ...
17
votes
9answers
2k views

Grep lines but let the first line through [duplicate]

Consider the following output from df. Filesystem Size Used Avail Use% Mounted on /dev/sda1 23G 6.1G 16G 29% / udev 10M 0 10M 0% /dev tmpfs 397M 420K ...
16
votes
10answers
13k views

What's a good example of piping commands together?

If you were helping someone to learn the concept of pipes on the command line what example would you use? The example that actually came up was as follows: cat whatever.txt | less I feel like ...
15
votes
5answers
1k views

Conditional pipeline

Say I've got the following pipeline: cmd1 < input.txt |\ cmd2 |\ cmd4 |\ cmd5 |\ cmd6 |\ (...) |\ cmdN > result.txt Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. ...
15
votes
5answers
788 views

Is it possible to make zcat output text even if it's uncompressed? [duplicate]

The problem is I have some database dumps which are either compressed or in plain text. There is no difference in file extension etc. Using zcat on uncompressed files produces an error instead of the ...
15
votes
2answers
7k views

/proc/PID/fd/X link number

In Linux, in /proc/PID/fd/X, the links for file descriptors that are pipes or sockets have a number, like: l-wx------ 1 user user 64 Mar 24 00:05 1 -> pipe:[6839] l-wx------ 1 user user 64 Mar 24 ...
15
votes
4answers
3k views

How to understand pipes

When I just used pipe in bash, I didn't think more about this. But when I read some C code example using system call pipe() together with fork(), I wonder how to understand pipes, including both ...
15
votes
1answer
490 views

Make program first read from pipe, then from keyboard

Consider the interactive program interactive. I have to run this program fairly frequently, and each time I run it the first few commands are the same. Obviously, having to type those commands over ...
15
votes
2answers
493 views

Use of ^ as a shell metacharacter

I wrote a small script today which contained grep -q ^local0 /etc/syslog.conf During review, a coworker suggested that ^local0 be quoted because ^ means "pipe" in the Bourne shell. Surprised by ...
13
votes
7answers
4k views

How to pipe output from one process to another but only execute if the first has output?

How can I rewrite this command to only email if there is output from the mailq | grep? mailq | egrep 'rejected|refused' -A 5 -B 5 | mail -s 'dd' email@email Is this even possible on one line?
13
votes
2answers
1k views

How do I “cat and follow” a file?

A file is being sequentially downloaded by wget. If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster. How to ...
13
votes
3answers
5k views

Find -exec + vs find | xargs. Which one to choose?

I understand that the -exec can take a + option to mimic the behaviour of xargs. Is there any situation where you'd prefer one form over the other? I personally tend to prefer the first form, if ...
13
votes
4answers
281 views

pipe, { list; } only works with some programs

Need explanations from power users for such unpredictable behaviour: ps -eF | { head -n 1;grep worker; } UID PID PPID C SZ RSS PSR STIME TTY TIME CMD root 441 2 0 ...
13
votes
2answers
13k views

Bash: How to read one line at a time from output of a command?

I am trying to read the output of a command in bash using a while loop. while read -r line do echo "$line" done <<< $(find . -type f) The output I got ranveer@ranveer:~/tmp$ bash ...
12
votes
2answers
2k views

Where do my ANSI escape codes go when I pipe to another process? Can I keep them?

I sometime want to pipe the color-coded output fror a process, eg. grep... but when I pipe it to another process, eg. sed, the color codes are lost... Is the some way to keep thes codes intact ? ...
12
votes
3answers
429 views

How to display the number of lines output by a command in real time?

I'm using svn export as part of a packager script for my application, and it looks like this command, like many others, doesn't have any kind of progress bar. I have two choices at the moment: ...
11
votes
3answers
3k views

How can I pipe output to another process, but retain the error state of the first process? [duplicate]

Possible Duplicate: Get exit code of process that's piped to another I am using the following command line (in a makefile) to pipe the verbose error messages from my compiler through a ...
11
votes
3answers
5k views

Pseudo files for temporary data

I often want to feed relatively short string data (could be several lines though) to commandline programs which accept only input from files (e.g. wdiff) in a repeated fashion. Sure I can create one ...
11
votes
4answers
21k views

Difference between 2>&1 > output.log and 2>&1 | tee output.log

I wanted to know the difference between the following two commands 2>&1 > output.log and 2>&1 | tee output.log I saw one of my colleague use second option to redirect. I know ...
10
votes
2answers
976 views

How can I redirect `time` output and command output to the same pipe?

Suppose I have a binary called foo. If I want to redirect the output of foo to some other process bar, I could write ./foo | bar. On the other hand, if I wanted to time foo, and redirect the output ...
10
votes
3answers
6k views

Non-blocking buffered named pipe?

I'm looking for something I suspect doesn't exist: A non-blocking buffered named pipe (fifo) for use from the command line. Is there such a thing? Here's the use case: Suppose I have a process that's ...
10
votes
1answer
2k views

What are guarantees for concurrent writes into a named pipe?

For example, I created a named pipe like the following: mknod myPipe p And I read from it from some process (for example, some server). For example purposes, I used tail: tail -f myPipe If ...
10
votes
1answer
1k views

Readable comments on separate lines in a multi-line bash command with pipelines?

When creating shell scripts using pipelines, and using the backslash to continue lines, I want to insert comments on separate lines, in a robust, readable and portable fashion. For example, given ...
9
votes
4answers
682 views

Utility to buffer an unbounded amount of data in a pipeline?

Is there a utility that I can stick in a pipeline to decouple read and write speeds? $ producer | buf | consumer Basically, I want a utility buf that reads its input as fast as possible, storing it ...
9
votes
2answers
2k views

Piping for loop output prevents local variable modification

I'm trying to write a simple bash function that takes, as its arguments, a number of files and/or directories. It should: Fully qualify the filenames. Sort them. Remove duplicates. Print all that ...
9
votes
4answers
5k views

Are FIFO, pipe & Unix domain socket the same thing in Linux kernel?

I heard that FIFOs are named pipes. And they have exactly the same semantics. On the other hand, I think Unix domain socket is quite similar to pipe (although I've never made use of it). So I wonder ...
9
votes
1answer
507 views

communication between multiple processes

I have a bash script, which runs manager() function as a separate process for x-times. How is it possible to forward messages to all manager() processes from within the script? I've read about ...
9
votes
2answers
271 views

How a piped shell programs balance their output/input rates? [duplicate]

Possible Duplicate: Bash while loop and reading from pipe I come from web programming background, and find myself interested in one peculiarity of using a local shell. I understand that ...
9
votes
2answers
2k views

echo vs <<<, or Useless Use of echo in Bash Award?

By now the Useless Use of cat Award is very well known, and there's also a mention of a Useless Use of echo (not relevant for this question). I'm wondering if there should be a "Useless Use of echo in ...