Chains the standard streams of a series of commands

learn more… | top users | synonyms

159
votes
10answers
59k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice ...
109
votes
1answer
8k views

Bash: What does “>|” do?

I have just seen this written down; $ some-command >| /tmp/output.txt Vertical pipes are used in standard redirects "piping" the output of one command to another, is >| in fact completely ...
87
votes
13answers
35k views

Get exit status of process that's piped to another

I have two processes foo and bar, connected with a pipe: $ foo | bar bar always exits 0; I'm interested in the exit code of foo. Is there any way to get at it?
53
votes
4answers
49k views

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command ...
51
votes
4answers
6k views

In what order do piped commands run?

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about ...
36
votes
5answers
13k views

Can I pipe stdout on one server to stdin on another server?

stdout on one CentOS server needs to be piped to stdin on another CentOS server. Is this possible? Update ScottPack, MikeyB and jofel all have valid answers. I awarded the answer to Scott because, ...
36
votes
5answers
10k views

Program that passes STDIN to STDOUT with color codes stripped?

I have a command that produces output in color, and I would like to pipe it into a file with the color codes stripped out. Is there a command that works like cat except that it strips color codes? I ...
32
votes
4answers
17k views

What are the advantages of using named pipe over unnamed pipe?

I was reviewing a set of interview questions that are asked from a unix admin; I found a topic called "named pipe". I googled the topic; to some extent I have been able to understand it :- named ...
32
votes
4answers
26k views

What is the mknod command used for?

I just started using Ubuntu as my main OS and I wanted to learn about things I should not do, and learn by the bad things people have done in the past. I came across these email about horror stories ...
31
votes
2answers
21k views

How do I pass a list of files to grep

I am using find and getting a list of files I want to grep through. How do I pipe that list to grep?
30
votes
4answers
11k views

Can't pipe into diff?

I wanted to be clever and compare a remote file to a local file without downloading it. I can get the contents of the remote file by ssh user@remote-host "cat path/file.name" However, piping that ...
29
votes
3answers
2k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
28
votes
5answers
9k views

How to make bidirectional pipe between two programs?

Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second. But how to make bidirectional pipe, i.e. cross-bind stdin and ...
27
votes
4answers
2k views

Why does “tail -f … | tail” fail to produce any output?

Why does the following command not produce any output? $ tail -f /etc/passwd | tail After reading about buffering, I tried the following to no avail: $ tail -f /etc/passwd | stdbuf -oL tail Note ...
25
votes
6answers
2k views

Why can't I redirect a path name output from one command to “cd”?

I am trying to get cd to accept a directory name redirected to it from another command. Neither of these methods work: $ echo $HOME | cd $ echo $HOME | xargs cd This does work: $ cd $(echo $HOME) ...
25
votes
3answers
2k views

Multicolored Grep

I'm trying to get each grep command to highlight it's results in a different color. I can do it manually with a line like this: ls -l GREP_COLORS='mt=01;32' grep c | GREP_COLORS='mt=01;31' grep o | ...
24
votes
5answers
3k views

Pipe to multiple files in the shell

I have an application which will produce a large amount of data which I do not wish to store onto the disk. The application mostly outputs data which I do not wish to use, but a set of useful ...
24
votes
1answer
1k views

Why does echo >file use more real time than echo | sed >file?

The example, below, surprised me. It seems to be counter intuitive... aside from the fact that there is a whisker more user time for the echo | sed combo. Why is echo using so much sys time when it ...
24
votes
1answer
715 views

How do I redirect command completion output?

On the command line, I can redirect or pipe output of a command to a file or another command using the > or | operator after the command. I have come across a less standard situation that I would ...
22
votes
4answers
4k views

Process substitution and pipe

I was wondering how to understand the following: Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple ...
22
votes
4answers
1k views

Are Linux utilities smart when running piped commands?

I was just running a few commands in a terminal and I started wondering, does Unix/Linux take shortcuts when running piped commands? For example, let's say I have a file with one million lines, the ...
21
votes
2answers
10k views

/proc/PID/fd/X link number

In Linux, in /proc/PID/fd/X, the links for file descriptors that are pipes or sockets have a number, like: l-wx------ 1 user user 64 Mar 24 00:05 1 -> pipe:[6839] l-wx------ 1 user user 64 Mar 24 ...
20
votes
4answers
929 views

Is it possible to redirect the output of a command into more than one command?

As far as I know, I can use the tee command to split the standard output onto the screen and further files: command -option1 -option2 argument | tee file1 file2 file3 Is it possible to redirect ...
20
votes
2answers
1k views

Pipes, how do data flow in a pipeline?

I don't understand how the data flows in the pipeline and hope someone could clarify what is going on there. I thought a pipeline of commands processes files (text, arrays of strings) in line by line ...
19
votes
9answers
3k views

Grep lines but let the first line through [duplicate]

Consider the following output from df. Filesystem Size Used Avail Use% Mounted on /dev/sda1 23G 6.1G 16G 29% / udev 10M 0 10M 0% /dev tmpfs 397M 420K ...
19
votes
4answers
1k views

Is there a way to pipe the output of one program into two other programs?

Sorry if this is a silly question but I'm trying to accomplish something like this but on one line: $ prog1 | prog2 $ prog1 | prog3 So, I basically want to execute prog1 and pipe the output to ...
18
votes
3answers
598 views

Use of ^ as a shell metacharacter

I wrote a small script today which contained grep -q ^local0 /etc/syslog.conf During review, a coworker suggested that ^local0 be quoted because ^ means "pipe" in the Bourne shell. Surprised by ...
17
votes
6answers
2k views

Conditional pipeline

Say I've got the following pipeline: cmd1 < input.txt |\ cmd2 |\ cmd4 |\ cmd5 |\ cmd6 |\ (...) |\ cmdN > result.txt Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. ...
16
votes
3answers
9k views

Pseudo files for temporary data

I often want to feed relatively short string data (could be several lines though) to commandline programs which accept only input from files (e.g. wdiff) in a repeated fashion. Sure I can create one ...
16
votes
10answers
19k views

What's a good example of piping commands together?

If you were helping someone to learn the concept of pipes on the command line what example would you use? The example that actually came up was as follows: cat whatever.txt | less I feel like ...
16
votes
4answers
637 views

How can I implement a circular flow of data among interconnected commands?

I know of two types how commands can be connected with each other: by using a Pipe (putting std-output into std-input of the next command). by using a Tee (splice the output into many ...
16
votes
3answers
7k views

Find -exec + vs find | xargs. Which one to choose?

I understand that the -exec can take a + option to mimic the behaviour of xargs. Is there any situation where you'd prefer one form over the other? I personally tend to prefer the first form, if ...
16
votes
4answers
3k views

How to understand pipes

When I just used pipe in bash, I didn't think more about this. But when I read some C code example using system call pipe() together with fork(), I wonder how to understand pipes, including both ...
16
votes
2answers
22k views

Bash: How to read one line at a time from output of a command?

I am trying to read the output of a command in bash using a while loop. while read -r line do echo "$line" done <<< $(find . -type f) The output I got ranveer@ranveer:~/tmp$ bash ...
15
votes
7answers
5k views

How to pipe output from one process to another but only execute if the first has output?

How can I rewrite this command to only email if there is output from the mailq | grep? mailq | egrep 'rejected|refused' -A 5 -B 5 | mail -s 'dd' email@email Is this even possible on one line?
15
votes
5answers
2k views

Is it possible to make zcat output text even if it's uncompressed? [duplicate]

The problem is I have some database dumps which are either compressed or in plain text. There is no difference in file extension etc. Using zcat on uncompressed files produces an error instead of the ...
15
votes
2answers
2k views

How do I “cat and follow” a file?

A file is being sequentially downloaded by wget. If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster. How to ...
15
votes
1answer
612 views

Make program first read from pipe, then from keyboard

Consider the interactive program interactive. I have to run this program fairly frequently, and each time I run it the first few commands are the same. Obviously, having to type those commands over ...
15
votes
3answers
9k views

How to check if pipe is empty

I have piped line in bash script and want to check if pipe has data, before feeding program Searching I found about test -t 0 but it doesn't work here. Always returns false. So how to be sure that ...
14
votes
5answers
13k views

how do I trim each line in bash?

I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into?
14
votes
2answers
2k views

Where do my ANSI escape codes go when I pipe to another process? Can I keep them?

I sometime want to pipe the color-coded output fror a process, eg. grep... but when I pipe it to another process, eg. sed, the color codes are lost... Is the some way to keep thes codes intact ? ...
14
votes
5answers
6k views

Shell Script mktemp, what's the best method to create temporary named pipe?

I'm aware its best to create temporary files with mktemp, but what about named pipes? I have an idea, I'm sure it would work, but I would really like an expert opinion. I prefer things to be as ...
14
votes
9answers
654 views

Filter or pipe certain sections of a file

I have an input file with some sections the are demarcated with start and end tags, for example: line A line B @@inline-code-start line X line Y line Z @@inline-code-end line C line D I want to ...
14
votes
5answers
30k views

Difference between 2>&1 > output.log and 2>&1 | tee output.log

I wanted to know the difference between the following two commands 2>&1 > output.log and 2>&1 | tee output.log I saw one of my colleague use second option to redirect. I know ...
13
votes
4answers
292 views

pipe, { list; } only works with some programs

Need explanations from power users for such unpredictable behaviour: ps -eF | { head -n 1;grep worker; } UID PID PPID C SZ RSS PSR STIME TTY TIME CMD root 441 2 0 ...
13
votes
1answer
3k views

What are guarantees for concurrent writes into a named pipe?

For example, I created a named pipe like the following: mknod myPipe p And I read from it from some process (for example, some server). For example purposes, I used tail: tail -f myPipe If ...
12
votes
3answers
661 views

No colored output in less for the ls command

If I run ls I get colored output which I find pretty handy for quickly getting a glance of the kind of file. When I try to pipe it to less even with the -r and -R flags the coloring always get lost. I ...
12
votes
2answers
2k views

How can I redirect `time` output and command output to the same pipe?

Suppose I have a binary called foo. If I want to redirect the output of foo to some other process bar, I could write ./foo | bar. On the other hand, if I wanted to time foo, and redirect the output ...
12
votes
3answers
558 views

How to display the number of lines output by a command in real time?

I'm using svn export as part of a packager script for my application, and it looks like this command, like many others, doesn't have any kind of progress bar. I have two choices at the moment: ...
12
votes
3answers
7k views

Non-blocking buffered named pipe?

I'm looking for something I suspect doesn't exist: A non-blocking buffered named pipe (fifo) for use from the command line. Is there such a thing? Here's the use case: Suppose I have a process that's ...