Questions tagged [pipe]

A Unix pipe connects file descriptors of two processes. A pipe is created with the POSIX pipe() function declared in <unistd.h>. Shells provide pipe creation between processes using "|".

Filter by
Sorted by
Tagged with
453
votes
14answers
248k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints progress but I'm unhappy with it. I'm using print_progress to make it nicer (namely, I ...
369
votes
16answers
182k views

Get exit status of process that's piped to another

I have two processes foo and bar, connected with a pipe: $ foo | bar bar always exits 0; I'm interested in the exit code of foo. Is there any way to get at it?
238
votes
15answers
476k views

How do I trim leading and trailing whitespace from each line of some output?

I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into? Example file: test space at back test ...
161
votes
7answers
442k views

Pass the output of previous command to next as an argument

I've a command that outputs data to stdout (command1 -p=aaa -v=bbb -i=4). The output line can have the following value: rate (10%) - name: value - 10Kbps I want to grep that output in order to store ...
161
votes
1answer
23k views

Bash: What does “>|” do?

I have just seen this written down; $ some-command >| /tmp/output.txt Vertical pipes are used in standard redirects "piping" the output of one command to another, is >| in fact completely ...
158
votes
6answers
167k views

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command produced ...
120
votes
3answers
72k views

Pseudo files for temporary data

I often want to feed relatively short string data (could be several lines though) to commandline programs which accept only input from files (e.g. wdiff) in a repeated fashion. Sure I can create one ...
114
votes
4answers
131k views

How do I pass a list of files to grep

I am using find and getting a list of files I want to grep through. How do I pipe that list to grep?
106
votes
4answers
15k views

What is the difference between “cat file | ./binary” and “./binary < file”?

I have a binary (that I can't modify) and I can do: ./binary < file I also can do: ./binary << EOF > "line 1 of file" > "line 2 of file" ... > "last line of file" > EOF But ...
105
votes
4answers
25k views

In what order do piped commands run?

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about ...
103
votes
5answers
36k views

Process substitution and pipe

I was wondering how to understand the following: Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple commands? ...
93
votes
5answers
43k views

How to use watch command with a piped chain of commands/programs

I usually use watch Linux utility to watch the output of a command repeatedly every n seconds, like in watch df -h /some_volume/. But I seem not to be able to use watch with a piped series of command ...
87
votes
4answers
53k views

Can't pipe into diff?

I wanted to be clever and compare a remote file to a local file without first manually downloading it. I can get the contents of the remote file by ssh user@remote-host "cat path/file.name" However,...
84
votes
6answers
63k views

Can I pipe stdout on one server to stdin on another server?

stdout on one CentOS server needs to be piped to stdin on another CentOS server. Is this possible? Update ScottPack, MikeyB and jofel all have valid answers. I awarded the answer to Scott because, ...
83
votes
3answers
83k views

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz ...
69
votes
8answers
43k views

How to make bidirectional pipe between two programs?

Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second. But how to make bidirectional pipe, i.e. cross-bind stdin and ...
68
votes
12answers
73k views

How to check if a pipe is empty and run a command on the data if it isn't?

I have piped a line in bash script and want to check if the pipe has data, before feeding it to a program. Searching I found about test -t 0 but it doesn't work here. Always returns false. So how to ...
65
votes
7answers
23k views

Program that passes STDIN to STDOUT with color codes stripped?

I have a command that produces output in color, and I would like to pipe it into a file with the color codes stripped out. Is there a command that works like cat except that it strips color codes? I ...
57
votes
3answers
131k views

Bash: How to read one line at a time from output of a command?

I am trying to read the output of a command in bash using a while loop. while read -r line do echo "$line" done <<< $(find . -type f) The output I got ranveer@ranveer:~/tmp$ bash test....
57
votes
4answers
50k views

What are the advantages of using named pipe over unnamed pipe?

I was reviewing a set of interview questions that are asked from a unix admin; I found a topic called "named pipe". I googled the topic; to some extent I have been able to understand it :- named ...
55
votes
4answers
7k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
54
votes
5answers
78k views

What is the mknod command used for?

I just started using Ubuntu as my main OS and I wanted to learn about things I should not do, and learn by the bad things people have done in the past. I came across these email about horror stories ...
51
votes
7answers
11k views

How to trick a command into thinking its output is going to a terminal

Given a command that changes its behaviour when its output is going to a terminal (e.g. produce coloured output), how can that output be redirected in a pipeline while preserving the changed behaviour?...
50
votes
3answers
87k views

Piping commands after a piped xargs

HP-UX ***** B.11.23 U ia64 **** unlimited-user license find . -type d -name *log* | xargs ls -la gives me the directory names (the ones which contain log in the directory name) followed by all files ...
47
votes
7answers
20k views

Conditional pipeline

Say I've got the following pipeline: cmd1 < input.txt |\ cmd2 |\ cmd4 |\ cmd5 |\ cmd6 |\ (...) |\ cmdN > result.txt Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. ...
47
votes
2answers
30k views

Why does process substitution result in a file called /dev/fd/63 which is a pipe?

I am trying to understand named pipes in the context of this particular example. I type <(ls -l) in my terminal and get the output as, bash: /dev/fd/63: Permission denied. If I type cat <(ls -...
44
votes
9answers
126k views

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
43
votes
7answers
164k views

Difference between 2>&1 > output.log and 2>&1 | tee output.log

I wanted to know the difference between the following two commands 2>&1 > output.log and 2>&1 | tee output.log I saw one of my colleague use second option to redirect. I know ...
42
votes
3answers
23k views

How can I do the equivalent of tail -f with ls? [duplicate]

Possible Duplicate: Is it possible to follow a command (run repeatedly)? as one would follow a file using tail -f? I would like to monitor files that are being downloaded to a directory in real ...
40
votes
2answers
5k views

What is the portable (POSIX) way to achieve process substitution?

Some shells, like bash, support Process Substitution which is a way to present process output as a file, like this: $ diff <(sort file1) <(sort file2) However, this construct isn't POSIX and, ...
39
votes
6answers
108k views

How do I pipe ls to grep, and delete the files filtered by grep?

I wanted to delete some package in my home file, but the filename was too long (google-chrome-stable_current_i386.deb). So, I decided to use the command ls|grep chrome|rm to pipe the files to grep to ...
39
votes
4answers
12k views

How to invoke vim editor and pipe output to bash

Sometimes I need to write text and then pipe that text into another command. My usual workflow goes something like this: vim # I edit and save my file as file.txt cat file.txt | pandoc -o file.pdf # ...
39
votes
3answers
7k views

How do pipelines limit memory usage?

Brian Kernighan explains in this video the early Bell Labs attraction to small languages/programs being based on memory limitations A big machine would be 64 k-bytes--K, not M or G--and so that ...
39
votes
5answers
32k views

Are FIFO, pipe & Unix domain socket the same thing in Linux kernel?

I heard that FIFOs are named pipes. And they have exactly the same semantics. On the other hand, I think Unix domain socket is quite similar to pipe (although I've never made use of it). So I wonder ...
39
votes
2answers
8k views

Where do my ANSI escape codes go when I pipe to another process? Can I keep them?

I sometime want to pipe the color-coded output fror a process, eg. grep... but when I pipe it to another process, eg. sed, the color codes are lost... Is the some way to keep thes codes intact ? ...
38
votes
3answers
36k views

Can I chain pgrep with kill?

I have noticed that | is used to send results of first command to the another. I would like to kill all processes that match a name. This is what pgrep normally does: $ pgrep name 5089 5105 And ...
38
votes
2answers
43k views

/proc/PID/fd/X link number

In Linux, in /proc/PID/fd/X, the links for file descriptors that are pipes or sockets have a number, like: l-wx------ 1 user user 64 Mar 24 00:05 1 -> pipe:[6839] l-wx------ 1 user user 64 Mar 24 ...
37
votes
4answers
11k views

Why does “tail -f … | tail” fail to produce any output?

Why does the following command not produce any output? $ tail -f /etc/passwd | tail After reading about buffering, I tried the following to no avail: $ tail -f /etc/passwd | stdbuf -oL tail Note ...
37
votes
5answers
18k views

Measure pipe throughput in the shell

There is a shell command that allows you to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you ...
37
votes
2answers
25k views

Piping data to a process's stdin without causing EOF afterward

I have an executable that starts a user-interactive shell. I would like to, upon launch of the shell, inject a few commands first, then allow the user to have their interactive session. I can do this ...
36
votes
4answers
16k views

find -exec + vs find | xargs: which one to choose?

I understand that the -exec can take a + option to mimic the behaviour of xargs. Is there any situation where you'd prefer one form over the other? I personally tend to prefer the first form, if ...
35
votes
5answers
20k views

Shell Script mktemp, what's the best method to create temporary named pipe?

I'm aware its best to create temporary files with mktemp, but what about named pipes? I prefer things to be as POSIX compliant as possible, but Linux only is acceptable. Avoiding Bashisms is my only ...
34
votes
14answers
68k views

What's a good example of piping commands together?

If you were helping someone to learn the concept of pipes on the command line what example would you use? The example that actually came up was as follows: cat whatever.txt | less I feel like that's ...
33
votes
2answers
11k views

Feed source command with a pipe

Previously I used source command like this: source file_name But what I'm trying to do is this: echo something | source Which doesn't work.
33
votes
5answers
91k views

Bash: Assign output of pipe to a variable

I am trying to get the output of a pipe into a variable. I tried the following things: echo foo | myvar=$(</dev/stdin) echo foo | myvar=$(cat) echo foo | myvar=$(tee) But $myvar is empty. I don’...
33
votes
3answers
16k views

How do I “cat and follow” a file?

A file is being sequentially downloaded by wget. If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster. How to ...
33
votes
1answer
16k views

What are guarantees for concurrent writes into a named pipe?

For example, I created a named pipe like the following: mknod myPipe p And I read from it from some process (for example, some server). For example purposes, I used tail: tail -f myPipe If several ...
32
votes
3answers
106k views

What makes a Unix process die with Broken pipe?

Here are some options I thought of, not sure which is the right one. There was an I/O error reading from the pipe. The process writing to the other end of the pipe died with a failure. All processes ...
32
votes
4answers
15k views

How can I time a pipe?

I want to time a command which consists of two separate commands with one piping output to another. For example, consider the two scripts below: $ cat foo.sh #!/bin/sh sleep 4 $ cat bar.sh #!/bin/sh ...
31
votes
6answers
9k views

Pipe to multiple files in the shell

I have an application which will produce a large amount of data which I do not wish to store onto the disk. The application mostly outputs data which I do not wish to use, but a set of useful ...

1
2 3 4 5
32