Chains the standard streams of a series of commands

learn more… | top users | synonyms

50
votes
12answers
17k views

Get exit status of process that's piped to another

I have two processes foo and bar, connected with a pipe: $ foo | bar bar always exits 0; I'm interested in the exit code of foo. Is there any way to get at it?
101
votes
10answers
32k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice ...
6
votes
4answers
6k views

Why is my variable being localized in one 'while read' loop, but not in another seemingly similar loop [duplicate]

Why do I get different values for $x from the snippets below? #!/bin/bash x=1 echo fred>junk ; while read var ; do x=55 ; done <junk echo x=$x # x=55 .. I'd expect this result x=1 cat ...
7
votes
2answers
2k views

Piping for loop output prevents local variable modification

I am trying to write a simple bash function that takes, as its arguments, a number of files and/or directories. It should: Fully qualify the filenames. Sort them. Remove duplicates. Print all that ...
35
votes
3answers
30k views

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command ...
24
votes
6answers
2k views

Why can't I redirect a path name output from one command to “cd”?

I am trying to get cd to accept a directory name redirected to it from another command. Neither of these methods work: $ echo $HOME | cd $ echo $HOME | xargs cd This does work: $ cd $(echo $HOME) ...
7
votes
1answer
5k views

How to check if pipe is empty

I have piped line in bash script and want to check if pipe has data, before feeding program Searching I found about test -t 0 but it doesn't work here. Always returns false. So how to be sure that ...
11
votes
3answers
4k views

Find -exec + vs find | xargs. Which one to choose?

I understand that the -exec can take a + option to mimic the behaviour of xargs. Is there any situation where you'd prefer one form over the other? I personally tend to prefer the first form, if ...
23
votes
3answers
992 views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
5
votes
3answers
298 views

Correct textual name for <<

I've just come across the << command, used like so: cat > outfile.txt <<EOF Multi-line content that will be written to outfile.txt EOF Now, I've no idea what this is called, but I'd ...
6
votes
2answers
3k views

what is meant by connecting STDOUT and STDIN?

This is a newbie question. I'm reading a book, it says: Every process has at least three communication channels available to it: “standard input” (STDIN), “standard output” (STDOUT), and ...
5
votes
2answers
469 views

Is it possible to follow a command (run repeatedly)? as one would follow a file using tail -f?

I have a script which produces a file 'Detail.out'. I know that the script is completed whenever the file contains a certain number of lines (roughly 21025). So I find myself sitting at the command ...
1
vote
5answers
3k views

Pass the output of previous command to next as an argument

I've a command that outputs data to stdout (command1 -p=aaa -v=bbb -i=4). The output line can have the following value: rate (10%) - name: value - 10Kbps I want to grep that output in order to ...
5
votes
3answers
435 views

Why does sed act differently depending on the output file?

If I run: cat messages.txt | sed -e 's/a/a/g' > messages.txt on one large file (2500+ lines) I find that the resulting file will only have about 900 lines after the command in cygwin and will ...
28
votes
4answers
7k views

Can I pipe stdout on one server to stdin on another server?

stdout on one CentOS server needs to be piped to stdin on another CentOS server. Is this possible? Update ScottPack, MikeyB and jofel all have valid answers. I awarded the answer to Scott because, ...
36
votes
4answers
4k views

In what order do piped commands run?

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about ...
29
votes
5answers
7k views

Program that passes STDIN to STDOUT with color codes stripped?

I have a command that produces output in color, and I would like to pipe it into a file with the color codes stripped out. Is there a command that works like cat except that it strips color codes? I ...
19
votes
5answers
5k views

How to make bidirectional pipe between two programs?

Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second. But how to make bidirectional pipe, i.e. cross-bind stdin and ...
15
votes
1answer
427 views

Make program first read from pipe, then from keyboard

Consider the interactive program interactive. I have to run this program fairly frequently, and each time I run it the first few commands are the same. Obviously, having to type those commands over ...
24
votes
3answers
549 views

Multicolored Grep

I'm trying to get each grep command to highlight it's results in a different color. I can do it manually with a line like this: ls -l GREP_COLORS='mt=01;32' grep c | GREP_COLORS='mt=01;31' grep o | ...
13
votes
2answers
5k views

/proc/PID/fd/X link number

In Linux, in /proc/PID/fd/X, the links for file descriptors that are pipes or sockets have a number, like: l-wx------ 1 user user 64 Mar 24 00:05 1 -> pipe:[6839] l-wx------ 1 user user 64 Mar 24 ...
11
votes
6answers
3k views

How to pipe output from one process to another but only execute if the first has output?

How can I rewrite this command to only email if there is output from the mailq | grep? mailq | egrep 'rejected|refused' -A 5 -B 5 | mail -s 'dd' email@email Is this even possible on one line?
7
votes
4answers
748 views

What is the easiest way to execute text from tail at the command line?

Sometimes I'm working on a new (ubuntu) box and I type git and am alerted: The program 'git' is currently not installed. You can install it by typing: apt-get install git-core If that happens I ...
3
votes
1answer
71 views

How can the order of execution in the shell as it relates to redirection be upset about 1 time out of a 1000 on my system? [duplicate]

Considering this Q&A about the order of execution in the shell as it relates to redirection, and despite the fact that if a file doesn't exist it gets created first so that cat example.txt | shuf ...
3
votes
3answers
1k views

Bash while loop and reading from pipe

I have a Windows command-line program that I'm running in a Bash script in Ubuntu via wine. The Bash script basically looks like this: wine myprogram.exe | while read line do # Process line done ...
0
votes
4answers
260 views

Why does the command shuf file > file leave an empty file?

I know this is sort of a duplicate of another question (Why this sort command gives me an empty file?) but I wanted to expand on the question in response to the answers given. The command shuf ...
18
votes
4answers
2k views

Process substitution and pipe

I was wondering how to understand the following: Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple ...
7
votes
5answers
4k views

Shell Script mktemp, what's the best method to create temporary named pipe?

I'm aware its best to create temporary files with mktemp, but what about named pipes? I have an idea, I'm sure it would work, but I would really like an expert opinion. I prefer things to be as ...
7
votes
2answers
9k views

Bash: How to read one line at a time from output of a command?

I am trying to read the output of a command in bash using a while loop. while read -r line do echo "$line" done <<< $(find . -type f) The output I got ranveer@ranveer:~/tmp$ bash ...
15
votes
10answers
10k views

What's a good example of piping commands together?

If you were helping someone to learn the concept of pipes on the command line what example would you use? The example that actually came up was as follows: cat whatever.txt | less I feel like ...
7
votes
3answers
1k views

tee + cat: use an output several times and then concatenate results

If I call some command, for instance an echo I can use the results from that command in several other commands with tee. Example: echo "Hello world!" | tee >(command1) >(command2) ...
6
votes
2answers
3k views

Pseudo files for temporary data

I often want to feed relatively short string data (could be several lines though) to commandline programs which accept only input from files (e.g. wdiff) in a repeated fashion. Sure I can create one ...
15
votes
5answers
1k views

Conditional pipeline

Say I've got the following pipeline: cmd1 < input.txt |\ cmd2 |\ cmd4 |\ cmd5 |\ cmd6 |\ (...) |\ cmdN > result.txt Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. ...
15
votes
4answers
2k views

How to understand pipes

When I just used pipe in bash, I didn't think more about this. But when I read some C code example using system call pipe() together with fork(), I wonder how to understand pipes, including both ...
12
votes
2answers
2k views

Where do my ANSI escape codes go when I pipe to another process? Can I keep them?

I sometime want to pipe the color-coded output fror a process, eg. grep... but when I pipe it to another process, eg. sed, the color codes are lost... Is the some way to keep thes codes intact ? ...
11
votes
3answers
3k views

How can I pipe output to another process, but retain the error state of the first process? [duplicate]

Possible Duplicate: Get exit code of process that's piped to another I am using the following command line (in a makefile) to pipe the verbose error messages from my compiler through a ...
7
votes
3answers
724 views

Why do some commands not read from their standard input?

I wonder what when we should use pipeline and when we shouldn't. Say for instance, to kill certain process which handling pdf files, the following will not work by using pipeline: ps aux | grep pdf ...
3
votes
3answers
2k views

How do I use tee to redirect to grep

I don't have much experience of using tee, so I hope this is not very basic. After viewing one of the answers to this question I came across a strange beheviour with tee. In order for me to output ...
6
votes
1answer
453 views

I'm confused as to why “| true” in a makefile has the same effect as “|| true”

I have a makefile where I'm stopping a service before removing a file. When it couldn't stop the service, it would break on error. This is clearly unwanted so I thought I'd add || true but missed a |. ...
5
votes
5answers
861 views

resolve all ip addresses in command output using standard command line tools

I have several log files that contain a bunch of ip addresses. I would love to be able to pipe the data through a program that would match and resolve ip addresses. I.E. cat /var/log/somelogfile | ...
5
votes
2answers
1k views

Combine input from multiple files/pipes without clobbering lines or blocking?

Is there a tool that will take input from multiple files or pipes and write it to stdout, without blocking on reads, such that individual input lines come out intact? I basically want to multiplex a ...
4
votes
0answers
159 views

Repeatably generate a read error for testing? [duplicate]

I'm trying to test the error handling in some software, in particular what happens when an error occurs reading from a file or pipe. Is there a simple way to send a certain amount of data to stdout ...
3
votes
2answers
1k views

How to compose bash functions using pipes?

I have few functions defined in this fashion: function f { read and process $1 ... echo $result } I want to compose them together so that invocation would look like f | g | h. What idiom ...
21
votes
4answers
799 views

Are Linux utilities smart when running piped commands?

I was just running a few commands in a terminal and I started wondering, does Unix/Linux take shortcuts when running piped commands? For example, let's say I have a file with one million lines, the ...
8
votes
3answers
4k views

Using in/out named pipes for a TCP connection

I've been fiddling with getting this to work for a while now, so I suspect some sort of fundamental misunderstanding about how pipes work is the root cause of my troubles. My goal is to initiate a ...
8
votes
3answers
3k views

Have Bash script wait for status message before continuing

I'm firing up Selenium server with a bash script and as you can see from the timestamps on the log below, it takes about 32 seconds for the thing to fully come online: Feb 28, 2012 10:19:02 PM ...
8
votes
3answers
7k views

How to process/pipe TCPDUMPs output in realtime

If I want to tcpdump DNS requests by clients (on an OpenWrt 10.04 router), then I root@ROUTER:/etc# tcpdump -n -i br-lan dst port 53 2>&1 tcpdump: verbose output suppressed, use -v or ...
7
votes
4answers
4k views

Are FIFO, pipe & Unix domain socket the same thing in Linux kernel?

I heard that FIFOs are named pipes. And they have exactly the same semantics. On the other hand, I think Unix domain socket is quite similar to pipe (although I've never made use of it). So I wonder ...
5
votes
1answer
320 views

Loop through binary data chunks from stdin in Bash

I'm looking for something like while IFS= read -r -n $length str; do ... done but for binary data. Is it possible to do this using dd or other tools? Is there some technique to make these tools able ...
5
votes
3answers
2k views

Output to stdout and at the same time grep into a file

I have a script that outputs text to stdout. I want to see all this output in my terminal, and at the same time I want to filter some lines and save them in a file. Example: $ myscript Line A Line B ...