1

This question already has an answer here:

I have a script that reads a file that contains urls and then prints out the ip addresses. However, I am unable to find resources as to why the pipe blocks when I add the command

cut -d" " -f4 

at the end of the pipe. Here is the full command.

for url in $(cat urls); do host $url; done | grep "has address" | cut -d" " -f4

The output is printed asynchronously till the grep command. Any command that comes after the grep has its output blocked till all the urls have been processed by the host command. Can anyone give me insight as to why this happens?

marked as duplicate by Gilles shell-script Jun 1 '16 at 21:44

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
  • See also google.co.uk/…" – Stéphane Chazelas Jun 1 '16 at 11:27
  • @StéphaneChazelas I have realized that it might be a buffering issue. What I don't understand is why are some commands buffered while other are not? – MykelXIII Jun 1 '16 at 11:38
  • 1
    Every command does as it pleases, but generally, output is not-buffered or line buffered where it goes to a terminal, and block buffered (with buffers of varying size, generally at least 4KiB) when not (like for grep when its output goes to a pipe above). – Stéphane Chazelas Jun 1 '16 at 11:41
  • @StéphaneChazelas Thank you for your reply. I am still confused why the grep above buffers but I have used grep a lot of times and most of the time it does not buffer. I am trying to figure out the exact condition that caused the command above to buffer. Thank you. – MykelXIII Jun 1 '16 at 11:51
1

grep buffers output if it does not write to a terminal. You can use grep --line-buffered to output each line immediately. Consider:

run slow output command and grep to terminal:

$ for i in {1..3}; do
>   echo printing >&2
>   echo to grep
>   sleep 1
> done | grep to
printing
to grep
printing
to grep
printing
to grep

run slow output command and grep to cut:

$ for i in {1..3}; do
>   echo printing >&2
>   echo to grep
>   sleep 1
> done | grep to | cut -d" " -f1-2
printing
printing
printing
to grep
to grep
to grep

run slow output command and grep to cut using --line-buffered option:

$ for i in {1..3}; do
>   echo printing >&2
>   echo to grep
>   sleep 1
> done | grep --line-buffered to | cut -d" " -f1-2
printing
to grep
printing
to grep
printing
to grep

Not the answer you're looking for? Browse other questions tagged or ask your own question.