1

I am wondering why

ls -1 | 
while read file; do 
     echo $file; tail -n 100 $file > >(sleep 1 && cat > $file)
done  

is faster than

ls -1 | 
while read file; do 
    echo $file; tail -n 100 $file | (sleep 1 && cat > $file)
done   

?

If there are 100 files in a directory then:

  • the second command takes almost 100 seconds to process
  • the first command is processed almost immediately.
  • 1
    That's because bash doesn't wait for the termination of that process substitution (which could be seen as a bug). See The process substitution output is out of the order for details. – Stéphane Chazelas Mar 29 '18 at 13:45
  • @StéphaneChazelas; This says different : unix.stackexchange.com/q/146263/186463 – haccks Mar 29 '18 at 14:10
  • @haccks, in that case grep < <(cmd), while the shell doesn't wait for cmd, grep does as it waits for eof on its stdin. Again, see details at the link I gave in my previous comment. – Stéphane Chazelas Mar 29 '18 at 14:22
  • 2
    All pipe components run in subshells in bash, but not in background unless the whole pipeline is put in background with & (note that background_/_foreground only apply to interactive shells, that's terminology linked to job control in terminals). process substitution are put in background unless they're run from a subshell itself run in foreground. In any case, that's not what I'm talking about. I'm talking about the shell waiting for the termination of processes it starts before continuing with the next command (tbc) – Stéphane Chazelas Mar 29 '18 at 15:37
  • 1
    So in both cases the components run asynchronously, but in the case of pipelines it still waits for them all to finish (unless you run the whole pipeline in the background with &). – Barmar Mar 30 '18 at 0:29
2

When you use a pipeline, the shell runs each command in the pipeline concurrently, and waits for all of them to finish before going to the next command. This is explained in the documentation:

If the pipeline is not executed asynchronously (see Lists), the shell waits for all commands in the pipeline to complete.

When the above refers to executing the pipeline asynchronously, it's talking about running the whole pipeline in the background with &.

When you use process substitution the shell doesn't wait for it to complete. The documentation simply says:

The process list is run asynchronously, and its input or output appears as a filename.

| improve this answer | |
  • I think I was confused with the term "asynchronous" and "concurrent". In case of pipe, elements run concurrently. But here I think "asynchronous" is specifically mean running process in the background, correct me if I am wrong? – haccks Mar 31 '18 at 12:06
  • That seems to be how the manual is using the term. – Barmar Mar 31 '18 at 14:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.