2

I am wondering why

ls -1 | 
while read file; do 
     echo $file; tail -n 100 $file > >(sleep 1 && cat > $file)
done  

is faster than

ls -1 | 
while read file; do 
    echo $file; tail -n 100 $file | (sleep 1 && cat > $file)
done   

?

If there are 100 files in a directory then:

  • the second command takes almost 100 seconds to process
  • the first command is processed almost immediately.
15
  • 1
    That's because bash doesn't wait for the termination of that process substitution (which could be seen as a bug). See The process substitution output is out of the order for details. Commented Mar 29, 2018 at 13:45
  • @StéphaneChazelas; This says different : unix.stackexchange.com/q/146263/186463
    – haccks
    Commented Mar 29, 2018 at 14:10
  • @haccks, in that case grep < <(cmd), while the shell doesn't wait for cmd, grep does as it waits for eof on its stdin. Again, see details at the link I gave in my previous comment. Commented Mar 29, 2018 at 14:22
  • 2
    All pipe components run in subshells in bash, but not in background unless the whole pipeline is put in background with & (note that background_/_foreground only apply to interactive shells, that's terminology linked to job control in terminals). process substitution are put in background unless they're run from a subshell itself run in foreground. In any case, that's not what I'm talking about. I'm talking about the shell waiting for the termination of processes it starts before continuing with the next command (tbc) Commented Mar 29, 2018 at 15:37
  • 1
    So in both cases the components run asynchronously, but in the case of pipelines it still waits for them all to finish (unless you run the whole pipeline in the background with &).
    – Barmar
    Commented Mar 30, 2018 at 0:29

1 Answer 1

2

When you use a pipeline, the shell runs each command in the pipeline concurrently, and waits for all of them to finish before going to the next command. This is explained in the documentation:

If the pipeline is not executed asynchronously (see Lists), the shell waits for all commands in the pipeline to complete.

When the above refers to executing the pipeline asynchronously, it's talking about running the whole pipeline in the background with &.

When you use process substitution the shell doesn't wait for it to complete. The documentation simply says:

The process list is run asynchronously, and its input or output appears as a filename.

2
  • I think I was confused with the term "asynchronous" and "concurrent". In case of pipe, elements run concurrently. But here I think "asynchronous" is specifically mean running process in the background, correct me if I am wrong?
    – haccks
    Commented Mar 31, 2018 at 12:06
  • That seems to be how the manual is using the term.
    – Barmar
    Commented Mar 31, 2018 at 14:37

Not the answer you're looking for? Browse other questions tagged .