I have a pre process command to output a file

./preprocess.sh > preprocessed_file 

and the preprocessed_file will be used like this

while read line

    ./research.sh $line &

done < preprocessed_file 

rm -f preprocessed_file

Is there any way to direct the output to the while read line part instead of outputting to the preprocessed_file? I think there should be a better way other than using this temp preprocessed_file.


You can use bash process substitution:

while IFS= read -r line; do
  ./research.sh "$line" &
done < <(./preprocess.sh)

Some advantages of process substitution:

  • No need to save temporary files.
  • Better performance. Reading from another process often faster than writing to disk, then read back in.
  • Save time to computation since when it is performed simultaneously with parameter and variable expansion, command substitution, and arithmetic expansion
  • what does the double left arrows (<<) mean? – Marcus Thornton Aug 25 '14 at 10:18
  • @MarcusThornton: < is a redirection, while <(...) is process substitution syntax. You should read: gnu.org/software/bash/manual/html_node/… for more details. – cuonglm Aug 25 '14 at 10:19
  • Got it. <(...) is a part of syntax. – Marcus Thornton Aug 25 '14 at 10:23
  • 2
    It's not necessarily faster. Because when reading from a pipe read has to read one byte at a time, while it can optimise things with reading larger chunks and seek backward when reading from a regular file. Best is to avoid while read loops altogether in the first place when possible. Also note that you need IFS= read -r line to read the line into $line. And leaving $line unquoted (invoking the split+glob operator) here probably doesn't make sense. – Stéphane Chazelas Aug 25 '14 at 11:24
  • 1
    @mikeserv, commands often line-buffer (as oppose to full-buffer) their output when it goes to a terminal. Here I'm saying that the read shell builtin reads one character at a time when reading from a pipe (regardless of what's at the other end of the pipe which read has no way to know), which is one of the reasons while read loops are tremendously slow. – Stéphane Chazelas Aug 26 '14 at 6:19

Yes! You can use a process pipe |.

./preprocess.sh |
    while IFS= read -r line
        ./research.sh "$line" &

A process pipe passes the standard output (stdout) of one process to the standard input (stdin) of the next.

You can optionally put a newline character following a | and extend the command to the next line.

Note: a|b is equivalent to b < <(a), but without the magic files, and in a more readable order, especially when the pipeline gets longer.

a|b|c is equivalent to c < <(b < <(a))


a|b|c|d|e is e < < (d < <(c < <(b < <(a))))

  • 3
    Note: This solution with the pipe has the advantage to be more portable than process substitution (not supported by some POSIX shells like dash). Still concerning portability, the right hand side of a pipe may be executed in a subshell (this depends on the shell), so that any side effect (such as setting variables) may not affect the environment of the shell script. – vinc17 Aug 25 '14 at 11:26
  • It’s generally safer to put variable references like $line into double quotes (for example, in your script, ./research.sh "$line" &). – G-Man Aug 25 '14 at 16:26
  • 1
    @G-Man Possibly not in this context. If research.sh works with the command line argument array and $line is, e.g., "one two", with the intention that the first argument be "one" and the second argument "two", quoting $line will make that impossible -- instead the first argument will be "one two" and there won't be a second one... – goldilocks Aug 25 '14 at 16:58
  • 2
    "a|b is equivalent to b < <(a)" - close, but not quite. In the pipe version, both sides of the pipe are run in subshells, whereas in the process-substitution version, only the substituted process is run in a subshell, but a is run in the scope of the currently executing shell level. This has important implications for the scope of variables set within a – Digital Trauma Aug 25 '14 at 23:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.