Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a pre process command to output a file

./preprocess.sh > preprocessed_file 

and the preprocessed_file will be used like this

while read line
do

    ./research.sh $line &

done < preprocessed_file 

rm -f preprocessed_file

Is there any way to direct the output to the while read line part instead of outputting to the preprocessed_file? I think there should be a better way other than using this temp preprocessed_file.

share|improve this question

2 Answers 2

up vote 6 down vote accepted

You can use bash process substitution:

while IFS= read -r line; do
  ./research.sh "$line" &
done < <(./preprocess.sh)

Some advantages of process substitution:

  • No need to save temporary files.
  • Better performance. Reading from another process often faster than writing to disk, then read back in.
  • Save time to computation since when it is performed simultaneously with parameter and variable expansion, command substitution, and arithmetic expansion
share|improve this answer
    
what does the double left arrows (<<) mean? –  Marcus Thornton Aug 25 at 10:18
    
@MarcusThornton: < is a redirection, while <(...) is process substitution syntax. You should read: gnu.org/software/bash/manual/html_node/… for more details. –  cuonglm Aug 25 at 10:19
    
Got it. <(...) is a part of syntax. –  Marcus Thornton Aug 25 at 10:23
2  
It's not necessarily faster. Because when reading from a pipe read has to read one byte at a time, while it can optimise things with reading larger chunks and seek backward when reading from a regular file. Best is to avoid while read loops altogether in the first place when possible. Also note that you need IFS= read -r line to read the line into $line. And leaving $line unquoted (invoking the split+glob operator) here probably doesn't make sense. –  Stéphane Chazelas Aug 25 at 11:24
1  
@mikeserv, commands often line-buffer (as oppose to full-buffer) their output when it goes to a terminal. Here I'm saying that the read shell builtin reads one character at a time when reading from a pipe (regardless of what's at the other end of the pipe which read has no way to know), which is one of the reasons while read loops are tremendously slow. –  Stéphane Chazelas Aug 26 at 6:19

Yes! You can use a process pipe |.

./preprocess.sh |
    while IFS= read -r line
    do
        ./research.sh "$line" &
    done

A process pipe passes the standard output (stdout) of one process to the standard input (stdin) of the next.

You can optionally put a newline character following a | and extend the command to the next line.

Note: a|b is equivalent to b < <(a), but without the magic files, and in a more readable order, especially when the pipeline gets longer.

a|b|c is equivalent to c < <(b < <(a))

and

a|b|c|d|e is e < < (d < <(c < <(b < <(a))))

share|improve this answer
3  
Note: This solution with the pipe has the advantage to be more portable than process substitution (not supported by some POSIX shells like dash). Still concerning portability, the right hand side of a pipe may be executed in a subshell (this depends on the shell), so that any side effect (such as setting variables) may not affect the environment of the shell script. –  vinc17 Aug 25 at 11:26
    
It’s generally safer to put variable references like $line into double quotes (for example, in your script, ./research.sh "$line" &). –  G-Man Aug 25 at 16:26
1  
@G-Man Possibly not in this context. If research.sh works with the command line argument array and $line is, e.g., "one two", with the intention that the first argument be "one" and the second argument "two", quoting $line will make that impossible -- instead the first argument will be "one two" and there won't be a second one... –  goldilocks Aug 25 at 16:58
    
"a|b is equivalent to b < <(a)" - close, but not quite. In the pipe version, both sides of the pipe are run in subshells, whereas in the process-substitution version, only the substituted process is run in a subshell, but a is run in the scope of the currently executing shell level. This has important implications for the scope of variables set within a –  DigitalTrauma Aug 25 at 23:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.