Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

By now the Useless Use of cat Award is very well known, and there's also a mention of a Useless Use of echo (not relevant for this question). I'm wondering if there should be a "Useless Use of echo in Bash Award": Piping seems to be much slower than heredocs and herestrings according to some highly unscientific measurements:

  • Heredocs:

    for reps in 1 2 3
    do
        time for i in {1..1000}
        do
            cat <<'END'
    test string
    END
        done > /dev/null
    done
    
    real    0m1.786s
    user    0m0.212s
    sys     0m0.332s
    
    real    0m1.817s
    user    0m0.232s
    sys     0m0.332s
    
    real    0m1.846s
    user    0m0.256s
    sys     0m0.320s
    
  • Herestrings

    for reps in 1 2 3
    do
        time for i in {1..1000}
        do
            cat <<< 'test string'
        done > /dev/null
    done
    
    real    0m1.932s
    user    0m0.280s
    sys     0m0.288s
    
    real    0m1.956s
    user    0m0.248s
    sys     0m0.348s
    
    real    0m1.968s
    user    0m0.268s
    sys     0m0.324s
    
  • Redirection

    for reps in 1 2 3
    do
        time for i in {1..1000}
        do
            echo 'test string' | cat
        done > /dev/null
    done
    
    real    0m3.562s
    user    0m0.416s
    sys     0m0.548s
    
    real    0m3.924s
    user    0m0.384s
    sys     0m0.604s
    
    real    0m3.343s
    user    0m0.400s
    sys     0m0.552s
    

In general, heredocs and herestrings are about the same speed (this is just one data set from several tests) while redirection is consistently more than 50% slower. Am I misunderstanding something, or could this be used as a general rule for commands reading standard input in Bash?

share|improve this question
3  
I'll certainly allow you a "useless use of seq" award ;-) –  Chris Down Dec 20 '12 at 13:42
2  
Note that you should either compare cat <<END vs. cat <<< "test string" vs. echo "test string" | cat or cat <<'END' vs. cat <<< 'test string' vs. echo 'test string' | cat to have the same amount of expansions performed by the shell on the strings. –  manatwork Dec 20 '12 at 13:47
    
@ChrisDown Derp. I always forget that one. Thanks! –  l0b0 Dec 20 '12 at 13:48
    
@manatwork Thanks, fixed and updated timings. –  l0b0 Dec 20 '12 at 13:51
    
Just a curiosity, what is with that reps? Wasn't your original intention to loop $reps times with $(seq $reps)? –  manatwork Dec 20 '12 at 13:55
show 10 more comments

2 Answers

up vote 5 down vote accepted

First, let's concentrate on performance. I ran benchmarks for a slightly different program on an otherwise mostly idle x86_64 processor running Debian squeeze.

herestring.bash, using a herestring to pass a line of input:

#! /bin/bash
i=0
while [ $i -lt $1 ]; do
  tr a-z A-Z <<<'hello world'
  i=$((i+1))
done >/dev/null

heredoc.bash, using a heredoc to pass a line of input:

#! /bin/bash
i=0
while [ $i -lt $1 ]; do
  tr a-z A-Z <<'EOF'
hello world
EOF
  i=$((i+1))
done >/dev/null

echo.bash, using echo and a pipe to pass a line of input:

#! /bin/bash
i=0
while [ $i -lt $1 ]; do
  echo 'hello world' | tr a-z A-Z
  i=$((i+1))
done >/dev/null

For comparison, I also timed the scripts under ATT ksh93 and under dash (except for herestring.bash, because dash doesn't have herestrings).

Here are median-of-three times:

$ time bash ./herestring.bash 10000
./herestring.bash 10000  0.32s user 0.79s system 15% cpu 7.088 total
$ time ksh ./herestring.bash 10000
ksh ./herestring.bash 10000  0.54s user 0.41s system 17% cpu 5.277 total
$ time bash ./heredoc.bash 10000
./heredoc.bash 10000  0.35s user 0.75s system 17% cpu 6.406 total
$ time ksh ./heredoc.bash 10000  
ksh ./heredoc.sh 10000  0.54s user 0.44s system 19% cpu 4.925 total
$ time sh ./heredoc.bash 10000  
./heredoc.sh 10000  0.08s user 0.58s system 12% cpu 5.313 total
$ time bash ./echo.bash 10000
./echo.bash 10000  0.36s user 1.40s system 20% cpu 8.641 total
$ time ksh ./echo.bash 10000
ksh ./echo.sh 10000  0.47s user 1.51s system 28% cpu 6.918 total
$ time sh ./echo.sh 10000
./echo.sh 10000  0.07s user 1.00s system 16% cpu 6.463 total

Conclusions:

  • A heredoc is faster than a herestring.
  • echo and a pipe is noticeably, but not dramatically faster. (Keep in mind that this is a toy program: in a real program, most of the processing time would be in whatever the tr call stands for here.)
  • If you want speed, ditch bash and call dash or even better ksh instead. Bash's features don't make up for its relative slowness, but ksh has both features and speed.

Beyond performance, there's also clarity and portability. <<< is a ksh93/bash/zsh extension which is less well-known than echo … | or <<. It doesn't work in ksh88/pdksh or in POSIX sh.

The only place where <<< is arguably significantly clearer is inside a heredoc:

foo=$(tr a-z A-Z <<<'hello world')

vs

foo=$(tr a-z A-Z <<'EOF'
hello world
EOF
)

(Most shells can't cope with closing the parenthesis at the end of the line containing <<EOF.)

share|improve this answer
2  
Note that <<< comes from rc, and was first brought to the Bourne-like shell world by zsh. rc didn't add a trailing newline, but all of bash, zsh and ksh93 do AFAICT. rc uses a pipe there, while bash, zsh and ksh93 use a temporary file like for heredocs. –  Stephane Chazelas Dec 21 '12 at 2:14
    
@StephaneChazelas Oops, I should have checked, thanks. This makes <<< even less useful. –  Gilles Dec 21 '12 at 2:35
    
Also note that zsh has an optimisation in =(<<<foo) to effectively create a temp file which contains "foo" which doesn't involve forking a process as =(echo foo) would. –  Stephane Chazelas Dec 21 '12 at 2:40
    
I'll also mention that pdksh does not include <<<. –  kurtm Oct 15 '13 at 14:54
    
@kurtm “It doesn't work in ksh88/pdksh or in POSIX sh.” –  Gilles Oct 15 '13 at 15:19
show 1 more comment

Another reason to use heredocs (if you didn't have enough) is that echo can fail if the stream isn't consumed. Consider having bash' pipefail option:

set -o pipefail
foo=yawn
echo $foo | /bin/true ; echo $?  # returns 0

/bin/true doesn't consume its standard input, but echo yawn completes nonetheless. However, if echo is asked to print a lot of data, it will not complete until after true has completed:

foo=$(cat /etc/passwd)
# foo now has a fair amount of data

echo $foo | /bin/true ; echo $?  # returns 0 sometimes 141
echo $foo$foo$foo$foo | /bin/true ; echo $?  # returns mostly 141

141 is SIGPIPE (128 + 13) (128 being added because bash does so according to bash(1):

When a command terminates on a fatal signal N, bash uses the value of 128+N as the exit status.

Heredocs don't have this problem:

/bin/true <<< $foo$foo$foo$foo ; echo $?  # returns 0 always
share|improve this answer
1  
+1 Very interesting results. –  l0b0 Oct 15 '13 at 12:17
    
Thanks, this particular nuance was causing my script to fail intermittently at random places, more so on Amazon servers than on kvm hosts, but it still failed from time to time, in seemingly impossible-to-fail places. It's a good thing that the pipefail defaults to being off! –  mogsie Oct 15 '13 at 13:29
2  
Oh, I enable pipefail at the top of every script. Gimme all the errors! –  l0b0 Oct 15 '13 at 15:48
    
@l0b0 Hmmm. Maybe I'll add pipefail to j.mp/safebash I'm all for getting all the errors during development! –  Richard Bronosky Jan 22 at 2:15
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.