7

Is it equivalent to have commands print to a file directly, as opposed to writing to a file descriptor?

Illustration

Writing to file directly:

for i in {1..1000}; do >>x echo "$i"; done

Using an fd:

exec 3>&1 1>x
for i in {1..1000}; do echo "$i"; done
exec 1>&3 3>&-

Is the latter one more efficient?

  • 5
    both write(2) to a file descriptor, how do you think one is more "direct" ? – thrig May 18 '18 at 14:32
  • 3
    What about for i in {1..1000}; do echo "$i"; done > x? That only opens the file once. – RonJohn May 18 '18 at 22:05
  • @RonJohn It's already mentioned in Matija Nalis's answer. – Tomasz May 18 '18 at 22:10
12

The main difference between opening the file before the loop with exec, and putting the redirection in the command in the loop is that the former requires setting up the file descriptor just once, while the latter opens and closes the file for each iteration of the loop.

Doing it once is likely to be more efficient, but if you were to run an external command inside the loop, the difference would probably disappear in the cost of launching the command. (echo here is probably builtin, so that doesn't apply)

If the output is going to be sent to something other than a regular file (e.g. if x is a named pipe), the act of opening and closing the file may be visible to other processes, so there may be differences in behaviour, too.


Note that there's really no difference between a redirection through exec and a redirection on the command, they both open the file and juggle file descriptor numbers.

These two should be pretty much equivalent, in that they both open() the file and write() to it. (There's differences in how fd 1 is stored for the duration of the command, though.):

for i in {1..1000}; do 
    >>x echo "$i"
done


for i in {1..1000}; do
    exec 3>&1 1>>x         # assuming fd 3 is available
    echo "$i"              # here, fd 3 is visible to the command
    exec 1>&3 3>&-
done

  • 6
    Of course, you could also redirect the whole loop into x: for i in {1..1000}; do echo $i; done >x. I'd say this is the most readable version. – Beat Bolli May 18 '18 at 18:33
  • I've measured the difference between both used in every iteration for 1m iterations, and the difference is rather substantial (70-90%). See my answer: unix.stackexchange.com/a/444729/181255 The ratio is even higher with just 1k iterations (ca. 300%). – Tomasz May 19 '18 at 0:03
7

Yes, it is more efficient

Easiest way to test is to increase count to say 500000 and time it:

> time bash s1.sh; time bash s2.sh
bash s1.sh  16,47s user 10,00s system 99% cpu 26,537 total
bash s2.sh  10,51s user 3,50s system 99% cpu 14,008 total

strace(1) reveals why (we have one simple write, instead of open+5*fcntl+2*dup+2*close+write):

for for i in {1..1000}; do >>x echo "$i"; done we get:

open("x", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
fcntl(1, F_GETFD)                       = 0
fcntl(1, F_DUPFD, 10)                   = 10
fcntl(1, F_GETFD)                       = 0
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
dup2(3, 1)                              = 1
close(3)                                = 0
write(1, "997\n", 4)                    = 4
dup2(10, 1)                             = 1
fcntl(10, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0
open("x", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
fcntl(1, F_GETFD)                       = 0
fcntl(1, F_DUPFD, 10)                   = 10
fcntl(1, F_GETFD)                       = 0
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
dup2(3, 1)                              = 1
close(3)                                = 0
write(1, "998\n", 4)                    = 4
dup2(10, 1)                             = 1
fcntl(10, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0
open("x", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
fcntl(1, F_GETFD)                       = 0
fcntl(1, F_DUPFD, 10)                   = 10
fcntl(1, F_GETFD)                       = 0
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
dup2(3, 1)                              = 1
close(3)                                = 0
write(1, "999\n", 4)                    = 4
dup2(10, 1)                             = 1
fcntl(10, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0
open("x", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
fcntl(1, F_GETFD)                       = 0
fcntl(1, F_DUPFD, 10)                   = 10
fcntl(1, F_GETFD)                       = 0
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
dup2(3, 1)                              = 1
close(3)                                = 0
write(1, "1000\n", 5)                   = 5
dup2(10, 1)                             = 1
fcntl(10, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0

while for exec 3>&1 1>x we get much cleaner

write(1, "995\n", 4)                    = 4
write(1, "996\n", 4)                    = 4
write(1, "997\n", 4)                    = 4
write(1, "998\n", 4)                    = 4
write(1, "999\n", 4)                    = 4
write(1, "1000\n", 5)                   = 5

But note that the difference is not due to "using a FD", but because of place where you do redirection. For example, if you were to do for i in {1..1000}; do echo "$i"; done > x you would get pretty much the same performance as your second example:

bash s3.sh  10,35s user 3,70s system 100% cpu 14,042 total
0

To sum things up and add a new bit of information in this thread, here's a comparison of four ways to do it, ordered by efficiency. I estimate the efficiency by time measurement (user + sys) for 1 million iterations, based on two test series.

  1. These two are about the same:
    • Simple > loop redirection (time: 100%)
    • Using exec once for the whole loop (time: ~100%)
  2. Using >> for each iteration (time: 200% - 250%)
  3. Using exec for each iteration (time: 340% - 480%)

The conclusion's this:

There's a small difference between using exec vs. simple redirections like >>. (Simple is cheaper). It doesn't show on the single command execution level, but with a high number of repetitions, the difference becomes visible. Though the execution weight of the command redirected to shadows the differences, as noticed by ikkachu in the other answer.

  • The relative costs depend on the OS and hardware, and the cost of what's in the loop. (e.g. on x86, Spectre + Meltdown mitigation significantly increased the cost of system calls, making redirection inside the loop worse). If you literally want to generate a sequence of integers, seq 1000, or maybe printf "%s\n" {1..1000} is going to be faster than a bash loop. (seq 1000000 > /dev/null runs in ~0.03s, time printf "%s\n" {1..1000000} > /dev/null in ~0.7s, time for i in {1..1000000}; do echo "$i"; done > /dev/null in 2.1s, on a 3.9GHz Skylake i7-6700k, Linux 4.15.8-1-ARCH. – Peter Cordes May 19 '18 at 3:50
  • Now I'm curious where the break-even point is for builtin printf vs. fork+exec of seq. I guess I could wrap a repeat loop around the whole thing and time that. And BTW, time awk 'BEGIN{ for(i=1; i<1000000 ; i++) {print i}}' > /dev/null is about 3x faster than printf "%s\n" {1..1000000}, so even if you don't have seq, a good AWK implementation can beat builtins for large problem sizes. – Peter Cordes May 19 '18 at 3:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.