Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let me give an example:

$ timeout 1 yes "GNU" > file1
$ wc -l file1
11504640 file1

$ for ((sec0=`date +%S`;sec<=$(($sec0+5));sec=`date +%S`)); do echo "GNU" >> file2; done
$ wc -l file2
1953 file2

Here you can see that the command yes writes 11504640 lines in a second while I can write only 1953 lines in 5 seconds using bash's for and echo.

As suggested in the comments, there are various tricks to make it more efficient but none come close to matching the speed of yes:

$ ( while :; do echo "GNU" >> file3; done) & pid=$! ; sleep 1 ; kill $pid
[1] 3054
$ wc -l file3
19596 file3

$ timeout 1 bash -c 'while true; do echo "GNU" >> file4; done'
$ wc -l file4
18912 file4

These can write up to 20 thousand lines in a second. And they can be further improved to:

$ timeout 1 bash -c 'while true; do echo "GNU"; done >> file5' 
$ wc -l file5
34517 file5

$ ( while :; do echo "GNU"; done >> file6 ) & pid=$! ; sleep 1 ; kill $pid
[1] 5690
$ wc -l file6
40961 file6

These get us up to 40 thousand lines in a second. Better, but still a far cry from yes which can write about 11 million lines in a second!

So, how does yes write to file so quickly?

share|improve this question
8  
8  
In the second example, you have two external command invocations for every iteration of the loop, and date is somewhat heavy-weight, plus the shell has to re-open the output stream for echo for every loop iteration. In the first example, there is only a single command invocation with a single output redirection, and the command is extremely lightweight. The two are in no way comparable. – Michael Kjörling Jan 24 at 14:50
    
@MichaelKjörling you're right date may be heavy-weight, see edit to my question. – Pandya Jan 24 at 15:13
1  
timeout 1 $(while true; do echo "GNU">>file2; done;) is the wrong way to use timeout since the timeout command will only start once the command substitution is finished. Use timeout 1 sh -c 'while true; do echo "GNU">>file2; done'. – muru Jan 24 at 21:56
    
A shell interprets its programs, and invokes them with a lot of argument preprocessing. This takes time, and that is what you see. – Thorbjørn Ravn Andersen Jan 25 at 10:35
up vote 57 down vote accepted

nutshell:

yes exhibits similar behavior to most other standard utilities which typically write to a FILE STREAM with output buffered by the libC via stdio. These only do the syscall write() every some 4kb (16kb or 64kb) or whatever the output block BUFSIZ is . echo is a write() per GNU. That's a lot of mode-switching (which is not, apparently, as costly as a context-switch).

And that's not at all to mention that, besides its initial optimization loop, yes is a very simple, tiny, compiled C loop and your shell loop is in no way comparable to a compiler optimized program.


but i was wrong:

When I said before that yes used stdio, I only assumed it did because it behaves a lot like those that do. This was not correct - it only emulates their behavior in this way. What it actually does is very like an analog to the thing I did below with the shell: it first loops to conflate its arguments (or y if none) until they might grow no more without exceeding BUFSIZ.

A comment from the source immediately preceding the relevant for loop states:

/* Buffer data locally once, rather than having the
large overhead of stdio buffering each item.  */

yes does its does its own write()s thereafter.


digression:

(As originally included in the question and retained for context to a possibly informative explanation already written here):

I've tried timeout 1 $(while true; do echo "GNU">>file2; done;) but unable to stop loop.

The timeout problem you have with the command substitution - I think I get it now, and can explain why it doesn't stop. timeout doesn't start because its command-line is never run. Your shell forks a child shell, opens a pipe on its stdout, and reads it. It will stop reading when the child quits, and then it will interpret all the child wrote for $IFS mangling and glob expansions, and with the results it will replace everything from $( to the matching ).

But if the child is an endless loop that never writes to the pipe, then the child never stops looping, and timeout's command-line is never completed before (as I guess) you do CTRL-C and kill the child loop. So timeout can never kill the loop which needs to complete before it can start.


other timeouts:

...simply aren't as relevant to your performance issues as the amount of time your shell program must spend switching between user- and kernel-mode to handle output. timeout, though, is not as flexible as a shell might be for this purpose: where shells excel is in their ability to mangle arguments and manage other processes.

As is noted elsewhere, simply moving your [fd-num] >> named_file redirection to the loop's output target rather than only directing output there for the command looped over can substantially improve performance because that way at least the open() syscall need only be done the once. This also is done below with the | pipe targeted as output for the inner loops.


direct comparison:

You might do like:

for cmd in  exec\ yes 'while echo y; do :; done'
do      set +m
        sh  -c '{ sleep 1; kill "$$"; }&'"$cmd" | wc -l
        set -m
done

256659456
505401

Which is kind of like the command sub relationship described before, but there's no pipe and the child is backgrounded until it kills the parent. In the yes case the parent has actually been replaced since the child was spawned, but the shell calls yes by overlaying its own process with the new one and so the PID remains the same and its zombie child still knows who to kill after all.


bigger buffer:

Now lets see about increasing the shell's write() buffer.

IFS="
";    set y ""              ### sets up the macro expansion       
until [ "${512+1}" ]        ### gather at least 512 args
do    set "$@$@";done       ### exponentially expands "$@"
printf %s "$*"| wc -c       ### 1 write of 512 concatenated "y\n"'s  

1024

I chose that number because output strings any longer than 1kb were getting split out into separate write()'s for me. And so here's the loop again:

for cmd in 'exec  yes' \
           'until [ "${512+:}" ]; do set "$@$@"; done
            while printf %s "$*"; do :; done'
do      set +m
        sh  -c $'IFS="\n"; { sleep 1; kill "$$"; }&'"$cmd" shyes y ""| wc -l
        set -m
done

268627968
15850496

That's 300 times the amount of data written by the shell in the same amount of time for this test than the last. Not too shabby. But it's not yes.


related:

As requested, there is a more thorough description than the mere code comments on what is done here at this link.

share|improve this answer
    
@heemayl - maybe? im not totally sure i understand what youre asking? when a program uses stdio to write output it does with either no buffering (like stderr by default) or line buffering (to terminals by default) or block buffering (basically most other stuff is set this way by default). im a little unclear on what sets the size of the output buffer - but it's usually some of 4kb. and so the stdio lib functions will gather their output until they can write a whole block. dd is one standard tool which definitely does not use stdio, for example. most others do. – mikeserv Jan 24 at 5:38
3  
The shell version is doing open (existing) write AND close (which I believe still waits for flush), AND creating a new process and executing date, for each loop. – dave_thompson_085 Jan 24 at 8:11
    
@dave_thompson_085 - go to /dev/chat. and what you say is not necessarily true, as you can see there. For example, doing that wc -l loop with bash for me gets 1/5th of the output the sh loop does - bash manages a little over 100k writes() to dash's 500k. – mikeserv Jan 24 at 8:13
    
Sorry I was ambiguous; I meant the shell version in the question, which at the time I read it had only the original version with the for((sec0=`date +%S`;... to control time and the redirection in the loop, not the subsequent improvements. – dave_thompson_085 Jan 26 at 3:07
    
@dave_thompson_085 - its fine. the answer was wrong anyway about some fundamental points, and should be pretty much correct now, as l hope. – mikeserv Jan 26 at 3:38

A better question would be why is your shell writing the file so slowly. Any self-contained compiled program that uses file writing syscalls responsibly (not flushing every character at a time) would do it reasonably quicky. What you are doing, is writing lines in an interpreted language (the shell), and in addition you do a lot of unnecessary input output operations. What yesdoes:

  • opens a file for writing
  • calls optimized and compiled functions for writing to a stream
  • the stream is buffered, so a syscall (an expensive switch to kernel mode) happens very rarely, in large chunks
  • closes a file

What your script does:

  • reads in a line of code
  • interprets the code, making a lot of extra operations to actually parse your input and figure out what to do
  • for each iteration of while loop (which is probably not cheap in an interpreted language):
    • call the date external command and store its output (only in the original version - in the revised version you gain a factor of 10 by not doing this)
    • test whether the loop's termination condition is met
    • open a file in append mode
    • parse echo command, recognize it (with some pattern matching code) as a shell builtin, call parameter expansion and everything else on the argument "GNU", and finally write the line to the open file
    • close the file again
    • repeat the process

The expensive parts: the whole interpretation is extremely expensive (bash is doing an awful lot of preprocessing of all the input - your string could potentially contain variable substitution, process substitution, brace expansion, escape characters and more), every call of a builtin is probably a switch statement with redirect to a function that deals with the builtin, and very importantly, you open and close a file for each and every line of output. You could put >> file outside the while loop to make it a lot quicker, but you're still in an interpreted language. You are quite lucky that echo is a shell builtin, not an external command - otherwise, your loop would involve creating a new process (fork & exec) on every single iteration. Which would grind the process to a halt - you saw how costly that is when you had the date command in the loop.

share|improve this answer

The other answers have addressed the main points. On a side note, you can increase the throughput of your while loop by writing to the output file at the end of the computation. Compare:

$ i=0;time while  [ $i -le 1000 ]; do ((++i)); echo "GNU" >>/tmp/f; done;

real    0m0.080s
user    0m0.032s
sys     0m0.037s

with

$ i=0;time while  [ $i -le 1000 ]; do ((++i)); echo "GNU"; done>>/tmp/f;

real    0m0.030s
user    0m0.019s
sys     0m0.011s
share|improve this answer
    
Yes, this matters and writing speed (at-least) doubles in my case – Pandya Jan 25 at 12:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.