Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to update the timestamp to the current time on all of the xml files in my directory (recursively). I'm using Mac OSX 10.8.5.

On about 300,000 files, the following echo command takes 10 seconds:

for file in `find . -name "*.xml"`; do echo >> $file; done

However, the following touch command takes 10 minutes! :

for file in `find . -name "*.xml"`; do touch $file; done

Why is echo so much faster than touch here?

share|improve this question

migrated from serverfault.com Apr 9 at 4:19

This question came from our site for professional system and network administrators.

16  
Just a side remark: You do know that those two commands are not equivalent, don't you? At least for Unix/Linux, the echo >> $file will append a newline to $file and thus modify it. I assume it will be the same for OS/X. If you do not want that, use echo -n >> $file. –  Dubu Apr 9 at 9:03
2  
Also wouldn't touch `find . -name "*.xml"` be even faster than both of the above? –  elmo Apr 9 at 10:27
4  
Or consider just >>$file –  gerrit Apr 9 at 12:48
7  
Not an answer to the explicit question, but why invoke touch so many times at all? find . -name '*.xml' -print0 | xargs -0 touch invokes touch much fewer times (possibly only once). Works on Linux, should work on OS X. –  Mike Renfro Apr 10 at 15:37
3  
@elmo argument list too long (easily, with 300.000 files...) –  Rmano Apr 11 at 14:10

3 Answers 3

In bash, touch is an external binary, but echo is a shell builtin:

$ type echo
echo is a shell builtin
$ type touch
touch is /usr/bin/touch

Since touch is an external binary, and you invoke touch once per file, the shell must create 300,000 instances of touch, which takes a long time.

echo, however, is a shell builtin, and the execution of shell builtins does not require forking at all. Instead, the current shell does all of the operations and no external processes are created; this is the reason why it is so much faster.

Here are two profiles of the shell's operations. You can see that a lot of time is spent cloning new processes when using touch. Using /bin/echo instead of the shell builtin should show a much more comparable result.


Using touch

$ strace -c -- bash -c 'for file in a{1..10000}; do touch "$file"; done'
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 56.20    0.030925           2     20000     10000 wait4
 38.12    0.020972           2     10000           clone
  4.67    0.002569           0     80006           rt_sigprocmask
  0.71    0.000388           0     20008           rt_sigaction
  0.27    0.000150           0     10000           rt_sigreturn
[...]

Using echo

$ strace -c -- bash -c 'for file in b{1..10000}; do echo >> "$file"; done'
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 34.32    0.000685           0     50000           fcntl
 22.14    0.000442           0     10000           write
 19.59    0.000391           0     10011           open
 14.58    0.000291           0     20000           dup2
  8.37    0.000167           0     20013           close
[...]
share|improve this answer
3  
Good answering. –  user973810 Apr 9 at 19:31
    
Did you compile strace on OS X or run your test on another OS? –  bmike Apr 9 at 19:49
1  
@bmike My test is on Linux, but the principle is identical. –  Chris Down Apr 10 at 2:24
    
I totally agree - see my comment on the main question about how /bin/echo is as slow as /bin/touch so the reasoning is sound. I just wanted to reproduce the timing of strace and failed using dtruss/dtrace and the bash -c syntax doesn't work as expected on OS X either. –  bmike Apr 10 at 2:56

As others have answered, using echo will be faster than touch as echo is a command which is commonly (though not required to be) built-in to the shell. Using it dispenses with the kernel overhead associated with running starting a new process for each file that you get with touch.

However, note that the fastest way to achieve this effect is still to use touch, but rather than running the program once for each file, it is possible to use the -exec option with find to ensure that is only run a few times. This approach will usually be faster since it avoids the overhead associated with a shell loop:

find . -name "*.xml" -exec touch {} +

Using the + (as opposed to \;) with find ... -exec runs the command only once if possible with each file as an argument. If the argument list is very long (as is the case with 300,000 files) multiple runs will be made with an argument list which has a length close to the limit (ARG_MAX on most systems).

Another advantage to this approach is that it behaves robustly with filenames containing all whitespace characters which is not the case with the original loop.

share|improve this answer
13  
+1 for pointing out the find + argument. I think many people are not aware of this (I wasn't). –  gerrit Apr 9 at 12:50
7  
Not all versions of find have the + argument. You can get a similar effect by piping to xargs. –  Barmar Apr 9 at 18:08
4  
@Barmar, the + part is required by POSIX, so should be portable. -print0 isn't. –  Graeme Apr 9 at 21:01
1  
I still occasionally run into implementations that don't have it. YMMV. –  Barmar Apr 9 at 21:03
1  
@ChrisDown, Something I have discovered is that the Busybox find has the option available but just treats it like a ; underneath the surface. –  Graeme Apr 10 at 9:25

echo is a shell builtin. On the other hand, touch is an external binary.

$ type echo
echo is a shell builtin
$ type touch
touch is hashed (/usr/bin/touch)

Shell builtins are much faster as there is no overhead involved in loading the program, i.e. there is no fork/exec involved. As such, you'd observe a significant time difference when executing a builtin vs an external command a large number of times.

This is the reason that utilities like time are available as shell builtins.

You can get the complete list of shell builtins by saying:

enable -p

As mentioned above, using the utility as opposed to the builtin results in a significant performance degradation. Following are the statistics of the time taken to create ~9000 files using the builtin echo and the utility echo:

# Using builtin
$ time bash -c 'for i in {1000..9999}; do echo > $i; done'

real    0m0.283s
user    0m0.100s
sys 0m0.184s

# Using utility /bin/echo
$ time bash -c 'for i in {1000..9999}; do /bin/echo > $i; done'

real    0m8.683s
user    0m0.360s
sys 0m1.428s
share|improve this answer
    
And I think there's an echo binary on most systems (for me it's /bin/echo), so you can retry the timing tests using that instead of the built-in –  Michael Mrozek Apr 9 at 5:31
    
@MichaelMrozek Added timing tests for the builtin and the binary. –  devnull Apr 9 at 13:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.