Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

When looping through files there are two ways:

1. use a for-loop

for f in *; do
    echo "$f"
done

2. use find

find * -prune | while read f; do 
    echo "$f"
done

Assuming these two loops will find the same list of files, what are the differences in those two options in perfomance and handling?

share|improve this question
    
on large amount of files, you might need to circumvent “Too many open files” in debian –  rubo77 Oct 22 '13 at 8:36
1  
Why? find doesn't open the files it finds. The only thing I can see biting you here with respect to a large number of files is ARG_MAX. –  kojiro Oct 22 '13 at 13:22
    
See the answers and comments that tell you that read f will mangle file names as it reads them (e.g. names with leading blanks). Also find * -prune seems to be a very convoluted way to say simply ls -1 yes? –  IDAllen Oct 22 '13 at 14:20
2  
Don't assume that the two loops will find the same set of files; in most cases, they won't. Also, that should be find ., not find *. –  alexis Oct 22 '13 at 14:22
5  
I suspect that we've each spent more time reading this question and responses than the total difference in performance over the life of the script in question. –  mpez0 Oct 22 '13 at 18:43
show 8 more comments

9 Answers

up vote 8 down vote accepted

1.

The first one:

for f in *; do
  echo "$f"
done

fails for files called -n, -e and variants like -nene and with some bash deployments, with filenames containing backslashes.

The second:

find * -prune | while read f; do 
  echo "$f"
done

fails for even more cases (files called !, -H, -name, (, file names that start or end with blanks or contain newline characters...)

It's the shell that expands *, find does nothing but print the files it receives as arguments. You could as well have used printf '%s\n' instead which as printf is builtin would also avoid the too many args potential error.

2.

The expansion of * is sorted, you can make it a bit faster if you don't need the sorting. In zsh:

for f (*(oN)) printf '%s\n' $f

or simply:

printf '%s\n' *(oN)

bash has no equivalent as far as I can tell, so you'd need to resort to find.

3.

find . ! -name . -prune ! -name '.*' -print0 |
  while IFS= read -rd '' f; do
    printf '%s\n' "$f"
  done

(above using a GNU/BSD -print0 non-standard extension).

That still involves spawning a find command and use a slow while read loop, so it will probably be slower than using the for loop unless the list of files is huge.

4.

Also, contrary to shell wildcard expansion, find will do a lstat system call on each file, so it's unlikely that the non-sorting will compensate for that.

With GNU/BSD find, that can be avoided by using their -maxdepth extension which will trigger an optimization saving the lstat:

find . -maxdepth 1 ! -name '.*' -print0 |
  while IFS= read -rd '' f; do
    printf '%s\n' "$f"
  done

Because find starts outputting file names as soon as it finds them (except for the stdio output buffering), where it may be faster is if what you do in the loop is time consuming and the list of file names is more than a stdio buffer (4/8 kB). In that case, the processing within the loop will start before find has finished finding all the files. On GNU and FreeBSD systems, you may use stdbuf to cause that to happen sooner (disabling stdio buffering).

5.

The POSIX/standard/portable way to run commands for each file with find is to use the -exec predicate:

find . ! -name . -prune ! -name '.*' -exec some-cmd {} ';'

In the case of echo though, that's less efficient than doing the looping in the shell as the shell will have a builtin version of echo while find will need to spawn a new process and execute /bin/echo in it for each file.

If you need to run several commands, you can do:

find . ! -name . -prune ! -name '.*' -exec cmd1 {} ';' -exec cmd2 {} ';'

But beware that cmd2 is only executed if cmd1 is successful.

6.

A canonical way to run complex commands for each file is to call a shell with -exec ... {} +:

find . ! -name . -prune ! -name '.*' -exec sh -c '
  for f do
    cmd1 "$f"
    cmd2 "$f"
  done' sh {} +

That time, we're back to being efficient with echo since we're using sh's builtin one and the -exec + version spawns as few sh as possible.

7.

In my tests on a directory with 200.000 files with short names on ext4, the zsh one (paragraph 2.) is by far the fastest, followed by the first simple for i in * loop (though as usual, bash is a lot slower than other shells for that).

share|improve this answer
    
what does the ! do in the find command? –  rubo77 Feb 17 at 9:43
    
@rubo77, ! is for negation. ! -name . -prune more... will do -prune (and more... since -prune always returns true) for every file but .. So it will do more... on all the files in ., but will exclude . and will not descend into subdirectories of .. So it's the standard equivalent of GNU's -mindepth 1 -maxdepth 1. –  Stephane Chazelas Feb 17 at 10:04
add comment

I tried this on a directory with 2259 entries, and used the time command.

The output of time for f in *; do echo "$f"; done (minus the files!) is:

real    0m0.062s
user    0m0.036s
sys     0m0.012s

The output of time find * -prune | while read f; do echo "$f"; done (minus the files!) is:

real    0m0.131s
user    0m0.056s
sys     0m0.060s

I ran each command several times, so as to eliminate cache misses. This suggests keeping it in bash (for i in ...) is quicker than using find and piping the output (to bash)

Just for completeness, I dropped the pipe from find, since in your example, it's wholly redundant. The output of of just find * -prune is:

real    0m0.053s
user    0m0.016s
sys     0m0.024s

Also, time echo * (output isn't newline separated, alas):

real    0m0.009s
user    0m0.008s
sys     0m0.000s

At this point, I suspect the reason echo * is quicker is it's not outputting so many newlines, so the output isn't scrolling as much. Let's test...

time find * -prune | while read f; do echo "$f"; done > /dev/null

yields:

real    0m0.109s
user    0m0.076s
sys     0m0.032s

while time find * -prune > /dev/null yields:

real    0m0.027s
user    0m0.008s
sys     0m0.012s

and time for f in *; do echo "$f"; done > /dev/null yields:

real    0m0.040s
user    0m0.036s
sys     0m0.004s

and finally: time echo * > /dev/null yields:

real    0m0.011s
user    0m0.012s
sys     0m0.000s

Some of the variation can be accounted for by random factors, but it seems clear:

  • output is slow
  • piping costs a bit
  • for f in *; do ... is slower than find * -prune, on its own, but for the constructions above involving pipes, is quicker.

Also, as an aside, both approaches appear to handle names with spaces just fine.

EDIT:

Timings for find . -maxdepth 1 > /dev/null vs. find * -prune > /dev/null:

time find . -maxdepth 1 > /dev/null:

real    0m0.018s
user    0m0.008s
sys     0m0.008s

find * -prune > /dev/null:

real    0m0.031s
user    0m0.020s
sys     0m0.008s

So, additional conclusion:

  • find * -prune is slower than find . -maxdepth 1 - in the former, the shell is processing a glob, then building a (large) command line for find. NB: find . -prune returns just ..

More tests: time find . -maxdepth 1 -exec echo {} \; >/dev/null:

real    0m3.389s
user    0m0.040s
sys     0m0.412s

Conclusion:

  • slowest way to do it so far. As was pointed out in the comments for the answer where this approach was suggested, each argument spawns a shell.
share|improve this answer
    
Which pipe is redundant? can you show the line you used without pipe? –  rubo77 Oct 22 '13 at 8:33
2  
@rubo77 find * -prune | while read f; do echo "$f"; done has the redundant pipe - all the pipe is doing is outputting exactly what find outputs on its own. Without a pipe, it would be simply find * -prune The pipe is only redundant specifically because the thing on the other side of the pipe simply copies stdin to stdout (for the most part). It's an expensive no-op. If you want to do stuff with the output of find, other than to just spit it back out again, that's different. –  Phil Oct 22 '13 at 8:36
    
Maybe the main timeconsuming is the *. As BitsOfNix stated: I still strongly suggest to not use * and . for find instead. –  rubo77 Oct 22 '13 at 8:45
    
@rubo77 seems that way. I guess I overlooked that. I've added findings for my system. I assume find . -prune is faster because find will be reading a directory entry verbatim, while the shell will be doing likewise, potentially matching against the glob (might optimize for *), then building the large command line for find. –  Phil Oct 22 '13 at 8:57
1  
find . -prune prints only . on my system. It does almost no work at all. It is not at all the same as find * -prune that shows all the names in the current directory. A bare read f will mangle file names with leading spaces. –  IDAllen Oct 22 '13 at 14:13
show 5 more comments

I would go definitely with find although I would change your find to just this:

find . -maxdepth 1 -exec echo {} \;

Performance wise, find is a lot faster depending on your needs of course. What you have currently with for it will only display the files/directories in the current directory but not the directories contents. If you use find it will also show the contents of the sub-directories.

I say find is better since with your "for" the '*' will have to be expanded first and I'm afraid that if you have a directory with a huge amount of files it might give the error argument list too long. Same goes for find *

As an example, in one of the systems that I currently use there is a couple of directories with over 2million files (<100k each):

find *
-bash: /usr/bin/find: Argument list too long
share|improve this answer
    
I added -prune to make the two examples more alike. and I prefer the pipe with while so it is easier to apply more commands in the loop –  rubo77 Oct 22 '13 at 7:03
    
You can circumvent “Too many open files” in debian –  rubo77 Oct 22 '13 at 7:04
    
changing the hard limit is hardly a proper workaround from my POV. Specially when talking about 2+ million files. Without digression from the Question, for simple cases as one level directory for is faster, but if you change your file/directory structure it will be harder to migrate. While with find and it's huge amount of options you can be better prepared. Still I still strongly suggest to not use * and . for find instead. It would be more portable than * where you might not be able to control the hardlimit ... –  BitsOfNix Oct 22 '13 at 7:37
3  
That will spawn one echo process per file (while in the shell for loop, it's the echo builtin that will be used without forking an extra process), and will descend into directories, so it will be a lot slower. Also note that it will include dot files. –  Stephane Chazelas Oct 22 '13 at 10:52
    
You are right, I added the maxdepth 1 so that it sticks to the current level only. –  BitsOfNix Oct 22 '13 at 10:53
show 1 more comment
find * -prune | while read f; do 
    echo "$f"
done

is a useless use of find - What you're saying is effectively "for each file in the directory (*), don't find any files. Also, it's not safe for several reasons:

  • Backslashes in paths are treated specially without the -r option to read. This is not an issue with the for loop.
  • Newlines in paths would break any non-trivial functionality inside the loop. This is not an issue with the for loop.

Handling any file name with find is difficult, so you should use the for loop option whenever possible for that reason alone. Also, running an external program like find will in general be slower than running an internal loop command like for.

share|improve this answer
    
@I0b0 What about find -path './*' -prune or find -path './[^.]*' -prune (to avoid hidden files and directories) as a better construct - in full form: find -path './*' -prune -print0 | xargs -0 sh -c ' ... '? –  AsymLabs Oct 23 '13 at 15:59
1  
Neither find's -print0 nor xargs' -0 are POSIX compatible, and you can't put arbitrary commands in sh -c ' ... ' (single quotes cannot be escaped within single quotes), so it's not quite so simple. –  l0b0 Oct 23 '13 at 22:54
add comment

But we are suckers for performance questions! This request-for-experiment makes at least two assumptions that make it not terribly valid.

A. Assume that they find the same files…

Well, they will find the same files at first, because they're both iterating over the same glob, namely *. But find * -prune | while read f suffers from several flaws that make it quite possible it won't find all the files you expect:

  1. POSIX find isn't guaranteed to accept more than one path argument. Most find implementations do, but still, you shouldn't rely on that.
  2. find * can break when you hit ARG_MAX. for f in * will not, because ARG_MAX applies to exec, not builtins.
  3. while read f can break with filenames starting and ending with whitespace, which will get stripped off. You could overcome this with while read and its default parameter REPLY, but that still won't help you when it comes to filenames with newlines in them.

B. echo. Nobody's going to do this just to echo the name of the file. If you want that, just do one of these:

printf '%s\n' *
find . -mindepth 1 -maxdepth 1 # for dotted names, too

The pipe to the while loop here creates an implicit subshell that closes when the loop ends, which might be unintuitive for some.

To answer the question, here are the results in a directory of mine that has 184 files and directories in it.

$ time bash -c 'for i in {0..1000}; do find * -prune | while read f; do echo "$f"; done >/dev/null; done'

real    0m7.998s
user    0m5.204s
sys 0m2.996s
$ time bash -c 'for i in {0..1000}; do for f in *; do echo "$f"; done >/dev/null; done'

real    0m2.734s
user    0m2.553s
sys 0m0.181s
$ time bash -c 'for i in {0..1000}; do printf '%s\n' * > /dev/null; done'

real    0m1.468s
user    0m1.401s
sys 0m0.067s

$ time bash -c 'for i in {0..1000}; do find . -mindepth 1 -maxdepth 1 >/dev/null; done '

real    0m1.946s
user    0m0.847s
sys 0m0.933s
share|improve this answer
    
I don't agree with the statement the while loop spawns a subshell - at worst case, a new thread: the following is trying to show before and after, apologies for the poor formatting $ ps ax | grep bash 20784 pts/1 Ss 0:00 -bash 20811 pts/1 R+ 0:00 grep bash $ while true; do while true; do while true; do while true; do while true; do sleep 100; done; done; done; done; done ^Z [1]+ Stopped sleep 100 $ bg [1]+ sleep 100 & $ ps ax | grep bash 20784 pts/1 Ss 0:00 -bash 20924 pts/1 S+ 0:00 grep bash –  Phil Oct 22 '13 at 18:37
    
Technically I misspoke: the pipe causes the implicit subshell, not the while loop. I'll edit. –  kojiro Oct 22 '13 at 18:43
add comment

find * will not work right if * produces tokens which look like predicates rather than paths.

You cannot use the usual -- argument to fix this because -- indicates the end of options, and find's options come before paths.

To fix this issue you can use find ./* instead. But then it's not producing exactly the same strings as for x in *.

Note that find ./* -prune | while read f .. does not actually use the scanning functionality of find. It is the globbing syntax ./* which actually traverses the directory and generates names. Then the find program will have to perform at least a stat check on each one of those names. You have the overhead of launching the program and having it access these files, and then doing I/O to read its output.

It is hard to imagine how it could be anything but less efficient than for x in ./* ....

share|improve this answer
add comment

Well for starters for is a shell keyword, built into Bash, while find is a separate executable.

$ type -a for
for is a shell keyword

$ type -a find
find is /usr/bin/find

The for loop will only find the files from the globstar character when it expands, it won't recurse into any directories it finds.

Find on the otherhand will also be given a list expanded by the globstar, but it will recursively find all the files and directories below this expanded list and pipe each one through to the while loop.

Both these approach might be considered dangerous in the sense that they don't handle paths or filenames that contain spaces.

That's about all I can think of worth commenting on these 2 approaches.

share|improve this answer
    
I added -prune to the find command, so they are more alike. –  rubo77 Oct 22 '13 at 7:08
add comment

If all of the files returned by find can be processed by a single command (obviously not applicable to your echo example above), you can use xargs:

find * |xargs some-command
share|improve this answer
add comment

For years I've been using this:-

find . -name 'filename'|xargs grep 'pattern'|more

to look for certain files (e.g. *.txt) that contain a pattern that grep can look for and pipe it into more so that it doesn't scroll off the screen. Sometimes I use the >> pipe to write the results to another file that I can look at later.

Here's a sample of the result:-

./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:Message-ID: <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:In-Reply-To: <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:  <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:  <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:Message-ID: <448E53556A3F442ABC58203D6281923E@hypermax>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2011-April.txt:URL: http://mylist.net/private/rodgersorganusers/attachments/20110420/3f
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.