Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm writing a bash script to do some work on a directory that contains many (100,000+) sub-directories. Is there a predefined limit to how many arguments you can pass to a for loop such as the following?

for dir in $(find . -type d)
do
  # My code
done

I'm worried the script will bork if the find command returns too many directories.

share|improve this question
    
Out of interest, what filesystem are you using? I was under the impression that ext3, for example, was limited to 32,000 directories... –  ire_and_curses Oct 5 '12 at 3:24
    
This is an ext4 filesystem. The directories are not all in one dir, rather they are organized by md5 in subdirs, e.g. 2/f/f80d0642f8ebfb07d23547adc107cb40 –  Magnus Oct 5 '12 at 12:01

4 Answers 4

up vote 6 down vote accepted

Don't parse find output, and just use shell globbing - it's safer and built into the shell. Shell built-ins like for are not subject to the same argument list length limit as external processes since no calls do exec* are made.

for dir in ./*/; do
    # ...
done
share|improve this answer
    
I actually omitted this because I didn't think it mattered, but I need to find only directories modified within a certain time frame, e.g. find . -type d -mtime 1. Is this possible with your syntax? –  Magnus Oct 5 '12 at 0:55
1  
@Magnus Either use find -exec or check the time in the for loop. Just don't try to parse the output of find or anything that simply prints file names. –  jw013 Oct 5 '12 at 0:58
    
Is there a source you can point to regarding why this would be a bad idea? I just tried a for..in $(find) with 40,000 directories and it seems to work fine. –  Magnus Oct 5 '12 at 1:14
    
Unix file names are allowed to contain any character except \0 and /. Things like that are basically binary blobs and can't be reliably parsed using things like command substitution - too many possible funny characters to mess up the parsing. Try mkdir test && cd test && mkdir 'one dir' && for dir in $(find . -type d); do printf '%s\n' "$dir"; done. You should see two directories, one and dir. –  jw013 Oct 5 '12 at 1:19
    
Ah, that's the point you were making. However, I control exactly the files in this directory, so I know they will not contain any "funny" characters. I guess my question was more "Will a bash for loop blow up if the find list becomes too large". –  Magnus Oct 5 '12 at 1:29

Don't generate file names with a command substitution: this will break if the file names contain whitespace or \[?*.

With bash ≥4, ksh93 or zsh, you can avoid most uses of find that only use -type d or -name … predicates by using the ** glob to match subdirectories at any depth. In bash, run shopt -s globstar first. In ksh, run set -o globstar first.

for dir in **/; do …; done

Portably, or in cases where you need more, use find … -exec … + or find … -print0 | xargs -0 …. This approach has the added benefit of running find somewhat in parallel with the action on the files, though this only becomes true for huge directory trees (at least thousands of matching files). You can make find or xargs run a shell if you need to do more than run one command.

find -type d -exec sh -c 'for x in "$@"; do echo "$x"; done' _ {} +

(That _ is $0 in the shell snippet, which we don't use.)

share|improve this answer

I don't agree with the answer you accepted -- you will run into memory issues if you use $(find), even without an exec present.

Instead, write it as

find . -type d | while IFS= read -r dir
do
  # My code
done

(Note: this assumes that there is no directory name containing a newline character.)

Then you won't need the temporary memory to store the find output as you would with the command substitution. This would also work if the find or other command never terminated, e.g.:

# will not work!
for line in $(yes) ; do echo "$line" ; done

# works
yes | while IFS= read -r line ; do echo "$line" ; done
share|improve this answer
    
So your solution runs the while loop for each line of output as soon as each output line from find appears, while mine pre-computes the entire thing and then runs the for loop. If I'm understanding things correctly. Surely the find will spit out dirs faster than the while loop can process them, so it would still need to buffer the output somewhere. –  Magnus Oct 5 '12 at 12:04
    
Yes, your first sentence is correct. For the second, no, the pipe will block and so find will also block while writing its output. In other words, find will simply wait if the loop can't keep up. –  Jim Paris Oct 5 '12 at 17:19
    
Wow, that is neat. I didn't even know pipes could block. You learn something new every day! I have just rewritten my script to do it your way and avoid any SNAFUs. I regret that I have but one accepted answer to give. –  Magnus Oct 5 '12 at 17:47

If you have bash 4, you can do something recursive like :

shopt -s globstar
for dir in **/; do echo "$dir"; done
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.