What happens when files are added/removed in the middle of a “for f in *” sh loop?

Question

I found one for loop example online. Now I want to use it in my code but I am not sure how does this loop operates

for entry in "$search_dir"/* 
do
  echo "$entry"
done

Now I want to ask that

1. Does it look in search_dir in each iteration and copies files in search_dir to entry variable one file in each iteration?
2. Or I take a snapshot of all the contents of search_dir and then store that snapshot to entry variable?
3. Is there any change in output if some one inserts some file in search_dir while the loop is still working?

Answer 1

When the shell gets to the for-statement, it will expand the value of $search_dir and perform the file name globbing to generate a list of directory entries that will be iterated over. This happens only once, and if the things in $search_dir disappears or if there are new files/directories added to that directory while the loop is executing, these changes will not be picked up.

If the loop operates on the directory entries whose names are in $entry, one might want to test for their existence in the loop, especially if the loop is known to take a long time to run and there are lots of files that are in constant flux for one reason or another:

for entry in "$search_dir"/*; do
    if [ -e "$entry" ]; then
        # operate on "$entry"
    else
        # handle the case that "$entry" went away
    fi
done

As Stéphane rightly points out in comments, this is a superfluous test in most cases.

Answer 2

The shell fully determines the list of values to loop on before it starts running the loop body. That is:

1. The shell builds a path using value of the variable search_dir.
2. The shell collects the list of file names in the specified directory, to build the list of matches for the wildcard pattern.
3. The shell executes the loop body, in turn, with each element of the list of matches.

You can change the value of the variable search_dir, and change the content of the directory, while the loop is running. This won't affect what files the loop acts on.

If a file is removed while the loop is processing other files, then once it gets to that file, that file won't exist. Depending on what you do in the loop, it may or may not matter. If there is a concurrent process that may remove files, note that testing whether the file exists before processing it won't really solve that problem, since the file could be removed between the time you test and the time you start processing.

If you need to mark a file as processed to make sure that you don't process it twice, then this script should move the files to another directory once it's processed them. Moving a file to another directory (on the same filesystem) is atomic: either it isn't done yet or it's done, there's no intermediate state. But once again, if a different process (possibly another instance of that scripts) moves files, then the loop will sometimes hit files that are moved while the loop is executing.

If you want to process new files as they are created, then you'll need to loop again. Obviously files could be created during the execution of the loop, or after all the prior files have been processed, so the script would need to keep running forever. There are tools to wait until a file is created in a directory. On Linux, the basic facility for that is inotify; if you need to process files as they are created, then inotifywait or incron should help you. Remember that inotify only notifies you of files created (or modified or accessed depending on your trigger) after the inotify-based commands start; you'll also need to take care of prior existing files, and you can't just do for entry in *; do …; done; inotifywait … because files can be created during the execution of the loop or even while the inotifywait command is starting up.