5

I found one for loop example online. Now I want to use it in my code but I am not sure how does this loop operates

for entry in "$search_dir"/* 
do
  echo "$entry"
done

Now I want to ask that

  1. Does it look in search_dir in each iteration and copies files in search_dir to entry variable one file in each iteration?
  2. Or I take a snapshot of all the contents of search_dir and then store that snapshot to entry variable?
  3. Is there any change in output if some one inserts some file in search_dir while the loop is still working?
5

When the shell gets to the for-statement, it will expand the value of $search_dir and perform the file name globbing to generate a list of directory entries that will be iterated over. This happens only once, and if the things in $search_dir disappears or if there are new files/directories added to that directory while the loop is executing, these changes will not be picked up.

If the loop operates on the directory entries whose names are in $entry, one might want to test for their existence in the loop, especially if the loop is known to take a long time to run and there are lots of files that are in constant flux for one reason or another:

for entry in "$search_dir"/*; do
    if [ -e "$entry" ]; then
        # operate on "$entry"
    else
        # handle the case that "$entry" went away
    fi
done

As Stéphane rightly points out in comments, this is a superfluous test in most cases.

  • 4
    In any case the file may still disappear in between the [ -e ] and the operate so that additional check doesn't necessarily add more. – Stéphane Chazelas Apr 25 '17 at 13:38
  • 4
    @unixbeginner, see en.wikipedia.org/wiki/Time_of_check_to_time_of_use – Stéphane Chazelas Apr 25 '17 at 13:44
  • 1
    @unixbeginner Unexpected in the sense that that if a file goes away, it will still be iterated over, since the generated list of files is static. – Kusalananda Apr 25 '17 at 14:01
  • 1
    @unixbeginner Um, don't mess with the stuff in that directory while the script is running? I have no idea of what context you are asking this in. It may be a private little script made to run in your home directory on a laptop, or it may be made to run in a high throughput production environment with thousands of other processes competing for resources and wanting to create and delete files all the time. – Kusalananda Apr 25 '17 at 14:07
  • 2
    @unixbeginner Those new files would not be processed by this loop. – Kusalananda Apr 25 '17 at 14:15
1

The shell fully determines the list of values to loop on before it starts running the loop body. That is:

  1. The shell builds a path using value of the variable search_dir.
  2. The shell collects the list of file names in the specified directory, to build the list of matches for the wildcard pattern.
  3. The shell executes the loop body, in turn, with each element of the list of matches.

You can change the value of the variable search_dir, and change the content of the directory, while the loop is running. This won't affect what files the loop acts on.

If a file is removed while the loop is processing other files, then once it gets to that file, that file won't exist. Depending on what you do in the loop, it may or may not matter. If there is a concurrent process that may remove files, note that testing whether the file exists before processing it won't really solve that problem, since the file could be removed between the time you test and the time you start processing.

If you need to mark a file as processed to make sure that you don't process it twice, then this script should move the files to another directory once it's processed them. Moving a file to another directory (on the same filesystem) is atomic: either it isn't done yet or it's done, there's no intermediate state. But once again, if a different process (possibly another instance of that scripts) moves files, then the loop will sometimes hit files that are moved while the loop is executing.

If you want to process new files as they are created, then you'll need to loop again. Obviously files could be created during the execution of the loop, or after all the prior files have been processed, so the script would need to keep running forever. There are tools to wait until a file is created in a directory. On Linux, the basic facility for that is inotify; if you need to process files as they are created, then inotifywait or incron should help you. Remember that inotify only notifies you of files created (or modified or accessed depending on your trigger) after the inotify-based commands start; you'll also need to take care of prior existing files, and you can't just do for entry in *; do …; done; inotifywait … because files can be created during the execution of the loop or even while the inotifywait command is starting up.

  • @giles is there any possibility that this loop will not work in case there are huge amount of files present in $searchdir since we are using wildcard * – ProgBeginner Jun 9 '17 at 11:42
  • @unixbeginner The number of files isn't a problem. The list of file names only has to fit in memory, it doesn't have to fit in a command line length limit like somecommand * would. – Gilles Jun 9 '17 at 12:06
  • currently I am observing that when no. of files in search_dir exceeds over 2 lac it show too many arguments error is there any other way to solve this issue – ProgBeginner Jun 9 '17 at 13:49
  • @unixbeginner The snippet you posted in the question would not show this message in bash. If you have another problem, ask a new question, with a complete script. – Gilles Jun 9 '17 at 20:39
  • that means this loop should work fine even there is more than 10 lac files in search_dir since i am not using wildcard with somecommand as you mentioned above in one of your comment – ProgBeginner Jun 10 '17 at 5:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.