Can multiple mv commands be run in the same directory (race condition)?

Question

I'm moving a large number of files (400K+) from one directory to another and I have the following script to do so (too many files for the mv command to work directly):

for file in *;
do
    mv $file ..
done

If I run this script twice (or more) at the same time, will there be a race condition when/if the mv commands are trying to access the same file?

I've looked around on the web but haven't found any definite answer. Thanks!

Answer 1

There is indeed a race condition (one that wouldn't cause harm, though).

The * is expanded on entry to the loop. If you run a second instance of this script simultaneously then it will probably do nothing because all files it tries to move have already been moved. If no files are created in the source directory during the moving operation then the error messages should be your biggest problem.

But in general this structure is a very bad idea. * expands to a sorted list. AFAIK it is not possible to deactivate that. Obviously, the sorting alone is a nightmare with 400K files. See man bash, section "Pathname Expansion":

After word splitting, unless the -f option has been set, bash scans each word for the characters *, ?, and [. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern.

Furthermore you should not run one mv instance per file as you can move several files at once.

This is a better solution (in the GNU world):

find . -mindepth 1 -maxdepth 1 -exec mv --target-directory=DIRECTORY {} +

Answer 2

An even better solution would be to use GNU Parallel to insert multiple arguments. By default, Parallel will run n jobs simultaneously, with n being the number of cores your CPU has.

When moving a lot of files like this: mv * destdir you will sometimes get the error:

bash: /bin/mv: Argument list too long

because there are too many files. You can instead do:

ls -1 | parallel mv {} destdir

This will run mv for each file. It can be done faster if mv gets as many arguments that will fit on the line:

ls -1 | parallel -m mv {} destdir

The -m option is really cool for moving or copying files in parallel:

-m      Multiple arguments. Insert as many arguments as the command
        line length permits. If multiple jobs are being run in
        parallel: distribute the arguments evenly among the jobs.
        Use -j1 to avoid this.

Answer 3

For those who don't have parallel available like me:

find source_dir -type f | xargs -n 1 -P 20 -I '{}' mv '{}' dest_dir/

You define the number of processes with the -P param on xargs (set to 20 in the example above).