5

I'm moving a large number of files (400K+) from one directory to another and I have the following script to do so (too many files for the mv command to work directly):

for file in *;
do
    mv $file ..
done

If I run this script twice (or more) at the same time, will there be a race condition when/if the mv commands are trying to access the same file?

I've looked around on the web but haven't found any definite answer. Thanks!

1
  • 1
    If you were thinking of running multiple instances in parallel to speed up the move, note that this is unlikely to help because either I/O or modifying the directory will be the bottleneck. Commented Nov 7, 2014 at 1:22

3 Answers 3

5

There is indeed a race condition (one that wouldn't cause harm, though).

The * is expanded on entry to the loop. If you run a second instance of this script simultaneously then it will probably do nothing because all files it tries to move have already been moved. If no files are created in the source directory during the moving operation then the error messages should be your biggest problem.

But in general this structure is a very bad idea. * expands to a sorted list. AFAIK it is not possible to deactivate that. Obviously, the sorting alone is a nightmare with 400K files. See man bash, section "Pathname Expansion":

After word splitting, unless the -f option has been set, bash scans each word for the characters *, ?, and [. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern.

Furthermore you should not run one mv instance per file as you can move several files at once.

This is a better solution (in the GNU world):

find . -mindepth 1 -maxdepth 1 -exec mv --target-directory=DIRECTORY {} +
4
  • Can you point me to some documentation about how * is expanded on entry to the loop? I would think it would pause before executing (to sort) but it just jumps right in (making it appear to not pre-sort the list).
    – D. Gibbs
    Commented Nov 6, 2014 at 5:02
  • @D.Gibbs I have extended the answer. What makes you believe it is not sorted? Commented Nov 6, 2014 at 10:43
  • Thank you for the extended answer, it has cleared things up for me. I misjudged the speed of the pre-sort before processing. It is quite fast.
    – D. Gibbs
    Commented Nov 6, 2014 at 15:55
  • @D.Gibbs Sorting itself is very fast — it's only a bit of computation in barely more than linear time. What can take long is that the script has to read the whole list of file names before acting on the first files. This won't take so long once the directory is in the cache. Commented Nov 7, 2014 at 1:22
4

An even better solution would be to use GNU Parallel to insert multiple arguments. By default, Parallel will run n jobs simultaneously, with n being the number of cores your CPU has.

When moving a lot of files like this: mv * destdir you will sometimes get the error:

bash: /bin/mv: Argument list too long

because there are too many files. You can instead do:

ls -1 | parallel mv {} destdir

This will run mv for each file. It can be done faster if mv gets as many arguments that will fit on the line:

ls -1 | parallel -m mv {} destdir


The -m option is really cool for moving or copying files in parallel:

-m      Multiple arguments. Insert as many arguments as the command
        line length permits. If multiple jobs are being run in
        parallel: distribute the arguments evenly among the jobs.
        Use -j1 to avoid this.
0

For those who don't have parallel available like me:

find source_dir -type f | xargs -n 1 -P 20 -I '{}' mv '{}' dest_dir/

You define the number of processes with the -P param on xargs (set to 20 in the example above).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .