Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I'm moving a large number of files (400K+) from one directory to another and I have the following script to do so (too many files for the mv command to work directly):
for file in *;
do
mv $file ..
done
If I run this script twice (or more) at the same time, will there be a race condition when/if the mv commands are trying to access the same file?
I've looked around on the web but haven't found any definite answer. Thanks!
There is indeed a race condition (one that wouldn't cause harm, though).
The * is expanded on entry to the loop. If you run a second instance of this script simultaneously then it will probably do nothing because all files it tries to move have already been moved. If no files are created in the source directory during the moving operation then the error messages should be your biggest problem.
But in general this structure is a very bad idea. * expands to a sorted list. AFAIK it is not possible to deactivate that. Obviously, the sorting alone is a nightmare with 400K files. See man bash, section "Pathname Expansion":
After word splitting, unless the -f option has been set, bash scans each word for the characters *, ?, and [. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern.
Furthermore you should not run one mv instance per file as you can move several files at once.
This is a better solution (in the GNU world):
find . -mindepth 1 -maxdepth 1 -exec mv --target-directory=DIRECTORY {} +
* is expanded on entry to the loop? I would think it would pause before executing (to sort) but it just jumps right in (making it appear to not pre-sort the list).
An even better solution would be to use GNU Parallel to insert multiple arguments. By default, Parallel will run n jobs simultaneously, with n being the number of cores your CPU has.
When moving a lot of files like this: mv * destdir you will sometimes get the error:
bash: /bin/mv: Argument list too long
because there are too many files. You can instead do:
ls -1 | parallel mv {} destdir
This will run mv for each file. It can be done faster if mv gets as many arguments that will fit on the line:
ls -1 | parallel -m mv {} destdir
The -m option is really cool for moving or copying files in parallel:
-m Multiple arguments. Insert as many arguments as the command
line length permits. If multiple jobs are being run in
parallel: distribute the arguments evenly among the jobs.
Use -j1 to avoid this.
For those who don't have parallel available like me:
find source_dir -type f | xargs -n 1 -P 20 -I '{}' mv '{}' dest_dir/
You define the number of processes with the -P param on xargs (set to 20 in the example above).
