Moving millions of files to a different directory with specfic name patterns

Question

I have millions of files with the following nomenclature on a Linux machine:

1559704165_a1ac6f55fef555ee.jpg

The first 10 digits are timestamp and the ones followed by a _ are specific ids. I want to move all the files matching specific filename ids to a different folder.

I tried this on the directory with files

find . -maxdepth 1 -type f | ??????????_a1ac*.jpg |xargs mv -t "/home/ubuntu/ntest"

However I am getting an error indicating:

bash 1559704165_a1ac6f55fef555ee.jpg: command not found

When I tried, mv ??????????_a1ac*.jpg I am getting argument list too long error. I have atleast 15 different filename patterns. How do I move them.

Answer 1

You should use:

find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' \
-exec mv -t destination "{}" +

So maxdepth 1 means that you want to search in current directory no subdirectories.

type f means find only files.

name '??????????_a1ac*.jpg' is a pattern that matches with file you are searching.

mv -t destination "{}" + means move matched files to destination. Here + adds new matched files to previous one like:

mv -t dest a b c d

Here a b c d are different files.

Answer 2

Your command,

find . -maxdepth 1 -type f | ??????????_a1ac*.jpg |xargs mv -t "/home/ubuntu/ntest"

Pipes the list of all the files TO all the files!

find . -maxdepth 1 -type f -name `*_a1ac*.jpg` -print0 |\
xargs  -0 -r mv -t "/home/ubuntu/ntest"

will do the trick.

Answer 3

You're very close. You should use the -name option to find. And remember to quote the pattern.

So

find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' |xargs mv -t "/home/ubuntu/ntest"

Answer 4

Not as "good" as the find solutions, but another valid solution is to make the mv commands more granular.

This does 4096 moves, with a fewer number of files moved per mv operation.

FILEPAT=a1ac
for i in $(seq $((0x000)) $((0xfff))); 
do 
   H=$(printf '%x\n' $i)
   mv 1559704165_${FILEPAT}${H}*.jpg /home/ubuntu/ntest
done

Answer 5

If you want to move files on the same host system, which I guess you are doing with your mv, rsync could be a faster option:

rsync -av --inplace -W /source/??????????_a1ac*.jpg /home/ubuntu/ntest/

--inplaceand -W are set to speed up the process.

Should this yield another argument list too long error then you could feed lists to rsync

Make the list with find, for example

find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' > /tmp/my_image_list.txt

and give it to rsync

rsync -av --inplace -W --files-from=/tmp/my_image_list.txt /path/to/files /home/ubuntu/ntest/

The source here is /path/to/files, because rsync will treat the list you give it as relative to your source.

---

The point being: rsync is faster than mv, if the files are not on the same filesystem.