Move 20,000 files from one directory sequentially and so on

Question

What I am trying to accomplish is I have a directory with lets say 176,000 files. I want the script to split the 176,000 files into folders with lets say 20,000 files each and the last one containing the odd ball #. I would like to copy the files sequentially and keeping them in order alpha numerical. This is what I currently have but it seems to be missing something, the first 2 folders it creates are out of order, the other 3 folders it creates are sequential.

Bottom line, folder contains 176,000 files, script would move the first 20,000 into a folder called 'split' then takes the next 20,000 and so on, creating split2, split3, etc..

Here is what I have:

#!/bin/bash

dir_size=20000
dir_name="split"
n=$((`find . -maxdepth 1 -type f | wc -l`/$dir_size+1))
for i in `seq 1 $n`;
do
    mkdir -p "$dir_name$i";
    find  .  -maxdepth 1 -type f | sort -n -z |  head -n $dir_size |  xargs -i mv "{}" "$dir_name$i"
done

Answer 1

• Make a list of your files (note that you cannot use ls * or ls*.dpx because this would create a command line with 176000 parameters to ls) and use split to split this list in chunks (files named files.01, files.02...)(the lone - is telling split to work on standard input).

ls | split -d --lines 20000 - files.

• For more stringent filtering, you can also use find as you did, piping output to split after a sort:

find . type f | sort | split -d --lines 20000 - files.

• Iterate the produced files.*, create the directory name using the numeric part, and then move the files:

for f in files.*  # iterate the files.*
do 
    dirname="/path/to/dir${f#*.}"  # generate directory name from numeric suffix
    mkdir -p "$dirname" # create the directory
    xargs mv -t "$dirname" < $f # move the files
done

Warning: untested, try on a small sample first.

If you have spaces in the file names use find -print0, sort -z, split --separator '\0' and xargs -0.