3

I have around 100,000 files in a directory. As you can imagine, that's not an ideal situation, and I'd like to split the directory up into subdirectories. But the filenames aren't such where I can put them into subdirectories based on their names or anything, so I'd like to create, say, 10 directories and put about 1000 files in each. Is there an easy bash command or loop that would do this?

marked as duplicate by don_crissti, John1024, Scott, slm bash May 26 '16 at 5:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • So, you want these 100000 files in one single directory to become 10 directories containing 1000 files each ? That's 10000 files, what do you plan to do with the remaining 90000 files ? – Olivier Dauby May 25 '16 at 23:09
2

Try:

 n=0; for f in *; do d="subdir$((n++ / 1000))"; mkdir -p "$d"; mv -- "$f" "$d/$f"; done

n keeps track of how many files have been moved. In batches of a thousands, each file is assigned to a subdirectory named subdir$((n++ / 1000)). (You can adjust the naming as you like.) If it hasn't already been created, mkdir -p "$d" creates that subdirectory. mv -- "$f" "$d/$f" transfers the file to that subdirectory.

  • @don_crissti Yes, very very close anyway. That other one was closed as a duplicate of a question that I did not think it was a duplicate of. Would it work to close this as a duplicate of the one you link to? – John1024 May 25 '16 at 23:08
  • @don_crissti OK, I have seconded your vote. – John1024 May 25 '16 at 23:15
1

Given the largish number of files in the source directory, the proposed solution will likely take a couple of minutes.

If you don't want to wait, you might want to consider a solution that doesn't spawn a process for each move.

To batch the move, you can utilize the fact that cpio with --pass-through and --link sort of behaves like a move -- it the source and destination are on the same filesystem, the op will be quick, otherwise a whole copy will be made.

Unlike mv, cpio -lpd will leave a hardlink to each source file, but those are easy to remove en-mass with rm -r later if the operation was a succcess:

The implementation of the directory splitter might look something like:

split_dir()(
  local src=$1 nbatches=$2 dest=$3
  local tmpd=
  trap 'rm -rf "$tmpd"' EXIT
  tmpd="$(mktemp -d "$tmpd")"
  find -type f "$src" > "$tmpd/files"
  local nfiles=$(wc -l < "$tmpd/files")
  local batchsz=$((nfiles/nbatches))
  ( cd "$tmpd"; split -l "$batchsz" -d files )
  local i
  for((i=0;i<nbatches;i++)); do
    mkdir -p "$dest/$i"
    < x$i cpio -pdl "$dest/$i"
  done
)

Not the answer you're looking for? Browse other questions tagged or ask your own question.