0

What I am trying to accomplish is I have a directory with lets say 176,000 files. I want the script to split the 176,000 files into folders with lets say 20,000 files each and the last one containing the odd ball #. I would like to copy the files sequentially and keeping them in order alpha numerical. This is what I currently have but it seems to be missing something, the first 2 folders it creates are out of order, the other 3 folders it creates are sequential.

Bottom line, folder contains 176,000 files, script would move the first 20,000 into a folder called 'split' then takes the next 20,000 and so on, creating split2, split3, etc..

Here is what I have:

#!/bin/bash

dir_size=20000
dir_name="split"
n=$((`find . -maxdepth 1 -type f | wc -l`/$dir_size+1))
for i in `seq 1 $n`;
do
    mkdir -p "$dir_name$i";
    find  .  -maxdepth 1 -type f | sort -n -z |  head -n $dir_size |  xargs -i mv "{}" "$dir_name$i"
done
0

1 Answer 1

0
  • Make a list of your files (note that you cannot use ls * or ls*.dpx because this would create a command line with 176000 parameters to ls) and use split to split this list in chunks (files named files.01, files.02...)(the lone - is telling split to work on standard input).
ls | split -d --lines 20000 - files.
  • For more stringent filtering, you can also use find as you did, piping output to split after a sort:
find . type f | sort | split -d --lines 20000 - files.
  • Iterate the produced files.*, create the directory name using the numeric part, and then move the files:
for f in files.*  # iterate the files.*
do 
    dirname="/path/to/dir${f#*.}"  # generate directory name from numeric suffix
    mkdir -p "$dirname" # create the directory
    xargs mv -t "$dirname" < $f # move the files
done

Warning: untested, try on a small sample first.

If you have spaces in the file names use find -print0, sort -z, split --separator '\0' and xargs -0.

6
  • So what would the full syntax be? I'm fairly new with bash I just dont want to break the whole script :( I know I'm close
    – CJ Aslan
    Feb 25, 2020 at 18:03
  • See augmented answer
    – xenoid
    Feb 25, 2020 at 19:21
  • thanks man! in this line, how can I specify it to make the folders in the current working directory vs a specified path? dirname="/path/to/dir${f#*.}" # generate directory name from numeric suffix
    – CJ Aslan
    Feb 26, 2020 at 17:56
  • Basically I want it to create the 'split' folders in the working directory
    – CJ Aslan
    Feb 26, 2020 at 18:14
  • ` dirname="dirprefix${f#*.}"` where "dirprefix" is anything you like.
    – xenoid
    Feb 26, 2020 at 18:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .